> ## Documentation Index
> Fetch the complete documentation index at: https://docs.tryhamsa.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Speech to text

> Stream the caller's audio over the realtime WebSocket and get live transcripts: speech start and stop events, end of speech, turns, numbers, backchannels, gender and speakers.

After `handshake_ack`, send the caller's audio as binary frames. Hamsa finds where speech starts and stops, transcribes it, and tells you when the caller's turn is over.

## Send audio

Send raw audio in the format you set in the handshake's `audio`:

| `encoding` | Audio |
| - | - |
| `pcm16` (default) | Signed 16-bit little-endian samples |
| `mulaw` | 8-bit G.711 µ-law, as in phone calls |

* **Sample rate:** 8000, 16000 (default), 22050, 24000, 32000, 44100 or 48000 Hz. 16000 is used as is, so it is the fastest.
* **Channels:** 1 (default) or 2. Stereo is interleaved and mixed down to mono.
* **Frames:** any size up to 1 MB. 20 to 100 ms per frame works best. A frame can end in the middle of a sample; the rest is taken from the next frame.
* **Pace:** send audio as it is captured, silence included. Hamsa decides where the speech is. When you stream a file, send it at real-time pace: if more than 5 seconds of audio is waiting to be analysed, the oldest of it is dropped.
* The audio format can't change during the session.
* Audio sent before `handshake_ack` is ignored.

> [accordion] Capture the microphone in a browser (16 kHz PCM16)
>
> ```javascript theme={null}
> // Calls onFrame with 16 kHz mono PCM16 frames of about 40 ms.
> async function startMicrophone(onFrame) {
>   const stream = await navigator.mediaDevices.getUserMedia({
>     audio: { channelCount: 1, echoCancellation: true, noiseSuppression: true },
>   });
>   const context = new AudioContext();
>   const source = context.createMediaStreamSource(stream);
>   const processor = context.createScriptProcessor(2048, 1, 1);
>   const ratio = context.sampleRate / 16000;
>   let pending = [];
>
>   processor.onaudioprocess = (event) => {
>     const input = event.inputBuffer.getChannelData(0);
>     for (let i = 0; i < input.length / ratio; i++) {
>       const sample = Math.max(-1, Math.min(1, input[Math.floor(i * ratio)]));
>       pending.push(sample < 0 ? sample * 0x8000 : sample * 0x7fff);
>     }
>     while (pending.length >= 640) { // 640 samples = 40 ms at 16 kHz
>       onFrame(Int16Array.from(pending.splice(0, 640)).buffer);
>     }
>   };
>   source.connect(processor);
>   processor.connect(context.destination);
>
>   return () => { processor.disconnect(); source.disconnect(); stream.getTracks().forEach((t) => t.stop()); context.close(); };
> }
> ```

## Speech detection

Hamsa analyses the audio in 32 ms windows and decides, for each one, whether it is speech:

1. After `vad.min_speech_ms` (default 100 ms) of speech, you get `vad.speech_started`.
2. After `vad.min_silence_ms` (default 550 ms) of silence, you get `vad.speech_stopped`, and the speech segment is transcribed. `vad.prefix_padding_ms` (default 250 ms) of audio from before the speech is included, so the first syllable isn't cut.

```json theme={null}
{ "type": "vad.speech_started", "ts": 1760000000000, "offset_ms": 768, "detect_ms": 128, "server_lag_ms": 1, "backlog_ms": 0 }
{ "type": "vad.speech_stopped", "ts": 1760000002080, "audio_ms": 2458, "offset_ms": 2400, "detect_ms": 576, "server_lag_ms": 1, "backlog_ms": 0 }
```

| Field | Meaning |
| - | - |
| `ts` | Server time, in Unix milliseconds |
| `offset_ms` | Where the speech started or stopped, in the audio you've sent this session |
| `detect_ms` | Audio Hamsa needed after that point to decide: `min_speech_ms` or `min_silence_ms`, rounded up to 32 ms |
| `server_lag_ms` | Time from the deciding audio arriving to the event being sent. Usually 0 to 2 ms |
| `backlog_ms` | Audio received but not yet analysed. Usually under 32 ms |
| `audio_ms` | `vad.speech_stopped` only: length of the segment sent for transcription, including `prefix_padding_ms` before the speech and the `min_silence_ms` after it |

The time from the caller starting to talk to `vad.speech_started` reaching you is `detect_ms` + `server_lag_ms` + network time. Use `vad.speech_started` to stop your agent talking when the caller interrupts. Turn the events off with `stt.vad_events: false`.

* `vad.threshold` (default 0.5): raise it in noisy places so noise isn't taken for speech.
* `vad.min_speech_ms`: lower fires sooner, but noise causes more false starts.
* `vad.min_silence_ms`: lower answers sooner, but may cut off callers who pause mid-sentence.
* `vad.max_buffered_speech_s` (default and most 30): the longest segment. Speech beyond it in the same segment isn't transcribed.

## Transcripts

```json theme={null}
{
  "type": "transcript",
  "turn_id": 3,
  "kind": "final",
  "is_final": true,
  "text": "مرحبا بك في اختبار تحويل النص إلى كلام",
  "language": "ar",
  "eos": { "prediction": true, "probability": 0.98 },
  "gender": "Male",
  "speaker": { "id": "Speaker1", "confidence": 0.81, "is_new": false },
  "backchannel": false,
  "audio_ms": 2458,
  "stt_ms": 148,
  "commit": { "reason": "final_settle", "wait_ms": 402 }
}
```

| Field | Meaning |
| - | - |
| `turn_id` | The caller's turn, from 1. Every update of a turn has the same `turn_id`; it goes up by one after each `final` |
| `kind` | `interim`: the caller isn't done; the text can still change. `preflight`: the caller looks done; this becomes `final` unless they keep talking. `final`: the turn is over |
| `is_final` | `true` for `final` |
| `text` | What the caller said. In `enhanced` mode, the whole turn so far |
| `language` | Your `stt.language` setting |
| `eos` | End of speech: `prediction` is `true` when the caller looks done, `probability` how sure (can be `null`). `null` when `stt.eos_enabled` is `false` |
| `gender` | `Male` or `Female`. `null` when `stt.gender_detection` is off, or for under 0.7 seconds of audio |
| `speaker` | `null` unless `stt.speaker_identification` is on, and when no voice sample could be taken. See [Speakers](#speakers) |
| `backchannel` | `true` for a short reply that didn't end the turn. See [Backchannels](#backchannels) |
| `audio_ms` | Audio transcribed for this message |
| `stt_ms` | Time the transcription took |
| `commit` | `final` only, in `enhanced` mode: `reason` the turn ended, and `wait_ms` waited after the transcription |

## Turns

With `stt.turn.mode: "enhanced"` (default), speech segments are grouped into the caller's turn. Each time a segment ends, the whole turn is transcribed again, which is more accurate than transcribing pieces. Then:

| The transcription says | You get | The turn is final | `commit.reason` |
| - | - | - | - |
| The caller is done (end of speech) | `preflight` | Once `final_settle_ms` (400 ms) have passed since `vad.speech_stopped` with no new speech, and at most `max_timeout_ms` after the `preflight` | `final_settle` |
| The caller is done, and `final_settle_ms` is `0` | `final` right away | Now | `eos` |
| The caller isn't done, or end of speech is off | `interim` | `max_timeout_ms` (700 ms) after the `interim`, if the caller doesn't talk again | `interim_timeout` |
| The caller stopped in the middle of saying a number | `interim` | `digits_interim_timeout_ms` (3,000 ms) after the `interim`, if the caller doesn't talk again | `interim_timeout` |
| The caller is done and said a long number (`digits_preflight_min`, 6, digits or more) | `preflight` | `preflight_timeout_ms` (2,000 ms) after the `preflight`, if the caller doesn't talk again | `preflight_timeout` |

* If the caller talks again before the turn is final, the new speech joins the same turn, the whole turn is transcribed again, and you get a new `interim` or `preflight` with the same `turn_id`.
* A number counts when the transcript has digits 0 to 9 separated by spaces, as in "0 5 9 9 1 2", the way a caller reads out a phone or account number. "In the middle of a number" means the text ends with fewer than 8 such digits. A number shorter than `digits_preflight_min` keeps the turn `interim`.
* `stt.flush` ends the turn now. If the caller is talking, you get `vad.speech_stopped` (with `detect_ms` 0), then the `interim` or `preflight` for that speech, then the `final` with reason `flush` (or `eos` when end of speech said done and `final_settle_ms` is `0`). If nothing is waiting, nothing is sent. A backchannel isn't made `final`.
* A turn longer than 30 seconds of audio is split: what came before is sent as `final` with reason `max_audio`, and a new turn starts.
* If the whole turn, transcribed again, has no words, it is dropped without a `final`, even after an earlier `interim` or `preflight`.
* After an `upstream_error`, the turn stays open until the caller talks again. Send `stt.flush` to end it with what was already transcribed.

With `stt.turn.mode: "segment"`, every speech segment gets one transcript of its own: `final` when end of speech says the caller is done and the text has no spaced-out number, otherwise `interim` (always `interim` with end of speech off). `turn_id` goes up with every message, there is no `preflight`, `commit` or backchannel, and you decide where turns end. `stt.flush` ends the current segment. The mode can't change during the session.

### Typical timing

With the default settings:

```text theme={null}
caller stops talking
+550 ms    vad.speech_stopped              (min_silence_ms)
+700 ms    transcript, kind = preflight    (about 150 ms of transcription)
+1,000 ms  transcript, kind = final        (final_settle_ms after vad.speech_stopped)
```

When end of speech says the caller isn't done, the `interim` comes at about +700 ms and the `final` about `max_timeout_ms` later, at about +1,400 ms.

Start your LLM on `preflight` to save that settle time. If a newer `interim` or `preflight` arrives for the same `turn_id`, the caller kept talking: drop that answer. The `final` that follows a `preflight` has the same text, so the answer stands.

## Backchannels

Short replies such as "mhm", "ok", "تمام", "نعم" or "أيوه" while your agent talks shouldn't end the caller's turn. With `stt.turn.backchannel`:

| Value | A turn that is only a short reply |
| - | - |
| `while_tts` (default) | Is a backchannel while speech is being generated on the session, or while your app sends `playback: playing` |
| `always` | Is always a backchannel |
| `off` | Is never a backchannel: it is a normal turn |

A backchannel comes as `interim` with `backchannel: true`. It isn't made `final`, and `turn_id` doesn't change. The match ignores letter case, Arabic diacritics and punctuation around the words. Replace the built-in Arabic and English list with `stt.turn.backchannel_phrases` (up to 500 phrases of up to 64 characters, fixed for the session).

Your app usually plays speech for a while after Hamsa finished sending it. Tell Hamsa while it plays:

```json theme={null}
{ "type": "playback", "state": "playing" }
{ "type": "playback", "state": "idle" }
```

Send `idle` whenever playback stops, including when you stop it for an interruption: until you do, Hamsa treats your app as playing and keeps counting short replies as backchannels. If you stop the speech as soon as the caller starts talking, nothing is playing when their words arrive, so a "mhm" becomes a normal turn. To keep backchannels, lower the volume first and decide on the transcript: see [Stop speaking](https://docs.tryhamsa.com/websocket/realtime/text-to-speech.md#stop-speaking-interruptions).

## End of speech

End of speech decides whether the caller is done or only pausing, from what they said and how. It is on by default (`stt.eos_enabled`). `stt.eos_threshold` (default 0.3) is the probability above which the caller counts as done: raise it if callers get cut off, lower it to answer sooner. With end of speech off, transcripts are `interim` (`eos` is `null`), and the turn becomes `final` `max_timeout_ms` after the last one, with reason `interim_timeout`.

## Gender

With `stt.gender_detection: true`, transcripts have `gender`: `Male` or `Female`. Audio under 0.7 seconds gives `null`, since short audio isn't reliable.

## Speakers

With `stt.speaker_identification: true`, transcripts have `speaker`, so you can tell people apart on the same audio:

```json theme={null}
"speaker": { "id": "Speaker2", "confidence": 0.12, "is_new": true }
```

* `id` is `Speaker1`, `Speaker2`, ... in the order they are first heard. IDs are only valid within the session.
* `is_new` is `true` the first time a speaker is heard.
* `confidence` is the similarity to the closest known speaker, from -1 to 1. For a new speaker it is the similarity to the closest one it didn't match, and `1` for the first speaker.
* A voice more similar than `stt.speaker_similarity_threshold` (default 0.2) to a known speaker is that speaker; otherwise it is a new one. Lower it if one person shows up as several speakers.
* In `enhanced` mode the turn is matched again each time it is transcribed, so a new speaker can first show up on an `interim`.
* Speaker identification and its threshold can't change during the session.

## Models and language

| `stt.model` | Language |
| - | - |
| `Hamsa-STT-S4` (default) | Detects the language itself; `stt.language` is ignored. `Hamsa-STT-S2` and `Hamsa-STT-S2-deprecated` are still accepted as deprecated names for it |
| `Hamsa-STT-S3` | Uses `stt.language`: `ar` (default), `en` or `auto` |
| `Hamsa-STT-English` | English. Set `stt.language: "en"` with it |

Both can change during the session with `config.update`.

## Text to speech only

To use the session only for speech, set `stt.enabled: false` in the handshake. Each audio frame then gets an `stt_disabled` error, so don't send audio. It can't change during the session.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.