Skip to main content
After handshake_ack, send the caller’s audio as binary frames. Hamsa finds where speech starts and stops, transcribes it, and tells you when the caller’s turn is over.

Send audio

Send raw audio in the format you set in the handshake’s audio:
  • Sample rate: 8000, 16000 (default), 22050, 24000, 32000, 44100 or 48000 Hz. 16000 is used as is, so it is the fastest.
  • Channels: 1 (default) or 2. Stereo is interleaved and mixed down to mono.
  • Frames: any size up to 1 MB. 20 to 100 ms per frame works best. A frame can end in the middle of a sample; the rest is taken from the next frame.
  • Pace: send audio as it is captured, silence included. Hamsa decides where the speech is. When you stream a file, send it at real-time pace: if more than 5 seconds of audio is waiting to be analysed, the oldest of it is dropped.
  • The audio format can’t change during the session.
  • Audio sent before handshake_ack is ignored.

Speech detection

Hamsa analyses the audio in 32 ms windows and decides, for each one, whether it is speech:
  1. After vad.min_speech_ms (default 100 ms) of speech, you get vad.speech_started.
  2. After vad.min_silence_ms (default 550 ms) of silence, you get vad.speech_stopped, and the speech segment is transcribed. vad.prefix_padding_ms (default 250 ms) of audio from before the speech is included, so the first syllable isn’t cut.
The time from the caller starting to talk to vad.speech_started reaching you is detect_ms + server_lag_ms + network time. Use vad.speech_started to stop your agent talking when the caller interrupts. Turn the events off with stt.vad_events: false.
  • vad.threshold (default 0.5): raise it in noisy places so noise isn’t taken for speech.
  • vad.min_speech_ms: lower fires sooner, but noise causes more false starts.
  • vad.min_silence_ms: lower answers sooner, but may cut off callers who pause mid-sentence.
  • vad.max_buffered_speech_s (default and most 30): the longest segment. Speech beyond it in the same segment isn’t transcribed.

Transcripts

Turns

With stt.turn.mode: "enhanced" (default), speech segments are grouped into the caller’s turn. Each time a segment ends, the whole turn is transcribed again, which is more accurate than transcribing pieces. Then:
  • If the caller talks again before the turn is final, the new speech joins the same turn, the whole turn is transcribed again, and you get a new interim or preflight with the same turn_id.
  • A number counts when the transcript has digits 0 to 9 separated by spaces, as in “0 5 9 9 1 2”, the way a caller reads out a phone or account number. “In the middle of a number” means the text ends with fewer than 8 such digits. A number shorter than digits_preflight_min keeps the turn interim.
  • stt.flush ends the turn now. If the caller is talking, you get vad.speech_stopped (with detect_ms 0), then the interim or preflight for that speech, then the final with reason flush (or eos when end of speech said done and final_settle_ms is 0). If nothing is waiting, nothing is sent. A backchannel isn’t made final.
  • A turn longer than 30 seconds of audio is split: what came before is sent as final with reason max_audio, and a new turn starts.
  • If the whole turn, transcribed again, has no words, it is dropped without a final, even after an earlier interim or preflight.
  • After an upstream_error, the turn stays open until the caller talks again. Send stt.flush to end it with what was already transcribed.
With stt.turn.mode: "segment", every speech segment gets one transcript of its own: final when end of speech says the caller is done and the text has no spaced-out number, otherwise interim (always interim with end of speech off). turn_id goes up with every message, there is no preflight, commit or backchannel, and you decide where turns end. stt.flush ends the current segment. The mode can’t change during the session.

Typical timing

With the default settings:
When end of speech says the caller isn’t done, the interim comes at about +700 ms and the final about max_timeout_ms later, at about +1,400 ms. Start your LLM on preflight to save that settle time. If a newer interim or preflight arrives for the same turn_id, the caller kept talking: drop that answer. The final that follows a preflight has the same text, so the answer stands.

Backchannels

Short replies such as “mhm”, “ok”, “تمام”, “نعم” or “أيوه” while your agent talks shouldn’t end the caller’s turn. With stt.turn.backchannel: A backchannel comes as interim with backchannel: true. It isn’t made final, and turn_id doesn’t change. The match ignores letter case, Arabic diacritics and punctuation around the words. Replace the built-in Arabic and English list with stt.turn.backchannel_phrases (up to 500 phrases of up to 64 characters, fixed for the session). Your app usually plays speech for a while after Hamsa finished sending it. Tell Hamsa while it plays:
Send idle whenever playback stops, including when you stop it for an interruption: until you do, Hamsa treats your app as playing and keeps counting short replies as backchannels. If you stop the speech as soon as the caller starts talking, nothing is playing when their words arrive, so a “mhm” becomes a normal turn. To keep backchannels, lower the volume first and decide on the transcript: see Stop speaking.

End of speech

End of speech decides whether the caller is done or only pausing, from what they said and how. It is on by default (stt.eos_enabled). stt.eos_threshold (default 0.3) is the probability above which the caller counts as done: raise it if callers get cut off, lower it to answer sooner. With end of speech off, transcripts are interim (eos is null), and the turn becomes final max_timeout_ms after the last one, with reason interim_timeout.

Gender

With stt.gender_detection: true, transcripts have gender: Male or Female. Audio under 0.7 seconds gives null, since short audio isn’t reliable.

Speakers

With stt.speaker_identification: true, transcripts have speaker, so you can tell people apart on the same audio:
  • id is Speaker1, Speaker2, … in the order they are first heard. IDs are only valid within the session.
  • is_new is true the first time a speaker is heard.
  • confidence is the similarity to the closest known speaker, from -1 to 1. For a new speaker it is the similarity to the closest one it didn’t match, and 1 for the first speaker.
  • A voice more similar than stt.speaker_similarity_threshold (default 0.2) to a known speaker is that speaker; otherwise it is a new one. Lower it if one person shows up as several speakers.
  • In enhanced mode the turn is matched again each time it is transcribed, so a new speaker can first show up on an interim.
  • Speaker identification and its threshold can’t change during the session.

Models and language

Both can change during the session with config.update.

Text to speech only

To use the session only for speech, set stt.enabled: false in the handshake. Each audio frame then gets an stt_disabled error, so don’t send audio. It can’t change during the session.