handshake_ack, send the caller’s audio as binary frames. Hamsa finds where speech starts and stops, transcribes it, and tells you when the caller’s turn is over.
Send audio
Send raw audio in the format you set in the handshake’saudio:
- Sample rate: 8000, 16000 (default), 22050, 24000, 32000, 44100 or 48000 Hz. 16000 is used as is, so it is the fastest.
- Channels: 1 (default) or 2. Stereo is interleaved and mixed down to mono.
- Frames: any size up to 1 MB. 20 to 100 ms per frame works best. A frame can end in the middle of a sample; the rest is taken from the next frame.
- Pace: send audio as it is captured, silence included. Hamsa decides where the speech is. When you stream a file, send it at real-time pace: if more than 5 seconds of audio is waiting to be analysed, the oldest of it is dropped.
- The audio format can’t change during the session.
- Audio sent before
handshake_ackis ignored.
Capture the microphone in a browser (16 kHz PCM16)
Capture the microphone in a browser (16 kHz PCM16)
Speech detection
Hamsa analyses the audio in 32 ms windows and decides, for each one, whether it is speech:- After
vad.min_speech_ms(default 100 ms) of speech, you getvad.speech_started. - After
vad.min_silence_ms(default 550 ms) of silence, you getvad.speech_stopped, and the speech segment is transcribed.vad.prefix_padding_ms(default 250 ms) of audio from before the speech is included, so the first syllable isn’t cut.
The time from the caller starting to talk to
vad.speech_started reaching you is detect_ms + server_lag_ms + network time. Use vad.speech_started to stop your agent talking when the caller interrupts. Turn the events off with stt.vad_events: false.
vad.threshold(default 0.5): raise it in noisy places so noise isn’t taken for speech.vad.min_speech_ms: lower fires sooner, but noise causes more false starts.vad.min_silence_ms: lower answers sooner, but may cut off callers who pause mid-sentence.vad.max_buffered_speech_s(default and most 30): the longest segment. Speech beyond it in the same segment isn’t transcribed.
Transcripts
Turns
Withstt.turn.mode: "enhanced" (default), speech segments are grouped into the caller’s turn. Each time a segment ends, the whole turn is transcribed again, which is more accurate than transcribing pieces. Then:
- If the caller talks again before the turn is final, the new speech joins the same turn, the whole turn is transcribed again, and you get a new
interimorpreflightwith the sameturn_id. - A number counts when the transcript has digits 0 to 9 separated by spaces, as in “0 5 9 9 1 2”, the way a caller reads out a phone or account number. “In the middle of a number” means the text ends with fewer than 8 such digits. A number shorter than
digits_preflight_minkeeps the turninterim. stt.flushends the turn now. If the caller is talking, you getvad.speech_stopped(withdetect_ms0), then theinterimorpreflightfor that speech, then thefinalwith reasonflush(oreoswhen end of speech said done andfinal_settle_msis0). If nothing is waiting, nothing is sent. A backchannel isn’t madefinal.- A turn longer than 30 seconds of audio is split: what came before is sent as
finalwith reasonmax_audio, and a new turn starts. - If the whole turn, transcribed again, has no words, it is dropped without a
final, even after an earlierinterimorpreflight. - After an
upstream_error, the turn stays open until the caller talks again. Sendstt.flushto end it with what was already transcribed.
stt.turn.mode: "segment", every speech segment gets one transcript of its own: final when end of speech says the caller is done and the text has no spaced-out number, otherwise interim (always interim with end of speech off). turn_id goes up with every message, there is no preflight, commit or backchannel, and you decide where turns end. stt.flush ends the current segment. The mode can’t change during the session.
Typical timing
With the default settings:interim comes at about +700 ms and the final about max_timeout_ms later, at about +1,400 ms.
Start your LLM on preflight to save that settle time. If a newer interim or preflight arrives for the same turn_id, the caller kept talking: drop that answer. The final that follows a preflight has the same text, so the answer stands.
Backchannels
Short replies such as “mhm”, “ok”, “تمام”, “نعم” or “أيوه” while your agent talks shouldn’t end the caller’s turn. Withstt.turn.backchannel:
A backchannel comes as
interim with backchannel: true. It isn’t made final, and turn_id doesn’t change. The match ignores letter case, Arabic diacritics and punctuation around the words. Replace the built-in Arabic and English list with stt.turn.backchannel_phrases (up to 500 phrases of up to 64 characters, fixed for the session).
Your app usually plays speech for a while after Hamsa finished sending it. Tell Hamsa while it plays:
idle whenever playback stops, including when you stop it for an interruption: until you do, Hamsa treats your app as playing and keeps counting short replies as backchannels. If you stop the speech as soon as the caller starts talking, nothing is playing when their words arrive, so a “mhm” becomes a normal turn. To keep backchannels, lower the volume first and decide on the transcript: see Stop speaking.
End of speech
End of speech decides whether the caller is done or only pausing, from what they said and how. It is on by default (stt.eos_enabled). stt.eos_threshold (default 0.3) is the probability above which the caller counts as done: raise it if callers get cut off, lower it to answer sooner. With end of speech off, transcripts are interim (eos is null), and the turn becomes final max_timeout_ms after the last one, with reason interim_timeout.
Gender
Withstt.gender_detection: true, transcripts have gender: Male or Female. Audio under 0.7 seconds gives null, since short audio isn’t reliable.
Speakers
Withstt.speaker_identification: true, transcripts have speaker, so you can tell people apart on the same audio:
idisSpeaker1,Speaker2, … in the order they are first heard. IDs are only valid within the session.is_newistruethe first time a speaker is heard.confidenceis the similarity to the closest known speaker, from -1 to 1. For a new speaker it is the similarity to the closest one it didn’t match, and1for the first speaker.- A voice more similar than
stt.speaker_similarity_threshold(default 0.2) to a known speaker is that speaker; otherwise it is a new one. Lower it if one person shows up as several speakers. - In
enhancedmode the turn is matched again each time it is transcribed, so a new speaker can first show up on aninterim. - Speaker identification and its threshold can’t change during the session.
Models and language
Both can change during the session with
config.update.
Text to speech only
To use the session only for speech, setstt.enabled: false in the handshake. Each audio frame then gets an stt_disabled error, so don’t send audio. It can’t change during the session.