tts settings, and each request can use another one.
Speak a complete text
Speak text as it arrives
When an LLM writes the reply, send each piece withtts.append, and tts.end after the last one:
- The first
tts.appendwith anidstarts the request. Voice settings are read from that first message. - A word is used once a space follows it, or at
tts.end. - For quick first audio, the first two pieces are spoken as soon as each has 5 words or more and ends with
.!?؟،or,. Everything after them is spoken as one piece attts.end, so the speech flows naturally. - Always send
tts.end. Until it arrives the request isn’t finished, and later requests wait behind it. tts.startedcomes once the request is first in line, often before its first piece of text is ready.- A
tts.appendto a request that is still waiting or speaking but already gottts.end, or to attsrequest, gets aninvalid_messageerror. - Once a request has finished or been stopped, a
tts.appendwith the sameidstarts a new request. Use a newidfor each reply, and aftertts.cancel, stop sending the rest of that reply.
Request order
Requests on a session run one at a time, in the order you send them. Every audio frame belongs to the request announced by the lasttts.started, so you never need to sort audio by request. queue_ms in timing shows how long a request waited.
Audio
Each request starts withtts.started, which gives the audio format:
Always play the audio at the
sample_rate from tts.started. Frames hold whole samples, and the audio has no WAV header.
Play the audio in a browser
Play the audio in a browser
tts.output: "base64", audio comes in JSON messages instead of binary frames, for clients that can’t read binary frames:
seq counts from 0 within the request.
If your app reads slowly, Hamsa waits instead of sending audio faster than you can take it.
How a request ends
tts.end or tts.cancel for an id that is unknown or already finished does nothing and gets no reply.
timing is measured on Hamsa’s side, in milliseconds:
A field is
null when it doesn’t apply, for example first_audio_ms for a request stopped before any audio.
Stop speaking (interruptions)
id, it stops that request. Without it, it stops the current request and every waiting one. Speech generation stops at once. Requests that hadn’t started get tts.cancelled right away; the current one gets it once it has stopped.
To let the caller interrupt your agent:
- On
vad.speech_started, stop playing audio in your app, sendtts.cancel, and sendplayback: idle. - Ignore audio frames until the next
tts.started: frames already on their way may still arrive. - Stop sending
tts.appendfor the reply you stopped. - Wait for the caller’s
finaltranscript, then answer with a newid.
vad.speech_started instead, keep sending playback: playing, and decide on the transcript:
backchannel: true: restore the volume and keep speaking.- Any other transcript: stop and send
tts.cancel. - No transcript within about 2 seconds of
vad.speech_stopped(it was noise): restore the volume.
Voices
Set the session’s voice in the handshake, and change it withconfig.update or for one request:
voiceis a voice name of the dialect (any letter case). Names: Realtime voice names. The Playground lists them by dialect.dialect:pls,egy,ksa,uae,jor,leb,syr,irq,kuw,qat,bah,oma,msa,ar-sa, orenfor English voices (withlanguage: "en").- Some names exist in more than one dialect, so always send
voiceanddialecttogether. expressiveness: deprecated. Still accepted, from0(flat, steady) to2(very expressive), default1; don’t build on it.- A voice by ID: pass the ID of one of your cloned voices, or of a public voice, as
voice. It speaks in its own language and dialect unless the request setslanguageordialect. Another account’s voice ID getstts.errorwithvoice_not_available. Preload a cloned voice before you use it. model: a custom voice model made for your account, if you have one.
Pronunciation fixes
tts.dictionary replaces words or phrases before they are spoken:
- Only whole words and phrases match; a word inside a longer word doesn’t.
- Keys are words separated by spaces. A key with punctuation in it, such as “Dr.” or “e-mail”, never matches, and a phrase doesn’t match across punctuation in your text.
- Arabic diacritics are ignored when matching. Letter case isn’t.
- When phrases overlap, the longest one wins.
- Keys up to 200 characters, replacements up to 500.
- Set it in the handshake or with
config.update(the new dictionary replaces the old one, and applies to the rest of a streamed request that is speaking).handshake_ackshows how many entries it has, asdictionary_entries.