Skip to main content
Send text and get speech back on the same session, while speech to text keeps listening. The voice comes from the handshake’s tts settings, and each request can use another one.

Speak a complete text

Speak text as it arrives

When an LLM writes the reply, send each piece with tts.append, and tts.end after the last one:
  • The first tts.append with an id starts the request. Voice settings are read from that first message.
  • A word is used once a space follows it, or at tts.end.
  • For quick first audio, the first two pieces are spoken as soon as each has 5 words or more and ends with . ! ? ؟ ، or ,. Everything after them is spoken as one piece at tts.end, so the speech flows naturally.
  • Always send tts.end. Until it arrives the request isn’t finished, and later requests wait behind it.
  • tts.started comes once the request is first in line, often before its first piece of text is ready.
  • A tts.append to a request that is still waiting or speaking but already got tts.end, or to a tts request, gets an invalid_message error.
  • Once a request has finished or been stopped, a tts.append with the same id starts a new request. Use a new id for each reply, and after tts.cancel, stop sending the rest of that reply.

Request order

Requests on a session run one at a time, in the order you send them. Every audio frame belongs to the request announced by the last tts.started, so you never need to sort audio by request. queue_ms in timing shows how long a request waited.

Audio

Each request starts with tts.started, which gives the audio format:
Always play the audio at the sample_rate from tts.started. Frames hold whole samples, and the audio has no WAV header.
With tts.output: "base64", audio comes in JSON messages instead of binary frames, for clients that can’t read binary frames:
seq counts from 0 within the request. If your app reads slowly, Hamsa waits instead of sending audio faster than you can take it.

How a request ends

Every request ends with exactly one of these three. tts.end or tts.cancel for an id that is unknown or already finished does nothing and gets no reply. timing is measured on Hamsa’s side, in milliseconds: A field is null when it doesn’t apply, for example first_audio_ms for a request stopped before any audio.

Stop speaking (interruptions)

With id, it stops that request. Without it, it stops the current request and every waiting one. Speech generation stops at once. Requests that hadn’t started get tts.cancelled right away; the current one gets it once it has stopped. To let the caller interrupt your agent:
  1. On vad.speech_started, stop playing audio in your app, send tts.cancel, and send playback: idle.
  2. Ignore audio frames until the next tts.started: frames already on their way may still arrive.
  3. Stop sending tts.append for the reply you stopped.
  4. Wait for the caller’s final transcript, then answer with a new id.
So that short replies like “mhm” don’t interrupt, lower the volume on vad.speech_started instead, keep sending playback: playing, and decide on the transcript:
  • backchannel: true: restore the volume and keep speaking.
  • Any other transcript: stop and send tts.cancel.
  • No transcript within about 2 seconds of vad.speech_stopped (it was noise): restore the volume.
See Backchannels. The Playground does this with Interrupt when I talk: Smart.

Voices

Set the session’s voice in the handshake, and change it with config.update or for one request:
  • voice is a voice name of the dialect (any letter case). Names: Realtime voice names. The Playground lists them by dialect.
  • dialect: pls, egy, ksa, uae, jor, leb, syr, irq, kuw, qat, bah, oma, msa, ar-sa, or en for English voices (with language: "en").
  • Some names exist in more than one dialect, so always send voice and dialect together.
  • expressiveness: deprecated. Still accepted, from 0 (flat, steady) to 2 (very expressive), default 1; don’t build on it.
  • A voice by ID: pass the ID of one of your cloned voices, or of a public voice, as voice. It speaks in its own language and dialect unless the request sets language or dialect. Another account’s voice ID gets tts.error with voice_not_available. Preload a cloned voice before you use it.
  • model: a custom voice model made for your account, if you have one.

Pronunciation fixes

tts.dictionary replaces words or phrases before they are spoken:
  • Only whole words and phrases match; a word inside a longer word doesn’t.
  • Keys are words separated by spaces. A key with punctuation in it, such as “Dr.” or “e-mail”, never matches, and a phrase doesn’t match across punctuation in your text.
  • Arabic diacritics are ignored when matching. Letter case isn’t.
  • When phrases overlap, the longest one wins.
  • Keys up to 200 characters, replacements up to 500.
  • Set it in the handshake or with config.update (the new dictionary replaces the old one, and applies to the rest of a streamed request that is speaking). handshake_ack shows how many entries it has, as dictionary_entries.