{
"type": "handshake",
"api_key": "YOUR_API_KEY",
"config": {
"audio": {
"encoding": "pcm16",
"sample_rate": 16000
},
"stt": {
"model": "Hamsa-STT-S4",
"language": "ar",
"gender_detection": true
},
"tts": {
"voice": "Amjad",
"dialect": "pls"
}
}
}{}{
"type": "config.update",
"config": {
"vad": {
"min_silence_ms": 350
},
"stt": {
"turn": {
"final_settle_ms": 200
}
}
}
}{
"type": "tts",
"id": "greeting",
"text": "أهلا وسهلا، كيف يمكنني مساعدتك؟"
}{
"type": "tts.append",
"id": "reply-1",
"text": "أهلا وسهلا، "
}{
"type": "tts.end",
"id": "reply-1"
}{
"type": "tts.cancel",
"id": "reply-1"
}{
"type": "stt.flush"
}{
"type": "playback",
"state": "playing"
}{
"type": "ping",
"ts": 1760000000000
}{
"type": "close"
}{
"type": "handshake_ack",
"session_id": "6f1c2b8e-3a4d-4f0e-9b7a-1c2d3e4f5a6b",
"config": {
"audio": {
"encoding": "pcm16",
"sample_rate": 16000,
"channels": 1
},
"vad": {
"threshold": 0.5,
"min_speech_ms": 100,
"min_silence_ms": 550,
"prefix_padding_ms": 250,
"max_buffered_speech_s": 30
},
"stt": {
"enabled": true,
"model": "Hamsa-STT-S4",
"language": "ar",
"eos_enabled": true,
"eos_threshold": 0.3,
"gender_detection": true,
"speaker_identification": false,
"speaker_similarity_threshold": 0.2,
"vad_events": true,
"turn": {
"mode": "enhanced",
"final_settle_ms": 400,
"max_timeout_ms": 700,
"preflight_timeout_ms": 2000,
"digits_interim_timeout_ms": 3000,
"digits_preflight_min": 6,
"backchannel": "while_tts",
"custom_backchannel_phrases": 0
}
},
"tts": {
"voice": "Amjad",
"dialect": "pls",
"language": "ar",
"model": null,
"mulaw": false,
"output": "binary",
"dictionary_entries": 0
}
}
}{
"type": "config.updated",
"config": {
"audio": {
"encoding": "pcm16",
"sample_rate": 16000,
"channels": 1
},
"vad": {
"threshold": 0.5,
"min_speech_ms": 100,
"min_silence_ms": 550,
"prefix_padding_ms": 250,
"max_buffered_speech_s": 30
},
"stt": {
"enabled": true,
"model": "Hamsa-STT-S4",
"language": "ar",
"eos_enabled": true,
"eos_threshold": 0.3,
"gender_detection": true,
"speaker_identification": false,
"speaker_similarity_threshold": 0.2,
"vad_events": true,
"turn": {
"mode": "enhanced",
"final_settle_ms": 400,
"max_timeout_ms": 700,
"preflight_timeout_ms": 2000,
"digits_interim_timeout_ms": 3000,
"digits_preflight_min": 6,
"backchannel": "while_tts",
"custom_backchannel_phrases": 0
}
},
"tts": {
"voice": "Amjad",
"dialect": "pls",
"language": "ar",
"model": null,
"mulaw": false,
"output": "binary",
"dictionary_entries": 0
}
}
}{
"type": "vad.speech_started",
"ts": 1760000000000,
"offset_ms": 768,
"detect_ms": 128,
"server_lag_ms": 1,
"backlog_ms": 0
}{
"type": "vad.speech_stopped",
"ts": 1760000002080,
"audio_ms": 2458,
"offset_ms": 2400,
"detect_ms": 576,
"server_lag_ms": 1,
"backlog_ms": 0
}{
"type": "transcript",
"turn_id": 3,
"kind": "interim",
"is_final": false,
"text": "رقم حسابي هو",
"language": "ar",
"eos": {
"prediction": false,
"probability": 0.12
},
"gender": "Male",
"speaker": null,
"backchannel": false,
"audio_ms": 1450,
"stt_ms": 131
}{
"type": "tts.started",
"id": "greeting",
"encoding": "pcm16",
"sample_rate": 16000,
"channels": 1,
"output": "binary"
}{}{
"type": "tts.audio",
"id": "greeting",
"seq": 0,
"data": "AAABAAIAAwA="
}{
"type": "tts.done",
"id": "greeting",
"audio_ms": 2840,
"characters": 31,
"timing": {
"queue_ms": 0,
"text_wait_ms": 0,
"upstream_ttfb_ms": 171,
"first_audio_ms": 172,
"total_ms": 1050
}
}{
"type": "tts.cancelled",
"id": "reply-1",
"audio_ms": 960,
"timing": {
"queue_ms": 0,
"text_wait_ms": 412,
"upstream_ttfb_ms": 168,
"first_audio_ms": 580,
"total_ms": 1490
}
}{
"type": "tts.error",
"id": "greeting",
"code": "upstream_error",
"message": "Text-to-speech service error"
}{
"type": "pong",
"ts": 1760000000000
}{
"type": "error",
"code": "unauthorized",
"message": "API key was revoked",
"reason": "ApiKeyRevoked"
}{
"type": "session.draining",
"reconnect_within_ms": 90000
}Message reference
Every message of the realtime WebSocket, with its fields and examples: what you send and what Hamsa sends back.
type, except audio, which goes in binary frames both ways. Start with a handshake; see Realtime WebSocket for the flow, errors and limits. To try the session with your microphone and hear the voices, use the Playground.
api_key), then click Connect and send it: the server waits 5 seconds for it.{
"type": "handshake",
"api_key": "YOUR_API_KEY",
"config": {
"audio": {
"encoding": "pcm16",
"sample_rate": 16000
},
"stt": {
"model": "Hamsa-STT-S4",
"language": "ar",
"gender_detection": true
},
"tts": {
"voice": "Amjad",
"dialect": "pls"
}
}
}{}{
"type": "config.update",
"config": {
"vad": {
"min_silence_ms": 350
},
"stt": {
"turn": {
"final_settle_ms": 200
}
}
}
}{
"type": "tts",
"id": "greeting",
"text": "أهلا وسهلا، كيف يمكنني مساعدتك؟"
}{
"type": "tts.append",
"id": "reply-1",
"text": "أهلا وسهلا، "
}{
"type": "tts.end",
"id": "reply-1"
}{
"type": "tts.cancel",
"id": "reply-1"
}{
"type": "stt.flush"
}{
"type": "playback",
"state": "playing"
}{
"type": "ping",
"ts": 1760000000000
}{
"type": "close"
}{
"type": "handshake_ack",
"session_id": "6f1c2b8e-3a4d-4f0e-9b7a-1c2d3e4f5a6b",
"config": {
"audio": {
"encoding": "pcm16",
"sample_rate": 16000,
"channels": 1
},
"vad": {
"threshold": 0.5,
"min_speech_ms": 100,
"min_silence_ms": 550,
"prefix_padding_ms": 250,
"max_buffered_speech_s": 30
},
"stt": {
"enabled": true,
"model": "Hamsa-STT-S4",
"language": "ar",
"eos_enabled": true,
"eos_threshold": 0.3,
"gender_detection": true,
"speaker_identification": false,
"speaker_similarity_threshold": 0.2,
"vad_events": true,
"turn": {
"mode": "enhanced",
"final_settle_ms": 400,
"max_timeout_ms": 700,
"preflight_timeout_ms": 2000,
"digits_interim_timeout_ms": 3000,
"digits_preflight_min": 6,
"backchannel": "while_tts",
"custom_backchannel_phrases": 0
}
},
"tts": {
"voice": "Amjad",
"dialect": "pls",
"language": "ar",
"model": null,
"mulaw": false,
"output": "binary",
"dictionary_entries": 0
}
}
}{
"type": "config.updated",
"config": {
"audio": {
"encoding": "pcm16",
"sample_rate": 16000,
"channels": 1
},
"vad": {
"threshold": 0.5,
"min_speech_ms": 100,
"min_silence_ms": 550,
"prefix_padding_ms": 250,
"max_buffered_speech_s": 30
},
"stt": {
"enabled": true,
"model": "Hamsa-STT-S4",
"language": "ar",
"eos_enabled": true,
"eos_threshold": 0.3,
"gender_detection": true,
"speaker_identification": false,
"speaker_similarity_threshold": 0.2,
"vad_events": true,
"turn": {
"mode": "enhanced",
"final_settle_ms": 400,
"max_timeout_ms": 700,
"preflight_timeout_ms": 2000,
"digits_interim_timeout_ms": 3000,
"digits_preflight_min": 6,
"backchannel": "while_tts",
"custom_backchannel_phrases": 0
}
},
"tts": {
"voice": "Amjad",
"dialect": "pls",
"language": "ar",
"model": null,
"mulaw": false,
"output": "binary",
"dictionary_entries": 0
}
}
}{
"type": "vad.speech_started",
"ts": 1760000000000,
"offset_ms": 768,
"detect_ms": 128,
"server_lag_ms": 1,
"backlog_ms": 0
}{
"type": "vad.speech_stopped",
"ts": 1760000002080,
"audio_ms": 2458,
"offset_ms": 2400,
"detect_ms": 576,
"server_lag_ms": 1,
"backlog_ms": 0
}{
"type": "transcript",
"turn_id": 3,
"kind": "interim",
"is_final": false,
"text": "رقم حسابي هو",
"language": "ar",
"eos": {
"prediction": false,
"probability": 0.12
},
"gender": "Male",
"speaker": null,
"backchannel": false,
"audio_ms": 1450,
"stt_ms": 131
}{
"type": "tts.started",
"id": "greeting",
"encoding": "pcm16",
"sample_rate": 16000,
"channels": 1,
"output": "binary"
}{}{
"type": "tts.audio",
"id": "greeting",
"seq": 0,
"data": "AAABAAIAAwA="
}{
"type": "tts.done",
"id": "greeting",
"audio_ms": 2840,
"characters": 31,
"timing": {
"queue_ms": 0,
"text_wait_ms": 0,
"upstream_ttfb_ms": 171,
"first_audio_ms": 172,
"total_ms": 1050
}
}{
"type": "tts.cancelled",
"id": "reply-1",
"audio_ms": 960,
"timing": {
"queue_ms": 0,
"text_wait_ms": 412,
"upstream_ttfb_ms": 168,
"first_audio_ms": 580,
"total_ms": 1490
}
}{
"type": "tts.error",
"id": "greeting",
"code": "upstream_error",
"message": "Text-to-speech service error"
}{
"type": "pong",
"ts": 1760000000000
}{
"type": "error",
"code": "unauthorized",
"message": "API key was revoked",
"reason": "ApiKeyRevoked"
}{
"type": "session.draining",
"reconnect_within_ms": 90000
}Send the key as api_key in the handshake message. The api_key query parameter and the x-api-key header also work. A secret key (sk_) or an older (legacy) key; a public key (pk_) is refused.
Authenticate and configure the session. Must be the first message, within 5 seconds.
Your audio, as binary frames in the handshake's audio format.
Change settings during the session.
Speak a complete text.
Add text to a streamed request, for example as an LLM writes it. The first tts.append with an id starts the request.
No more text for a streamed request: speak what is left.
Stop a request. Without id, stops the current request and every waiting one.
End the caller's turn now (for example when a push-to-talk button is released): the current speech is transcribed, then the turn is made final with reason flush.
Tell the server whether your app is still playing speech, so short replies are treated as backchannels while it plays.
Check the connection, or keep a quiet session open. The reply is a pong with your ts, or the server time if you sent none.
End the session. The server closes the socket with code 1000.
The session is open. Start sending audio and requests.
The settings after a config.update.
The caller started speaking. Use it to stop your agent talking (barge-in).
The caller stopped speaking; the segment is being transcribed.
What the caller said. interim and preflight can still change; final ends the turn.
A request started. Its audio follows, in this format.
Speech audio for the request announced by the last tts.started.
Speech audio as base64, when tts.output is base64.
All of a request's audio was sent.
A request was stopped. A request stopped before it started has only id.
A request failed. Later requests still run.
Reply to ping.
Something went wrong. Before the session opens, the socket is then closed; after, the session stays open.
The server is restarting. Finish what you are doing and open a new session.
Was this page helpful?