Skip to main content
WSS
Every message is JSON with a type, except audio, which goes in binary frames both ways. Start with a handshake; see Realtime WebSocket for the flow, errors and limits. To try the session with your microphone and hear the voices, use the Playground.
To try messages here, fill in the Handshake first (your key in api_key), then click Connect and send it: the server waits 5 seconds for it.
api_key
type:httpApiKey

Send the key as api_key in the handshake message. The api_key query parameter and the x-api-key header also work. A secret key (sk_) or an older (legacy) key; a public key (pk_) is refused.

handshake
type:object

Authenticate and configure the session. Must be the first message, within 5 seconds.

Audio (binary)
type:string

Your audio, as binary frames in the handshake's audio format.

config.update
type:object

Change settings during the session.

tts
type:object

Speak a complete text.

tts.append
type:object

Add text to a streamed request, for example as an LLM writes it. The first tts.append with an id starts the request.

tts.end
type:object

No more text for a streamed request: speak what is left.

tts.cancel
type:object

Stop a request. Without id, stops the current request and every waiting one.

stt.flush
type:object

End the caller's turn now (for example when a push-to-talk button is released): the current speech is transcribed, then the turn is made final with reason flush.

playback
type:object

Tell the server whether your app is still playing speech, so short replies are treated as backchannels while it plays.

ping
type:object

Check the connection, or keep a quiet session open. The reply is a pong with your ts, or the server time if you sent none.

close
type:object

End the session. The server closes the socket with code 1000.

handshake_ack
type:object

The session is open. Start sending audio and requests.

config.updated
type:object

The settings after a config.update.

vad.speech_started
type:object

The caller started speaking. Use it to stop your agent talking (barge-in).

vad.speech_stopped
type:object

The caller stopped speaking; the segment is being transcribed.

transcript
type:object

What the caller said. interim and preflight can still change; final ends the turn.

tts.started
type:object

A request started. Its audio follows, in this format.

Audio (binary)
type:string

Speech audio for the request announced by the last tts.started.

tts.audio
type:object

Speech audio as base64, when tts.output is base64.

tts.done
type:object

All of a request's audio was sent.

tts.cancelled
type:object

A request was stopped. A request stopped before it started has only id.

tts.error
type:object

A request failed. Later requests still run.

pong
type:object

Reply to ping.

error
type:object

Something went wrong. Before the session opens, the socket is then closed; after, the session stays open.

session.draining
type:object

The server is restarting. Finish what you are doing and open a new session.