# AvA Realtime Server (V2) – Client Integration Guide This guide explains how to build client experiences on top of the AvA realtime server (V2 stack). It focuses on what the server can do for your client, the message protocol, and recommended flows for text, speech, vision, and tool-calling use cases. ## Adopting the concurrent agentic flow (existing clients: start here) The server now runs multi-step work **concurrently with the conversation** — the user can keep talking while a task runs, independent actions run together, and the agent reports back when it finishes. Four things follow from that, in the order worth doing them: | # | What | Where | Skip it and… | |---|---|---|---| | 1 | Key chat bubbles on **`response_id`** | [`response_id` on `text-stream`](#server--client-messages) | the task's result overwrites the acknowledgement, or never appears | | 2 | Opt in with **`multi_response: true`** | same section | you stay on the old one-response-per-turn path (safe, just not concurrent) | | 3 | Send back a plugin's **artifact**, every time | [Returning a plugin's result](#returning-a-plugins-result-read-this-if-your-plugin-produces-an-artifact) | the agent honestly reports actions it cannot confirm, even though your UI shows them done | | 4 | Render **"waiting on you"** as its own step state | [Rendering a running plan](#rendering-a-running-plan) | the user watches a spinner for a step that is waiting on *them* | **Nothing breaks if you do none of it.** Concurrency is enabled per `client_source`, so an unadopted client keeps getting exactly one response per turn. Do (1) first — it is the only one that is a hard prerequisite; (2) turns the feature on; (3) and (4) are what make it feel like it is telling the truth. ## What the server provides - Bi-directional realtime over a single WebSocket for control + signaling, plus WebRTC for low-latency mic audio. - Multimodal input: text, speech (WebRTC or fallback WS PCM), optional image snapshots to add visual context. - Multimodal output: streaming text chunks and synthesized speech (Cartesia or Python TTS backend). Optional raw TTS PCM chunks to the client. - Agentic LLM with tool calling: server emits tool call requests; client executes tools and returns `tool-result`. - Server-executed legacy built-in tools for supported Groq GPT-OSS models via `agent_data.legacy_tools`. - Persona-aware behavior: fetches persona profile + chat history (when provided) and applies voice, language, and prompt defaults. - Dev log broadcast for debugging and admin visibility. - Session persistence hooks (chat transcript + summary) when tokens/persona IDs are provided. - Cross-session memory: rolling conversation summaries emitted as `rolling_summary` events for client-side persistence; `pre_chat_context` in `initial_config` restores memory on reconnect so Ava greets returning users by name and references prior conversations naturally. ## Quick start (happy path) 1. Open a WebSocket to the root path `/`. 2. Send `initial_config` with your user/persona + modality preferences, including STT provider and optional `lang` when needed (see config reference below). 3. Perform WebRTC signaling: send `offer`, then respond to `answer`/`candidate`. 4. Begin sending audio via WebRTC track (preferred) or fallback `mic_audio` frames; or send text via `get_chat_completion`. **Capture the mic with echo cancellation enabled and play the agent's audio from the WebRTC track — see "Echo cancellation is a client requirement" below.** Skipping this makes the agent hear itself on speaker setups. 5. Read `text-stream`, `ai_speech_start`, `speech_end`, `user_speech_start`, `user_speech_end`, and optional `speech_chunk` events for outputs. 6. If the server sends tool calls (`tool-calls`), run the tool client-side and reply with `tool-result`. 7. On exit, send `manual_disconnect` or simply close; server handles cleanup. ## Endpoints - WebSocket entry: `wss://avarealtime.pathor.ai/` - Health: `GET /healthz` - Status: `GET /status` returns uptime, version, active sessions - Active sessions: `GET /sessions` - Interactive docs: `GET /docs` — the developer site rendered from `V2/docs/index.html`. It includes a **live playground** (`/docs#playground`) that opens a real session against this server: paste a token, pick a persona or write a system prompt, talk to it, and watch every frame in a wire inspector. Get a token at . - This guide, raw: `GET /client_guide` — `text/markdown`, always the deployed version ## Authentication Every session requires a bearer token. The server validates the token against the PathOr persona API before setting up the agent. ### Token fields Send the token at the **top level** of `initial_config.value` or nested inside `agent_data` — the server accepts both: | Accepted field | Canonical name used internally | Notes | |---|---|---| | `curr_user_token` | `curr_user_token` | Preferred top-level field | | `api_key` | → normalised to `curr_user_token` | Alias accepted for compatibility | | `agent_data.curr_user_token` | same | Nested form, also accepted | ### Persona ID fields | Accepted field | Canonical name | Notes | |---|---|---| | `personaId` | `personaId` | Preferred top-level field | | `persona_id` | → normalised to `personaId` | Snake-case alias accepted | | `agent_data.persona_id` | same | Nested form, also accepted | ### Auth flow | Scenario | Server behaviour | |---|---| | **No token** | Sends `session-error` and closes the WebSocket immediately | | **Token only (no personaId)** | Authenticates the session, proceeds with AvA default persona and no persona fetch | | **Token + personaId** | Fetches persona profile from API — inflates system prompt, voice, language, tools, and chat history automatically | | **Invalid / expired token (401 or 403)** | Sends `session-error` and closes the WebSocket | ### Minimal config (recommended) Only token and personaId are required. Everything else — voice, language, system prompt, tools — is resolved from the persona profile automatically. ```json { "type": "initial_config", "value": { "curr_user_token": "your-jwt", "personaId": "your-persona-id" } } ``` ### Full `agent_data` passthrough If you already have the full persona payload (encrypted `tools_config` blob from the dashboard), you can pass it directly. The server will use it as-is after validating the token. ```json { "type": "initial_config", "value": { "curr_user_token": "your-jwt", "agent_data": { "persona_id": "your-persona-id", "curr_user_token": "your-jwt", "tools_config": "", "is_expressive_persona": true } } } ``` ### `client_source` `client_source` is **optional**. If omitted, the server falls back to `"web"`. The auth flow is identical regardless of `client_source`. ```json { "client_source": "web_agent" } // explicit { } // omitted → same result ``` Accepted values: `web`, `web_agent`, `ava_android`, `vidya_android`, or any custom string. `client_source` also decides the **default** for JIT plugin search — see below. ### `session_features` (what a session costs extra) Optional. Declares how a session is being DELIVERED, which the account is billed a percentage more for. Two values are understood, and anything else is dropped: ```json { "session_features": ["avatar", "presentation"] } ``` `avatar` — the reply is rendered by a 3D character with audio-driven lip-sync. `presentation` — that character is delivering a slide deck. Send it in `initial_config`, and send a frame when it changes: ```json { "type": "session_features", "features": ["avatar", "presentation"] } ``` The server **closes the billing meter before applying the change**, so every usage line is attributed to exactly the feature set it was produced under — a visitor who starts a presentation nine minutes in is charged the presentation rate for the presentation only. Sending it unchanged is a no-op, so a client may send it as often as it likes. Omit it entirely and the session bills exactly as it always has. **`ava_avatar_widget`** is the embeddable avatar widget (the kiosk renderer inside a visitor's browser on a customer's website). It is registered as a remote-execution source — plugin calls run on the Lambda, never on the visitor's device — and its sessions send `ephemeral_session: true` and a `curr_user_token` beginning `ava_wgt_`, a per-visitor credential the backend mints for one widget and one persona. Nothing else about the handshake differs. ### `plugin_search` (JIT plugin discovery) When a turn needs a capability none of the session's **connected** tools cover, the server can semantically search the plugin marketplace and either run a plugin the user already enabled or return a "connect this plugin?" consent card. That behavior is now **per client source**: | `client_source` | JIT plugin search default | | ---------------------------------- | ------------------------- | | `ava_android`, `vidya_android`, `ava_desktop` | **enabled** | | `web`, `web_agent`, anything else / omitted | **disabled** | Any client can override the default explicitly with `plugin_search`: ```json { "type": "initial_config", "value": { "client_source": "web", "plugin_search": true } } ``` - Accepts `true`/`false`, `"true"`/`"false"`, `"on"`/`"off"`, `1`/`0`. Aliases: `pluginSearch`, `plugin_search_enabled`, `jit_plugin_search`, `enable_plugin_search`. - **Toggle on the go:** send the same field on *any* later message (e.g. inside `get_chat_completion`) to flip discovery mid-session without reconnecting. The value is sticky until the client sends a different one, and applies from the next turn. ```json { "type": "get_chat_completion", "text": "book me a cab", "speech": true, "plugin_search": true } ``` When disabled the session is limited to its connected tools: no marketplace search, no consent cards, no JIT recovery after a tool failure — the agent answers honestly that it can't do it. Explicit plugin management ("enable the Spotify plugin") still works either way, and `disable_tool`/`disable_plugin` still overrides everything. ### Session error When auth fails, the server sends this before closing: ```json { "type": "session-error", "message": "Authentication failed. Invalid or missing token." } ``` ### Billing (`billing-blocked`) > **Enforcement is LIVE for every client except those in > `UNENFORCED_CLIENT_SOURCES`** (`V2/core/billingGate.js` — today only > `vidya_android`). Billing is a prepaid **USD balance**, not credits. > > This changed on 13 Aug 2026. Enforcement used to be an *allow-list* of > clients that had shipped the handling below; it is now a *deny-list*, so > **your client is enforced whether or not it implements this frame** — and a > session that sends no `client_source` at all (API-key and persona-token > integrations) is enforced too. > > Because of that the server no longer depends on you implementing it: see > [If your client does not handle the frame](#if-your-client-does-not-handle-the-frame). > Implementing it is still strongly recommended — it is the difference between > a recharge card and a sentence in the chat. When the balance runs out, the session **stops**. The server sends one frame and then refuses everything billable: ```json { "type": "billing-blocked", "reason": "insufficient_credits", "message": "Your balance is empty, so I can't start a new conversation right now. Top up and I'll be right back.", "remaining_credits": 0, "action": "open_credits", "fatal": true } ``` | Field | Meaning | | --- | --- | | `reason` | `insufficient_credits` — the balance was already empty at connect; the session is closed immediately and no audio stack is ever built. `exhausted_mid_session` — the balance ran out while the session was live; the socket closes ~2s later. `metering_unavailable` — usage could not be reported to billing for long enough that the session had run up more unmetered work than the server will hand out on trust; this is **not** a statement that the user is out of money, and a reconnect once billing recovers works normally. | | `message` | English fallback copy. A localized client should render its own text keyed on `reason` and use this only when it does not recognise the code. | | `remaining_credits` | Legacy field name; carries the remaining **USD** balance at the moment of the block. Zero or negative in practice — a turn already in flight is always billed, even into a negative balance, because work that happened cannot be un-done. | | `action` | Where to send the user. `open_credits` is the same action tag the assistant already emits for the billing screen, so clients that render action buttons need no new navigation. | | `fatal` | `true` = the session is over. Reserved for a future soft/per-request limit; nothing emits `false` today. | > **New reason code, 16 Aug 2026.** `metering_unavailable` was added alongside > the two above. Treat any unrecognised `reason` as "the session is over, show > `message`" — that is the whole point of `message` being on the frame, and it > means a new code never needs a client release to be handled safely. #### How the server notices mid-session (changed 16 Aug 2026) Exhaustion used to be detectable **only** as a side effect of a successful usage report on a fixed 30-second timer. That left three ways for a balance to go negative with nothing stopping it, all of them now closed: - **The gap between reports.** The check interval is now derived from how much runway the wallet has left, measured from consecutive balance readings. A healthy wallet is still checked every 30s and behaves exactly as before; a wallet seconds from zero is checked as often as every 2s. None of this runs on the turn path — there is no added latency to any request, at any balance. - **Sessions that report nothing.** A session producing no billable output sent no usage, so it learned no balance — while the user's *other* session, or a non-realtime service on the same wallet, spent it to zero. Idle sessions now read the balance directly once their last reading goes stale. - **Usage that was reported but never actually charged.** `/billing/usage/batch` prices each line independently and returns HTTP 200 even when it refused one, with the outcome in `results[i].recorded`. The server used to read only the status code, so a refused line was silently dropped — the work was served and never billed. Per-line verdicts are now applied, and a line the backend will never accept eventually stops the session rather than being served free forever. - **The user's other sessions.** A balance belongs to a user, not to a socket. The first session to see zero now stops **every** live session on the same wallet, so a phone and a desktop connected at once no longer keep spending after one of them has already been told to stop. (In-process: siblings on another server instance stop on their own, now much tighter, timer.) For clients this changes nothing structural — the frames and the required behaviour are identical. What changes is that `billing-blocked` now arrives **promptly** after the balance runs out rather than up to a full flush interval later, and it can arrive on a session that was sitting idle. **What the server does after sending it** - Discards every inbound message except the wind-down set: `ping`, `manual_disconnect`, `stop_speech`, `interrupt_operation`, `toggle_mute_speech`, WebRTC signalling, `end_video_chat`, `presentation_ended`, `end_dev_log`. A discarded message re-sends `billing-blocked` at most once every 5 seconds, so a client that keeps streaming `mic_audio` is not flooded. - Suppresses server-initiated turns too — auto-continue, a tool result coming back, an ambient decision to speak, a parked plan resuming. Being blocked is not something the session can talk its way out of. - Interrupts TTS and cancels in-flight generation, then closes the socket. **What the client must do** 1. Show the user why, in the conversation, with a control that opens the recharge screen. `action` tells you which one. 2. Stop sending. Do not queue the refused request for retry — it will be discarded again. 3. Do not auto-reconnect. A reconnect is re-gated at connect and closed again. Reconnect only after the user has been through the billing screen; the connect gate is the authority on whether they actually topped up. #### Advance warning (`billing-warning`) Before the hard stop, a session whose balance is running low gets: ```json { "type": "billing-warning", "reason": "low_balance", "balance_usd": 0.42, "action": "open_credits" } ``` - **Not fatal.** The session continues normally. This is a heads-up so the user can top up without being interrupted, and ignoring the frame is safe. - **At most once per session.** It is sent at connect if the balance is already low, and — new on 16 Aug 2026 — also when the balance *becomes* low while the session is live, which nothing previously reported. Before that change the first thing a user heard about a draining balance was the session ending. - `action` is the same `open_credits` tag as `billing-blocked`, so a client that renders one can render the other with no new navigation. #### If your client does not handle the frame A frame you do not implement is a silent refusal — mic open, nothing ever answers, no reason given. So when the blocked client is **not** one that renders `billing-blocked` as a recharge card, the server *also* narrates the refusal as an ordinary assistant line, on the `text-stream` channel every client already displays: ```json { "type": "text-stream", "text": "Your balance has just run out, so I have to stop here. Top up and we can pick up where we left off.", "chunk": "", "end": true, "response_id": "resp_7" } ``` - **You need no release to benefit from this.** It renders as a normal assistant message wherever chat is already displayed. - **The structured `billing-blocked` frame is still sent, always.** It remains the thing to parse. The text line is for the human, not the program. - **If your client declares `open_credits` in `availableActionTags`**, the text carries the tag inline — `… Top up and I'll be right back. [action: open_credits]` — and your existing action-button renderer turns it into a tappable control. If you do **not** declare it, the tag is omitted rather than leaked as raw markup, so you never show `[action: …]` to a user. - **Clients that already render the card do not get this line.** Saying the same thing twice — once as a card, once as a chat bubble — is noise. Today that means `ava_android` gets the card only. The practical consequence: on web and desktop the user now sees a clear explanation instead of a session that just stops, and on Android nothing changes. The card is still better than the sentence — implementing the frame is worth doing. **Fail-open, deliberately — but bounded.** If the billing API errors (500, timeout, anything that is not an explicit "no credits"), the session is **not** blocked. A billing outage must never silence a user who has paid. That fail-open used to be unbounded, which is its own failure mode: usage accrued locally for the whole session, invisible to the wallet, and landed as one debit at close — long after any gate could act on it. Since 16 Aug 2026 a session is stopped with `reason: "metering_unavailable"` once reporting has failed repeatedly **and** the unreported work is no longer small. Both conditions are required, so an ordinary blip is still absorbed silently, and the thresholds are server-side env knobs an incident can widen without a deploy. --- ## Client → server messages All messages are JSON unless noted. Binary-framed payloads are supported for `snapshot` and `mic_audio` to avoid Base64 overhead. - `offer` / `answer` / `candidate` - WebRTC signaling. Example: `{ "type": "offer", "offer": { ...sdp... } }` - `initial_config` - `{ "type": "initial_config", "value": { ...userConfigPatch } }` - Send once after connect; see config reference and Authentication section. - **`curr_user_token` (or alias `api_key`) is required.** The server rejects sessions without a token. - `personaId` (or alias `persona_id`) is optional. When provided alongside the token, the server fetches the persona profile automatically — voice, system prompt, tools, and chat history are all resolved server-side. When omitted, the session runs with AvA defaults. - `client_source` is optional (default `"web"`). - Preferred STT fields inside `value`: `stt_provider` and `lang`. - If `stt_provider` is `sarvam`, omitting `lang` enables Sarvam language auto-detection. - If `stt_provider` is `microsoft` or `openai_realtime`, send `lang` explicitly. - `update_stt_language` - Preferred: `{ "type": "update_stt_language", "lang": "en-IN" }` - Legacy alias also accepted: `{ "type": "update_stt_language", "language": "en-IN" }` - For Sarvam, use `lang: "unknown"` to force auto-detection mid-session. - `get_chat_completion` - `{ "type": "get_chat_completion", "text": "Hello", "speech": true, "image": "", "additional_current_context": "Cart: 2 items, total ₹1499", "exclude_from_chat_history": true }` - `image` is optional; sets the current snapshot for the turn. - `additional_current_context` is optional; when provided it replaces the session-sticky context used in future turns. - `exclude_from_chat_history` is optional boolean (default `false`); when `true`, the user's message and the assistant's generated response are not saved to the ongoing session history (temporary response-and-forget). - `plugin_search` is optional boolean; sending it here flips JIT plugin discovery for this and all following turns (see the `plugin_search` section above). - `update_additional_current_context` - `{ "type": "update_additional_current_context", "additional_current_context": "Cart: 2 items, total ₹1499" }` - Also accepted: `{ "type": "additional_current_context", "value": "Cart: 2 items, total ₹1499" }` - Updates (or clears with `""`) a sticky server-side context block appended to assistant instructions until replaced. - `update_user_details` - `{ "type": "update_user_details", "user_details": { "latitude": "18.8047601", "longitude": "73.3204305", "location": "Pune", "devicePermissions": { "approved": ["location"], "not_approved": [] } } }` - Also accepted: `{ "type": "user_details", "value": { ... } }` - **Merges** into the `user_details` sent in `initial_config` — send only the fields that changed. `null`/`undefined`/`""` values are ignored so a partial patch never blanks a known field. - Send this whenever the device location changes or a permission is granted; `initial_config` is a one-shot snapshot taken at connect. - Coordinates are the reason this exists: location-aware plugins (maps, nearby search, weather) take `latitude`/`longitude` tool parameters, and the agent can only fill them from what the client has pushed. Without them the agent honestly reports it cannot get the user's location. - Server replies with `ack_user_details` — `{ type, status: "ok", fields: string[], has_location: boolean }`. #### `user_details.availableActionTags` — declaring what in-app actions your client can handle The server emits `[action: …]` tags in assistant text to drive in-app behavior (open a screen, request a permission, open a URL). **The server no longer assumes which tags exist — the client declares them.** Send them in `user_details`, on `initial_config` and/or `update_user_details`, exactly like `devicePermissions`: ```json { "user_details": { "availableActionTags": { "open_settings": { "description": "Opens the Settings screen" }, "open_plugins": { "description": "Opens the Plugins screen" }, "open_profile": { "description": "Opens the user Profile screen" }, "ask_camera_permission": { "description": "Prompts the user to grant camera" }, "open_url": { "description": "Opens a URL in the in-app browser", "actionValue": "" } } } } ``` - **Key** = the exact tag token the client parses. **`description`** = what it does; the model uses this to pick the right one, so write it for a reader who cannot see your code. - **`actionValue`** (optional) marks a tag that carries a payload. The server renders it as `[action: open_url | actionValue: ]` — the form the client already parses. - Shorthand accepted: a bare string is treated as the description (`{"open_settings": "Opens Settings"}`), and a plain array of names works when you have no descriptions (`["open_settings", "open_plugins"]`). - **Permission tags must be named `ask__permission`** (e.g. `ask_camera_permission`). The server matches that middle keyword against `devicePermissions.not_approved` to decide whether a blocked capability is one it can ask you to unblock — see below. - **Send nothing and the assistant emits no tags at all.** There is deliberately no built-in fallback list: an undeclared tag is by definition one the app cannot service. Any `[action: …]` the model produces that is not on your list is stripped before it reaches you. - `show_plugin` is server-owned (plugin-consent cards) and always passes through — you do not need to declare it. **How it interacts with `devicePermissions`.** For a permission in `not_approved`: - a matching `ask_*_permission` tag **is** declared → the agent will not call the blocked tool; it explains and offers that exact tag. - no matching tag → the agent **calls the tool anyway** and reports the real device error. This is intentional: refusing up front on a permission the app cannot even prompt for is a dead end, and a stale permission list would otherwise silently disable working capabilities. - `tool-result` - `{ "type": "tool-result", "toolCallId": "id", "result": { ... }, "isDirect": false }` - `tool-result` is only used for explicit client-executed function tools emitted through `tool-calls`. It is not used for server-side legacy built-in tools configured through `agent_data.legacy_tools`. - **Only answer a `tool-calls` frame whose `executor` is `"client"`.** A frame with `executor: "server"` is telemetry — the server is running that call itself (internal subagent dispatch, in-process handler, or remote plugin execution) and is not waiting on you. Sending a `tool-result` for one is ignored, and older clients that answered everything are the reason this field exists: a client-authored refusal used to resolve the agent's pending call and get spoken as the answer while the real result was still in flight. Do not synthesise a placeholder or error result for a call you cannot run — an unanswered `executor: "client"` frame times out cleanly, and an invented answer becomes the agent's ground truth. - `toggle_mute_speech` - `{ "type": "toggle_mute_speech", "mute": true }` - `stop_speech` or `interrupt_operation` - Cancel ongoing LLM+TTS turn. - `manual_disconnect` - Graceful close acknowledgment. - `start_dev_log` / `end_dev_log` - Enable/disable dev log streaming. - `clear_agent_chat_history` - Clears server-side LLM history. - `snapshot` (binary-framed recommended) - Frame: `{"type":"snapshot","contentType":"image/jpeg"}\n` - Server uses it for the next LLM call. - `mic_audio` (fallback audio) - Base64 JSON: `{ "type": "mic_audio", "data": "" }` - Binary-framed: `{"type":"mic_audio","format":"s16le","sampleRate":48000,"channels":1}\n` - `start_video_chat` / `end_video_chat` - Turns on/off visual chat mode; optional `prompt` can be attached to `start_video_chat`. - `ping` - `{ "type": "ping", "client_ts": }` — application-level keepalive. Server replies with `pong`. `client_ts` is optional; used for round-trip latency measurement. - `update_meeting_participants` - Runtime roster management for ambient mode. No-op when ambient mode is not active. - join: `{ "type": "update_meeting_participants", "action": "join", "participant": { "name": "Ravi Sharma", "designation": "CTO", "speakerId": "speaker_0" } }` — `designation` and `speakerId` are optional. - leave: `{ "type": "update_meeting_participants", "action": "leave", "speakerId": "speaker_1" }` - sync: `{ "type": "update_meeting_participants", "action": "sync", "participants": [...] }` — full roster replacement. - Join/leave events are recorded in the ambient transcript so the engagement gate has contextual awareness of who is present. - `presentation_started` - `{ "type": "presentation_started", ... }` — notify server that a presentation has begun. Used by AI Presenter Mode. - `presentation_ready` - `{ "type": "presentation_ready", ... }` — notify server that the presentation is rendered and ready. Server replies with `ack_presentation_ready`. - `presentation_page_changed` - `{ "type": "presentation_page_changed", ... }` — notify server of a page/slide navigation event. - `presentation_ended` - `{ "type": "presentation_ended" }` — notify server that the presentation has ended. - `plugin_enabled` - `{ "type": "plugin_enabled", "plugin": { "_id": "...", "title": "Spotify" } }` - Sent when the user accepts a JIT consent card (see `server_instruction` → `show_plugin`). The server connects the plugin, then **resumes the parked request automatically** — no re-prompt, no second confirmation. Wiring this is what makes a consent card more than decoration. - `refresh_plugins` - `{ "type": "refresh_plugins" }` — optional `plugins` array to push a specific set. - Reloads the session's plugin/tool set mid-session without reconnecting; conversation context is preserved. Use after the user connects or disconnects a plugin in your own UI. - Server replies with `refresh_plugins_success` or `refresh_plugins_error`. A session started with `disable_tool: true` replies `{ toolsCount: 0, toolsDisabled: true }` and loads nothing — that is intentional and lasts the whole session. - `dashboard_init` - `{ "type": "dashboard_init", "weather": "28°C, light rain" }` — `weather` optional. - Asks the server to generate home-screen content before any conversation starts. Server replies once with `dashboard_init_response`. Duplicate requests while one is in flight are ignored. - `speaker_context` - `{ "type": "speaker_context", "speaker_name": "Ravi Sharma", "speaker_identity": "user_8821" }` - Active-speaker hint for meeting clients. The server receives a single **mixed** audio track, so STT cannot diarize it — this message tells the server who is currently talking so the ambient transcript and LLM context carry the right name. Send it on every active-speaker change (e.g. from LiveKit's `ActiveSpeakersChanged`). `speaker_identity` is optional. - `update_sys_prompt` - `{ "type": "update_sys_prompt", "prompt": "You are assisting with a live product demo." }` - Appends client-supplied framing to the system prompt. Shares a handler with `start_video_chat`, so it also turns visual chat on — use `get_chat_completion` or `additional_current_context` if you want to add context *without* enabling vision. ## Server → client messages - `session-created` — `{ type, clientId, auth, message }` - **Arrives as soon as the session is genuinely usable, which is BEFORE `persona-data`.** It is emitted once the connect-time wallet/auth gate has answered and the audio stack is up — it does not wait for the persona profile, because nothing a client does with this frame needs it. `auth` is always a boolean, never null. - **Do not treat it as "persona data has arrived."** The two frames were never ordered by contract and are now reliably the other way round. Read the persona from `persona-data` when it lands. - Sending a turn immediately after this frame is safe: the server holds it until the persona and agents are ready, so the first reply is never generated against a half-built session. - `session-error` — `{ type, message }` - `persona-data` — persona + tool config when fetched with credentials. Arrives shortly AFTER `session-created` (see above). - `text-stream` — `{ type, chunk, end?, text?, finish_reason? }` - `ai_speech_start` — marks start of TTS playback - `speech_end` — marks end/flush of TTS playback. For a spoken reply it is sent once the whole reply has been synthesised and its last frame has left the server's playback queue (`TTS_END_IDLE_MS`, default 120ms, after the queue drains), so no more audio for that reply is coming. The client's jitter buffer is still playing the last ~100-200ms at that point, so a half-duplex client should reopen its mic on the event **plus measured silence** on the received track, not on the event alone. It is also sent from reset/cancel paths with no audio behind it, so ignore one that arrives without a preceding `ai_speech_start`. - `user_speech_start` — `{ type, reason, interrupted }` — emitted when VAD or STT-interim detects the user has started speaking. `reason` is one of `"silero_vad"` | `"stt_interim"` | `"unknown"`. **`interrupted: true`** means the AI was actively playing TTS when the barge-in fired — clients should stop playback and discard their audio queue. **`interrupted: false`** means the AI was silent (e.g. user speaks before the AI's first response, or after `speech_end`) — clients should update their listening UI but must NOT discard a pending/queued AI response. - `user_speech_end` — `{ type, text }` — emitted when STT produces a final (accepted) transcript, signalling the end of the user's utterance. `text` is the recognised speech. Paired with `user_speech_start` for each complete utterance. - `speech_chunk` — `{ type, encoding: "base64", data }` (only when `receive_tts_chunks=true`) - `dev_log` — `{ type, message, log_level }` - `tool-activity` — `{ type, tool, tools, provider, path, phase, status_text, response_text, evidence?, citation_count? }` for server-side built-in tool usage such as browser search. Client-facing labels use Ava naming, for example `provider: "ava_live_tools"` and `path: "ava_legacy_tool_fast"`. `phase` is `requested` when the built-in-tool-backed request is dispatched and `completed` when usage is confirmed from the response. `status_text` carries the same conversational keep-alive phrase used for the spoken ephemeral progress update. - `tool-activity` is distinct from `tool-calls`: it reports work the server/provider performs internally, so the client should treat it as status/telemetry rather than a request to execute anything. - `server_instruction` — out-of-band UI instruction. Today the only action is `show_plugin`: `{ type: "server_instruction", action: "show_plugin", plugin: { _id, title, name, logo, human_description, creator, authentication_required, status, os?, isAuthenticated? } }` - Sent whenever the agent offers a plugin the user hasn't connected (JIT consent). The assistant's text carries a matching `[action: show_plugin | actionValue: <_id>]` tag — render the card in place of the tag; if the instruction arrives with no tag in the transcript, inject a standalone card. - **Tapping the card's toggle must send `plugin_enabled`.** The server has the user's original request parked behind that message and resumes it the moment it arrives — on every client and modality, with no further confirmation. A card with no `plugin_enabled` wired to it is a dead end. - `os` is present only when the plugin declares specific platforms; absent or empty means "runs everywhere". Disable the toggle when the list excludes this client rather than letting a tap fail. - `isAuthenticated` is present only when applicable; `authentication_required && !isAuthenticated` means the user must sign in before the plugin can work. - `status: "connected"` means already connected — show connected state, never a fresh "connect" prompt. - **`response_id` on `text-stream` (all clients — read this before touching chat rendering)** Every `text-stream` frame now carries `response_id` (e.g. `"resp_3"`), monotonic per session. A **response** is one unit of assistant output: a stream of chunk frames terminated by exactly one `end: true`. Until now a turn produced exactly one response, so clients could safely treat *"the turn is over"* and *"this reply is over"* as the same event. **That equivalence is going away.** The server now runs multi-step work **concurrently with the conversation** — the user can keep talking while a task runs, and the agent reports back when it finishes. A session therefore has **more than one response in flight**: an acknowledgement, anything said while the work runs, then the task's own result. `end: true` terminates **that `response_id`**, not the turn. Concretely, one "play a song, take a selfie, text Sneha" request now looks like: ``` resp_7 "On it — I'll let you know as soon as it's done." end:true ← turn closes here resp_8 "Kya kar rahi ho?" → "Bas isi pe lagi hoon!" end:true ← user talked mid-task resp_9 "Song chal raha hai, selfie bhi le li — Sneha ko end:true ← the task, reporting back message bhej diya hai, delivery confirm nahi kar paayi." ``` `resp_9` arrives with **no user message before it**. That is rule 3, and it is the whole point. **Client rules** 1. Key the assistant bubble on `response_id`. Chunks with a new id open a **new** bubble; do not append them to the bubble you were streaming into. 2. `end: true` finalizes **only** the bubble for that `response_id`. Never use it to clear global "is the assistant replying" state — another response may still be streaming. 3. A frame may arrive with **no preceding user message** (a task finishing, a proactive update). Render it as a normal assistant bubble; don't require a pending user turn. 4. Ignore ordering assumptions between different `response_id`s. Within one id, chunks are ordered. 5. `finish_reason` remains per-response. **Backwards compatible — by gating, not by luck.** Concurrency is enabled **per client source**. A client that has not adopted the rules above is never given more than one response per turn, so it behaves exactly as before. Nothing breaks by waiting. **Opting in.** Implement the five rules, then either ask for your `client_source` to be added server-side (`DETACHED_PLAN_EXECUTION_SOURCES`) or send the flag yourself in `initial_config` (or any later payload — it is read per turn, so it can be toggled mid-session): ```json { "multi_response": true } ``` `multi_response: false` opts back out at any time; `supports_multi_response` and `multiResponse` are accepted spellings. Adopted and verified live: **ava_android**, **ava_desktop** — both are on by default server-side, so those clients need not send the flag at all. Pending: vidya, web. **How to verify you adopted it correctly.** Ask for something with two or three device actions in it. You should see the acknowledgement bubble finalize, be able to type or speak *while the work runs* and get an answer, and then see a **separate** final bubble with the outcome. Failure modes to look for: the result overwriting the acknowledgement (rule 1), the result never appearing (rule 2 — you cleared global replying state on the first `end`), or the result being dropped for having no user message (rule 3). See `V2/AGENTIC_CONCURRENCY_ROADMAP.md` for why this exists and what lands next. - `ack_disconnect`, `ack_mute_speech` - `ack_additional_current_context` — `{ type, status: "ok", has_context: boolean, length: number }` - `ack_user_details` — `{ type, status: "ok", fields: string[], has_location: boolean }` — reply to `update_user_details` - `self_disconnect` — server-initiated close. `{ type, reason }` where `reason` is `"timeout"` (transport inactivity) or `"abandoned"` (no conversational activity for `SESSION_ABANDON_MS`, default 15 min). Treat both the same way — stop, and reconnect if the user comes back. A client that ignores `reason` is unaffected. - `tool-calls` — `{ type: "tool-calls", tools: , executor: "client" | "server" }` — `executor: "client"` means run it and reply with `tool-result`; `executor: "server"` means the server runs it and the frame is informational (render it as activity if useful, never answer it). Clients written before this field existed and treating every frame as client-executable still work, because the server rejects results for calls it did not delegate — but they will do pointless work. - `tool-result-raw` — `{ type: "tool-result-raw", tool_call_id, tool_name, executor: "server", arguments, ok, result, truncated, error, duration_ms, timestamp }` — **opt-in** (`stream_tool_results` in `initial_config`), emitted only for plugins the *server* executed. Read-only data for your UI: never answer it, and never treat it as the end of the turn. Off by default, so an existing client sees no change. - `pong` — `{ type: "pong", client_ts: , server_ts: }` — reply to application-level `ping` - `ack_presentation_ready` — `{ type: "ack_presentation_ready", status: "ok" }` — sent in response to `presentation_ready` - `rolling_summary` — `{ type: "rolling_summary", summary, foundation, turn_count, timestamp }` — emitted after every N turns once the conversation compacts. Client should persist (keyed by persona ID, up to 3 entries FIFO) and send back as `pre_chat_context` on the next connect. - `turn_suggestions` — `{ type: "turn_suggestions", suggestions: string[] }` — exactly three follow-up prompts, generated in the background after a persisted assistant turn. Render as tappable chips. Not emitted for a detached plan's acknowledgement ("on it, I'll let you know") — the real ones arrive with the plan's result a moment later. Generation failures are silent: no message, no error. - `dashboard_init_response` — `{ type: "dashboard_init_response", thought_of_the_day: string, suggestions: string[3], error?: string }` — reply to `dashboard_init`. On failure the server still sends usable fallback content **plus** `error`, so render the payload either way rather than branching on `error`. - `refresh_plugins_success` — `{ type: "refresh_plugins_success", status?: "success", message?: string, toolsCount: number, toolsDisabled?: true }` — reply to `refresh_plugins`. - `refresh_plugins_error` — `{ type: "refresh_plugins_error", status: "error", message: string }` — the plugin set is unchanged; the session keeps the tools it already had. - `speaker_identified` — `{ type: "speaker_identified", speakerId: "speaker_0", name: "Ravi Sharma", designation: string | null }` — ambient mode only. Emitted when a diarization label is auto-mapped to the next unassigned entry in the `meeting_participants` roster (appearance order). Use it to confirm or correct the mapping in a meeting UI; the mapping sticks for the rest of the session. ## Raw tool results (`tool-result-raw`) **Opt-in. Off by default. Changes nothing about the conversation.** On `web`, `web_agent`, `ava_kiosk`, `ava_ios` and `vidya_android` — and for any plugin that declares `web` in its supported OS list — the plugin runs **on the server**, not on your device. You see a `tool-calls` frame with `executor: "server"`, and then the only thing that comes back is the assistant's reply. That reply is a *summary* by design: the whole job of the orchestrator is to turn a JSON blob into one spoken sentence. So a front end that wants to draw the five emails, the live train position, or the search results has nothing to draw from. Set `stream_tool_results: true` in `initial_config` and the server also sends you the untouched result: ```json { "type": "initial_config", "value": { "client_source": "web", "stream_tool_results": true } } ``` Each server-executed plugin call then produces one frame: ```json { "type": "tool-result-raw", "tool_call_id": "call_9f2a__m1x7q", "tool_name": "gmail_v1_search", "executor": "server", "arguments": { "query": "is:unread", "max_results": 5 }, "ok": true, "result": { "emails": [ { "id": "…", "subject": "…" } ], "count": 5 }, "truncated": false, "error": null, "duration_ms": 812, "timestamp": "2026-08-23T11:04:07.913Z" } ``` On failure: ```json { "type": "tool-result-raw", "tool_name": "railways_v2_status", "ok": false, "result": null, "error": { "message": "Request failed with status code 502", "status": 502, "detail": { "error_type": "UpstreamError" } }, "duration_ms": 4180 } ``` ### Rules - **Never answer it.** This is not a `tool-calls` frame. There is nothing to reply with, and `tool-result` is not part of this exchange. - **It is not the end of the turn.** The assistant's own reply still streams over `text-stream` and TTS exactly as before, and arrives *after* this frame. Keep rendering the conversation the way you already do; treat this as a side panel, a card, a table — a second surface, not a replacement. - **`ok: false` is not the user-facing error.** The assistant will explain the failure in its own words. Show the raw error in a debug/details view if you want it; do not put a stack-shaped message in front of the user. - **Check `truncated`.** Results over 256 KB arrive as a truncated string with `truncated: true` and `result_bytes` giving the real size. Rare in practice, but do not parse a truncated payload as JSON. - **`result` keeps its type.** An object stays an object, a string stays a string — it is passed through byte-for-byte, exactly as the plugin returned it. Do not assume it is always JSON. - **Server-executed only.** Tools *you* ran (`executor: "client"`) are never mirrored back — you already have those results. Nothing else about the session changes: honesty reconciliation, the action ledger, follow-up handling and TTS all run identically whether or not this flag is set. Leaving it off keeps the exact behaviour you have today. ## Returning a plugin's result (READ THIS if your plugin produces an artifact) Some plugins produce something — a photo, a scan, a file, a screenshot — and the natural thing to do is post it back into the conversation as an ordinary message. That is fine and supported. **But the artifact has to reach the server.** ### Why it matters A device action the server dispatches but cannot verify is recorded as `dispatched_unconfirmed`: *"I sent it to your device, I can't confirm it worked."* That is deliberate — the server refuses to claim an outcome it has no evidence for. The **artifact is the evidence.** When it arrives, the server resolves that action to `confirmed`, and every downstream surface changes accordingly: the closing summary says it's done, a later "did that go through?" answers correctly, and any reply made while the work runs stops hedging. If the artifact never arrives, none of that happens — and it is not a bug. The assistant saying *"I couldn't confirm the selfie"* while the photo sits in your chat window means **the photo was rendered locally and never sent to the server.** ### The contract - Attach the artifact to the message you post back. For an image, `image` as base64 (without the data-URI prefix) on the same `get_chat_completion` payload that carries the caption. A binary frame works too. - **Send it every time**, not only when convenient. Intermittent delivery is worse than none: the assistant confirms the action on some runs and hedges on others, for reasons the user cannot see. - The caption alone is not evidence. A message reading "Captured the moment" with no attachment tells the server a user spoke, not that a capture succeeded. - Late is fine. The server resolves the action whenever the artifact lands — a different turn, several seconds later, any route. It does not need to arrive in the same frame as the caption, and the server will not block waiting for it. ### How to check Ask for something that produces an artifact. In the server log you want: ``` dispatch_confirmed_by_inbound_artifact source: 'chat_message' ``` If that line is absent, the artifact did not reach the server, and the assistant will correctly report the action as unconfirmed no matter what your UI is showing. ## Rendering a running plan With `multi_response` enabled a multi-step request unfolds over several responses, and a step has more states than "running" and "finished". | State | What it means | What the user needs to see | |---|---|---| | executing | the step is running | progress | | done | confirmed, or dispatched with its artifact returned | success | | **waiting on you** | the step asked a question and **cannot proceed until it is answered** | the question, prominently | | failed | it did not happen | plainly, without softening | **"Waiting on you" is the one that matters.** It is not a slower kind of executing — nothing will happen until the user replies. Showing it as "Executing" is actively misleading: the user watches a spinner for a step that is waiting on *them*. If your task card has two states, this is the one to add. The question itself arrives as a normal assistant response (its own `response_id`) and carries the options inline. If the user says something else in the meantime, the server re-asks it rather than dropping it — so expect the same question to legitimately appear more than once, and key any de-duplication on `response_id`, not on message text. Steps whose dependency failed are reported as **not done**, not silently omitted. If your UI hides them, the user sees "1 of 4 succeeded" as an unqualified success. ## User config reference (patchable via `initial_config`) The server maintains a per-session `userConfig`. Send the fields you care about; unspecified fields use defaults. | Key | Meaning | Common values / notes | | --------------------------------------- | ---------------------------------------------------------------------------- | ---------------------------------------------------------------------- | | `system_prompt` | System prompt seed | Defaults to a friendly AvA persona | | `additional_current_context` | Sticky client-provided latest app/user state appended to instructions | Example: cart, selected product, checkout step; empty string clears it | | `lang` | Preferred client-facing STT language key | `en-IN`, `hi-IN`, `unknown`; normalized server-side to `stt_lang` | | `tts_lang` | TTS language | Matches voice; defaults to `en-IN` | | `voice_type` | `non-custom` or `custom` | `custom` routes to Cartesia; `tts_provider` overrides too | | `voice_id` | Voice selection | e.g., `en-IN-Chirp3-HD-Leda` or Cartesia voice IDs | | `tts_provider` | `google` (Python WS) or `cartesia` | `cartesia` also used when `voice_type=custom` | | `tts_style_prompt` | Optional persona voice direction for Gemini TTS | e.g. "Speak in a soft, intimate tone"; ≤100 bytes (word-safe truncated) | | `stt_provider` | STT provider | `microsoft`, `sarvam`, `deepgram`, `openai_realtime`. **Send it explicitly** — the default is a server/env setting (`DEFAULT_STT_PROVIDER`), not a stable part of the API | | `stt_engine` | **Opt-in, Sarvam only.** `"sarvam_realtime"` moves the session to Sarvam's realtime endpoint (real partials, `stt_keyterms`, `stt_prompt`). Anything else, or omitted, keeps the existing Sarvam engine unchanged. See "Sarvam realtime STT". | string (default: omitted) | | `stt_keyterms` | Vocabulary to bias recognition toward (station names, train names, "PNR"). Realtime engine only; forces model `saaras:v4`. | array of strings — max 50, each ≤64 chars (over-long terms are dropped, not truncated) | | `stt_prompt` | Short context hint applied to each final transcript. Realtime engine only. | string, ≤600 chars | | `stt_model` | Realtime engine model override | `saaras:v4` (default) or `saaras:v3-realtime`; ignored (→ `saaras:v4`) when `stt_keyterms` is set | | `modalities` | Array of enabled outputs | `['text', 'audio']` | | `use_vad` | Enable VAD on incoming audio | `true`/`false` | | `vad_config` | VAD tuning. `threshold`: Silero speech probability. `minSpeechDuration` (s): sustained speech required to trigger barge-in **while the agent is speaking** (clamped 0.12–2.0s). `minSilenceDuration` (s): silence that re-arms the barge-in latch (clamped 0.2–3.0s). | `{ threshold: 0.75, minSpeechDuration: 0.2, minSilenceDuration: 0.4 }` | | `receive_tts_chunks` | If true, server sends PCM frames (`speech_chunk`, base64 24kHz) instead of injecting into the WebRTC audio track. **Warning: audio you play yourself (e.g. Web Audio API) is invisible to the browser/OS echo canceller — on speaker setups the mic will pick the agent's voice back up. Keep this `false` unless you implement your own AEC-safe playback path.** | bool (default `false`) | | `use_openai_realtime` | Route generation to OpenAI Realtime instead of internal agent | bool | | `use_groq`, `groq_model` | Groq routing flags | optional | | `max_completion_tokens`, `temperature` | LLM params | numbers | | `curr_user_token` | **Required.** Auth token for the session (see Authentication section) | Alias: `api_key` also accepted | | `personaId` | Optional persona ID to fetch profile from API | Alias: `persona_id` also accepted; omit to use AvA defaults | | `client_source` | Optional client identifier | Default `"web"`. Values: `web`, `web_agent`, `ava_android`, etc. | | `agent_data` | Optional full persona + tools payload (pass-through if you have the encrypted blob) | When present, server uses it directly after validating token | | `start_prompt` | Auto-run a prompt after session create | string — automatically overridden for returning users when `pre_chat_context` is present | | `pre_chat_context` | Prior-session rolling summaries to restore on reconnect | Array of `{ summary, foundation, turn_count, timestamp }` objects sent in `initial_config` | | `notification_context` | Simulated notification message clicked by the user to start/resume session | Object containing `{ assistant_message, assistant_name, timestamp }` | | `noise_supression`, `echo_cancellation` | **Server-side** DSP toggles (RNNoise denoise / NLMS echo filter on inbound mic audio). These do NOT replace client-side echo cancellation — real AEC must run on the device, where the speaker signal is known. See "Echo cancellation is a client requirement". | bool | | `mute_speech` | Soft-mute server TTS | bool | | `ephemeral_session` | **Shared-terminal mode.** Nothing this session says is read from or written to the persona's stored chat history — the history fetch is skipped at connect and persistence is refused. Persona, voice, tools and plugins are unaffected. Use on public kiosks/demo booths where the speaker is not the account holder. | bool or `"true"` (default `false`). Aliases: `ephemeral`, `no_chat_history`, `disable_chat_persistence` | | `game_mode` | Isolate session for game play (disables persona plugins, restricts tools to client-supplied `tools`) | bool (default `false`) | | `disable_tool` / `disable_plugin` | Load persona + chat history but **no plugins/tools** — session runs a single lowest-latency fast-chat path (no validator, no JIT plugin discovery, no consent gate). Vision still works. Set once in `initial_config`. | bool or `"true"` (default `false`). Aliases `disable_tools` / `disable_plugins` also accepted | | `plugin_search` | Enable/disable JIT plugin discovery (marketplace search + connect cards) for the session. Also accepted on any later message to toggle mid-session. | bool or `"true"`/`"false"` — default depends on `client_source` (ON for `ava_android`/`vidya_android`/`ava_desktop`, OFF for `web`/`web_agent`). Aliases: `pluginSearch`, `plugin_search_enabled`, `jit_plugin_search`, `enable_plugin_search` | | `stream_tool_results` | Mirror the **raw result of every server-executed plugin** back as a `tool-result-raw` frame, so a front end can render the actual data instead of only the assistant's spoken summary. Purely additive — the conversation is unchanged. Set once in `initial_config`. See "Raw tool results (`tool-result-raw`)". | bool or `"true"` (default `false`). Aliases: `streamToolResults`, `stream_raw_tool_results`, `raw_tool_results`, `emit_tool_results`, `send_tool_results` | | `tools` | Client-supplied custom tools (OpenAI function format) used when `game_mode` is true | Array of tool objects | | `ambient_intelligence` | Enable ambient conversational mode (see below) | `false` (default) | | `ambient_meeting_context` | Short description of the meeting/call type fed to the engagement gate | e.g. `"weekly sales team standup"` — optional | | `ambient_trigger_names` | Reserved; currently unused — engagement gate uses self-identity from the persona system prompt instead of keyword lists | `[]` — kept for forward compatibility | | `ambient_mode_config` | Per-session ambient tuning overrides (all fields optional, merged with server env defaults) | `{ "stt_provider": "deepgram", "silence_gap_ms": 400, "question_gap_ms": 150, "context_window_turns": 8, "max_tokens": 150 }` | | `meeting_participants` | Participant roster for Deepgram diarization label resolution | `[{ "name": "Ravi Sharma", "designation": "CTO", "speakerId": "speaker_0" }]` — `speakerId` optional; auto-mapped by first-speech order when omitted | | `capability_notes` | Free-text hints about client-side features injected into the ambient gate prompt | e.g. `"You can present URLs and PDFs as a screen share."` | ### STT configuration Preferred client contract for speech recognition: - `stt_provider`: STT backend to use — `microsoft`, `sarvam`, `deepgram`, or `openai_realtime`. - `lang`: STT language. **Send `stt_provider` explicitly.** When you omit it the session takes the server's `DEFAULT_STT_PROVIDER` (code default `microsoft`; the current deployment sets `deepgram`), which is an operational setting and can change under you without any client-visible API change. Behavior by provider: - `sarvam`: `lang` is optional. If omitted, the server lets Sarvam auto-detect the spoken language. You can also send `lang: "unknown"` to force auto-detection explicitly. Sarvam additionally runs a turn endpointer that merges a continued utterance into one turn instead of splitting it. - `microsoft`: send `lang` explicitly, for example `en-IN`. - `deepgram`: send `lang` explicitly. This is the provider that supports **speaker diarization**, so ambient/meeting sessions default to it. - `openai_realtime`: send `lang` explicitly for predictable behavior. Compatibility notes: - The server still accepts legacy `stt_lang` in `initial_config`. - The server also accepts legacy `language` in `update_stt_language`. Examples: ```json { "type": "initial_config", "value": { "stt_provider": "sarvam" } } ``` ```json { "type": "initial_config", "value": { "stt_provider": "sarvam", "lang": "hi-IN" } } ``` ```json { "type": "initial_config", "value": { "stt_provider": "microsoft", "lang": "en-IN" } } ``` ### Sarvam realtime STT (`stt_engine: "sarvam_realtime"`, opt-in) For sessions where specific words must be heard right — station and train names at a railway kiosk, drug names, product names — Sarvam's newer realtime endpoint accepts a vocabulary list. Opt in per session: ```json { "type": "initial_config", "value": { "stt_provider": "sarvam", "lang": "hi-IN", "stt_engine": "sarvam_realtime", "stt_keyterms": ["गोमती नगर", "गोंडा", "बादशाहनगर", "चारबाग", "Gomti Nagar", "Gonda", "Badshahnagar", "Charbagh", "Pushpak Express", "PNR"], "stt_prompt": "Indian Railways enquiry kiosk at Gomti Nagar station, Lucknow." } } ``` | Field | Meaning | | --- | --- | | `stt_engine` | `"sarvam_realtime"` selects `wss://api.sarvam.ai/speech-to-text-realtime/ws`. Anything else, or omitted, keeps the existing engine (`/speech-to-text/ws`, `saaras:v3`) unchanged. Only read when the session's STT provider is `sarvam`. | | `stt_keyterms` | Up to **50** terms, each ≤**64** characters, one phrase per item (`"Gomti Nagar"`, never `"Gonda, Basti"`). Order is priority: if the list ever has to be shortened it is cut from the end. Duplicates (case-insensitive) are removed; an over-long term is dropped rather than truncated. Setting it forces `saaras:v4`, the only model Sarvam accepts keyterms on. Fixed for the session. | | `stt_prompt` | Optional context hint Sarvam applies to each final transcript. ≤600 chars (trimmed beyond). | | `stt_model` | Optional: `saaras:v4` (default) or `saaras:v3-realtime`. Ignored when `stt_keyterms` is set. | What does not change: `lang` (explicit code, or omitted / `"unknown"` for auto-detect — sent to Sarvam as `language_code=auto`), the turn endpointer (one turn per utterance, `user_speech_start`/`user_speech_end`, `stt` frames), reconnect with back-off, the deaf watchdog and provider failover, keepalive, and pending-audio replay after a reconnect. What is new: `stt` frames with `interim: true` now arrive **while the user is still speaking** (the legacy engine only had text once a segment ended). Keyterms bias recognition; Sarvam does not promise a term will appear. **List the local names in both scripts.** A `hi-IN` session writes Devanagari, and Latin-only keyterms reach it only partly. In the benchmark, Latin-only scored 13/20 names in concourse noise and Latin + Devanagari scored 17/20 (the legacy engine scored 6/20). Keyterms have a cost too: on audio it could not make out, the model sometimes returned a keyterm that was never said ("Kanpur Central" → "Gomti Nagar"). Keep the list to names the site actually needs. Treat a station name in a transcript as something to confirm before acting on it, not as ground truth. **Failure mode.** If Sarvam refuses the realtime configuration (WebSocket close `4000`, or a 4xx at the handshake — e.g. HTTP 414 when the keyterm list makes the URL too long), the session falls back to the existing engine for the rest of the session and logs `sarvam_rt_rejected_fallback_legacy`. The user keeps being heard; only the vocabulary bias is lost. **Latency.** Measured from the end of speech to the committed turn (turn endpointer included), the median is 1.95 s against 1.63 s on the legacy engine. With `SARVAM_RT_SILENCE_DURATION_MS=500` it is 1.73 s, at the cost of one extra noisy miss in 20. Server tuning (env, all optional): `SARVAM_RT_MODEL`, `SARVAM_RT_STREAM_TYPE` (`fast` default), `SARVAM_RT_SILENCE_DURATION_MS` (700), `SARVAM_RT_PREFIX_PADDING_MS` (400), `SARVAM_RT_MIN_SPEECH_DURATION_MS`, `SARVAM_RT_THRESHOLD`. Benchmark before changing any of it: `node V2/scratch/sarvamRealtimeSttBench.js` (Sarvam-TTS clips of station/train names, clean and with concourse babble, streamed in real time through the real provider). Offline wiring checks: `node V2/scratch/sarvamRealtimeSttCheck.js`. `agent_data` fields used by persona-aware flows: - `curr_user_token`, `persona_id`, `persona_value`, `persona_languages` - `tools_config`, `main_agent_tools` - `legacy_tools` - `is_expressive_persona` (enables richer prosody) - `chat_history` (seeded to system prompt) ### Legacy built-in tools via `agent_data.legacy_tools` Use `agent_data.legacy_tools` when you want the model provider to execute supported built-in tools directly instead of the server emitting client-side `tool-calls`. - Supported values today: `browser_search`, `code_interpreter` - Alias accepted on input: `code_execution` normalizes to `code_interpreter` - Provider restriction: only used on Groq-backed GPT-OSS legacy-tool paths - Precedence rule: legacy built-ins are attached only when no normal function tools are active for that turn - Client contract: these tools do not generate `tool-calls`; instead the client may receive `tool-activity` events Example: ```json { "type": "initial_config", "value": { "use_groq": true, "groq_model": "openai/gpt-oss-120b", "agent_data": { "legacy_tools": ["browser_search", "code_execution"] } } } ``` Behavior notes: - If `main_agent_tools` or persona-provided function tools are present, those function tools take precedence and `legacy_tools` are ignored for that turn. - If the selected model does not support a requested legacy tool, unsupported entries are dropped. - Browser-search turns may emit an early `tool-activity` event with `phase: "requested"` and a conversational `status_text` before the final answer is ready. - The keep-alive status phrase is spoken server-side but is not sent as normal `text-stream` assistant content. ## Recommended flows ### Additional current context (important for integration) `additional_current_context` is the canonical way for clients to send current app state (cart, selected item, active screen, checkout step, permissions state, etc.) to the server-side agent. - Scope: per WebSocket session (sticky until replaced/cleared). - Semantics: **replace**, not merge. Sending a new value overwrites the previous context. - Clear behavior: send empty string `""` (or non-string, which normalizes to empty) to clear. - Limit: server trims whitespace and caps length to `MAX_ADDITIONAL_CURRENT_CONTEXT_CHARS` (default `8000`). Over-cap values are truncated at a word boundary and a `additional_current_context_truncated` warning is logged (originalLength/cappedLength). - Prompt wiring: appended to the system instructions for direct agent and orchestrator paths; also re-applied to OpenAI Realtime session instructions when that mode is active. Ways to send it: 1. In `initial_config`: ```json { "type": "initial_config", "value": { "client_source": "web_agent", "additional_current_context": "cart_total=1499; currency=INR; selected_sku=ABC123" } } ``` 2. Mid-session update (recommended): ```json { "type": "update_additional_current_context", "additional_current_context": "cart_total=1999; currency=INR; selected_sku=XYZ789" } ``` 3. Per-turn with `get_chat_completion` (also updates sticky context): ```json { "type": "get_chat_completion", "text": "Can I apply available coupons?", "speech": true, "additional_current_context": "cart_total=1999; coupon_screen=true" } ``` Server ack after context update: ```json { "type": "ack_additional_current_context", "status": "ok", "has_context": true, "length": 47 } ``` Client best practices: - Keep it compact and factual (state, IDs, flags), avoid long prose. - Update only when state changes materially. - Prefer stable keys (`selected_sku`, `cart_total`, `step`, `locale`) for predictable behavior. - Do not put secrets/PII unless required; this context is used in model instructions. ### Server defaults (env) - `DEFAULT_STT_PROVIDER`: STT provider used when the client omits `stt_provider`. Code default `microsoft`; the current deployment sets `deepgram`. Supported values: `microsoft`, `sarvam`, `deepgram`, `openai_realtime`. Treat it as deployment configuration, not API surface — clients that care should send `stt_provider`. ### Speech-to-speech (WebRTC-first) 1. Connect WS → send `initial_config` with `modalities: ['audio','text']`, `stt_provider` if you want to override the server default, and `lang` when you are not relying on Sarvam auto-detect. 2. WebRTC negotiate (`offer`/`answer`/`candidate`). 3. Start sending microphone audio on the WebRTC track (captured per the AEC requirements below). VAD runs server-side and feeds the configured STT provider + LLM. 4. Listen for `text-stream` for transcripts and `ai_speech_start` / `speech_end` for synthesized speech events. 5. Keep `receive_tts_chunks=false`; the server injects TTS into the WebRTC audio track, which is the only playback path the platform echo canceller can subtract. Setting it to true delivers PCM chunks you must play yourself — outside the echo-cancelled path. ### Echo cancellation is a client requirement The agent's voice comes out of the device speaker and back into the device mic. The only place that loop can be cancelled properly is **on the device**, where the OS knows exactly what is being played. The server adds echo-resistant barge-in gating and self-echo transcript filtering as a safety net, but without client AEC a speakerphone setup will degrade (delayed barge-in commits, occasional swallowed first words). This is the same contract Gemini Live and OpenAI Realtime rely on. **Browser (web) clients:** ```js // 1. Capture the mic WITH processing enabled (do not disable these): const mic = await navigator.mediaDevices.getUserMedia({ audio: { echoCancellation: true, // REQUIRED — cancels the agent's own voice noiseSuppression: true, autoGainControl: true, channelCount: 1, }, }); pc.addTrack(mic.getAudioTracks()[0], mic); // 2. Play the agent's audio via the remote WebRTC track and a media element. // Do NOT decode/play agent audio through the Web Audio API // (AudioContext/AudioWorklet) — Chromium's AEC cannot see that path and // will not cancel it. pc.ontrack = (ev) => { const audioEl = document.getElementById("agent-audio"); //