# AvA Realtime Server (V2) – Client Integration Guide

This guide explains how to build client experiences on top of the AvA realtime server (V2 stack). It focuses on what the server can do for your client, the message protocol, and recommended flows for text, speech, vision, and tool-calling use cases.

## Adopting the concurrent agentic flow (existing clients: start here)

The server now runs multi-step work **concurrently with the conversation** — the user can keep talking while a task runs, independent actions run together, and the agent reports back when it finishes. Four things follow from that, in the order worth doing them:

| # | What | Where | Skip it and… |
|---|---|---|---|
| 1 | Key chat bubbles on **`response_id`** | [`response_id` on `text-stream`](#server--client-messages) | the task's result overwrites the acknowledgement, or never appears |
| 2 | Opt in with **`multi_response: true`** | same section | you stay on the old one-response-per-turn path (safe, just not concurrent) |
| 3 | Send back a plugin's **artifact**, every time | [Returning a plugin's result](#returning-a-plugins-result-read-this-if-your-plugin-produces-an-artifact) | the agent honestly reports actions it cannot confirm, even though your UI shows them done |
| 4 | Render **"waiting on you"** as its own step state | [Rendering a running plan](#rendering-a-running-plan) | the user watches a spinner for a step that is waiting on *them* |

**Nothing breaks if you do none of it.** Concurrency is enabled per `client_source`, so an unadopted client keeps getting exactly one response per turn. Do (1) first — it is the only one that is a hard prerequisite; (2) turns the feature on; (3) and (4) are what make it feel like it is telling the truth.

## What the server provides

- Bi-directional realtime over a single WebSocket for control + signaling, plus WebRTC for low-latency mic audio.
- Multimodal input: text, speech (WebRTC or fallback WS PCM), optional image snapshots to add visual context.
- Multimodal output: streaming text chunks and synthesized speech (Cartesia or Python TTS backend). Optional raw TTS PCM chunks to the client.
- Agentic LLM with tool calling: server emits tool call requests; client executes tools and returns `tool-result`.
- Server-executed legacy built-in tools for supported Groq GPT-OSS models via `agent_data.legacy_tools`.
- Persona-aware behavior: fetches persona profile + chat history (when provided) and applies voice, language, and prompt defaults.
- Dev log broadcast for debugging and admin visibility.
- Session persistence hooks (chat transcript + summary) when tokens/persona IDs are provided.
- Cross-session memory: rolling conversation summaries emitted as `rolling_summary` events for client-side persistence; `pre_chat_context` in `initial_config` restores memory on reconnect so Ava greets returning users by name and references prior conversations naturally.

## Quick start (happy path)

1. Open a WebSocket to the root path `/`.
2. Send `initial_config` with your user/persona + modality preferences, including STT provider and optional `lang` when needed (see config reference below).
3. Perform WebRTC signaling: send `offer`, then respond to `answer`/`candidate`.
4. Begin sending audio via WebRTC track (preferred) or fallback `mic_audio` frames; or send text via `get_chat_completion`. **Capture the mic with echo cancellation enabled and play the agent's audio from the WebRTC track — see "Echo cancellation is a client requirement" below.** Skipping this makes the agent hear itself on speaker setups.
5. Read `text-stream`, `ai_speech_start`, `speech_end`, `user_speech_start`, `user_speech_end`, and optional `speech_chunk` events for outputs.
6. If the server sends tool calls (`tool-calls`), run the tool client-side and reply with `tool-result`.
7. On exit, send `manual_disconnect` or simply close; server handles cleanup.

## Endpoints

- WebSocket entry: `wss://avarealtime.pathor.ai/`
- Health: `GET /healthz`
- Status: `GET /status` returns uptime, version, active sessions
- Active sessions: `GET /sessions`
- Interactive docs: `GET /docs` — the developer site rendered from `V2/docs/index.html`. It includes a **live playground** (`/docs#playground`) that opens a real session against this server: paste a token, pick a persona or write a system prompt, talk to it, and watch every frame in a wire inspector. Get a token at <https://console.ava.pathor.ai/>.
- This guide, raw: `GET /client_guide` — `text/markdown`, always the deployed version

## Authentication

Every session requires a bearer token. The server validates the token against the PathOr persona API before setting up the agent.

### Token fields

Send the token at the **top level** of `initial_config.value` or nested inside `agent_data` — the server accepts both:

| Accepted field | Canonical name used internally | Notes |
|---|---|---|
| `curr_user_token` | `curr_user_token` | Preferred top-level field |
| `api_key` | → normalised to `curr_user_token` | Alias accepted for compatibility |
| `agent_data.curr_user_token` | same | Nested form, also accepted |

### Persona ID fields

| Accepted field | Canonical name | Notes |
|---|---|---|
| `personaId` | `personaId` | Preferred top-level field |
| `persona_id` | → normalised to `personaId` | Snake-case alias accepted |
| `agent_data.persona_id` | same | Nested form, also accepted |

### Auth flow

| Scenario | Server behaviour |
|---|---|
| **No token** | Sends `session-error` and closes the WebSocket immediately |
| **Token only (no personaId)** | Authenticates the session, proceeds with AvA default persona and no persona fetch |
| **Token + personaId** | Fetches persona profile from API — inflates system prompt, voice, language, tools, and chat history automatically |
| **Invalid / expired token (401 or 403)** | Sends `session-error` and closes the WebSocket |

### Minimal config (recommended)

Only token and personaId are required. Everything else — voice, language, system prompt, tools — is resolved from the persona profile automatically.

```json
{
  "type": "initial_config",
  "value": {
    "curr_user_token": "your-jwt",
    "personaId": "your-persona-id"
  }
}
```

### Full `agent_data` passthrough

If you already have the full persona payload (encrypted `tools_config` blob from the dashboard), you can pass it directly. The server will use it as-is after validating the token.

```json
{
  "type": "initial_config",
  "value": {
    "curr_user_token": "your-jwt",
    "agent_data": {
      "persona_id": "your-persona-id",
      "curr_user_token": "your-jwt",
      "tools_config": "<encrypted-blob>",
      "is_expressive_persona": true
    }
  }
}
```

### `client_source`

`client_source` is **optional**. If omitted, the server falls back to `"web"`. The auth flow is identical regardless of `client_source`.

```json
{ "client_source": "web_agent" }   // explicit
{ }                                 // omitted → same result
```

Accepted values: `web`, `web_agent`, `ava_android`, `vidya_android`, or any custom string.

`client_source` also decides the **default** for JIT plugin search — see below.

### `plugin_search` (JIT plugin discovery)

When a turn needs a capability none of the session's **connected** tools cover, the server can semantically search the plugin marketplace and either run a plugin the user already enabled or return a "connect this plugin?" consent card. That behavior is now **per client source**:

| `client_source`                    | JIT plugin search default |
| ---------------------------------- | ------------------------- |
| `ava_android`, `vidya_android`, `ava_desktop` | **enabled**    |
| `web`, `web_agent`, anything else / omitted   | **disabled**   |

Any client can override the default explicitly with `plugin_search`:

```json
{ "type": "initial_config", "value": { "client_source": "web", "plugin_search": true } }
```

- Accepts `true`/`false`, `"true"`/`"false"`, `"on"`/`"off"`, `1`/`0`. Aliases: `pluginSearch`, `plugin_search_enabled`, `jit_plugin_search`, `enable_plugin_search`.
- **Toggle on the go:** send the same field on *any* later message (e.g. inside `get_chat_completion`) to flip discovery mid-session without reconnecting. The value is sticky until the client sends a different one, and applies from the next turn.

```json
{ "type": "get_chat_completion", "text": "book me a cab", "speech": true, "plugin_search": true }
```

When disabled the session is limited to its connected tools: no marketplace search, no consent cards, no JIT recovery after a tool failure — the agent answers honestly that it can't do it. Explicit plugin management ("enable the Spotify plugin") still works either way, and `disable_tool`/`disable_plugin` still overrides everything.

### Session error

When auth fails, the server sends this before closing:

```json
{
  "type": "session-error",
  "message": "Authentication failed. Invalid or missing token."
}
```

---

## Client → server messages

All messages are JSON unless noted. Binary-framed payloads are supported for `snapshot` and `mic_audio` to avoid Base64 overhead.

- `offer` / `answer` / `candidate`
  - WebRTC signaling. Example: `{ "type": "offer", "offer": { ...sdp... } }`
- `initial_config`
  - `{ "type": "initial_config", "value": { ...userConfigPatch } }`
  - Send once after connect; see config reference and Authentication section.
  - **`curr_user_token` (or alias `api_key`) is required.** The server rejects sessions without a token.
  - `personaId` (or alias `persona_id`) is optional. When provided alongside the token, the server fetches the persona profile automatically — voice, system prompt, tools, and chat history are all resolved server-side. When omitted, the session runs with AvA defaults.
  - `client_source` is optional (default `"web"`).
  - Preferred STT fields inside `value`: `stt_provider` and `lang`.
  - If `stt_provider` is `sarvam`, omitting `lang` enables Sarvam language auto-detection.
  - If `stt_provider` is `microsoft` or `openai_realtime`, send `lang` explicitly.
- `update_stt_language`
  - Preferred: `{ "type": "update_stt_language", "lang": "en-IN" }`
  - Legacy alias also accepted: `{ "type": "update_stt_language", "language": "en-IN" }`
  - For Sarvam, use `lang: "unknown"` to force auto-detection mid-session.
- `get_chat_completion`
  - `{ "type": "get_chat_completion", "text": "Hello", "speech": true, "image": "<base64-jpeg>", "additional_current_context": "Cart: 2 items, total ₹1499", "exclude_from_chat_history": true }`
  - `image` is optional; sets the current snapshot for the turn.
  - `additional_current_context` is optional; when provided it replaces the session-sticky context used in future turns.
  - `exclude_from_chat_history` is optional boolean (default `false`); when `true`, the user's message and the assistant's generated response are not saved to the ongoing session history (temporary response-and-forget).
  - `plugin_search` is optional boolean; sending it here flips JIT plugin discovery for this and all following turns (see the `plugin_search` section above).
- `update_additional_current_context`
  - `{ "type": "update_additional_current_context", "additional_current_context": "Cart: 2 items, total ₹1499" }`
  - Also accepted: `{ "type": "additional_current_context", "value": "Cart: 2 items, total ₹1499" }`
  - Updates (or clears with `""`) a sticky server-side context block appended to assistant instructions until replaced.
- `update_user_details`
  - `{ "type": "update_user_details", "user_details": { "latitude": "18.8047601", "longitude": "73.3204305", "location": "Pune", "devicePermissions": { "approved": ["location"], "not_approved": [] } } }`
  - Also accepted: `{ "type": "user_details", "value": { ... } }`
  - **Merges** into the `user_details` sent in `initial_config` — send only the fields that changed. `null`/`undefined`/`""` values are ignored so a partial patch never blanks a known field.
  - Send this whenever the device location changes or a permission is granted; `initial_config` is a one-shot snapshot taken at connect.
  - Coordinates are the reason this exists: location-aware plugins (maps, nearby search, weather) take `latitude`/`longitude` tool parameters, and the agent can only fill them from what the client has pushed. Without them the agent honestly reports it cannot get the user's location.
  - Server replies with `ack_user_details` — `{ type, status: "ok", fields: string[], has_location: boolean }`.
- `tool-result`
  - `{ "type": "tool-result", "toolCallId": "id", "result": { ... }, "isDirect": false }`
- `tool-result` is only used for explicit client-executed function tools emitted through `tool-calls`. It is not used for server-side legacy built-in tools configured through `agent_data.legacy_tools`.
- `toggle_mute_speech`
  - `{ "type": "toggle_mute_speech", "mute": true }`
- `stop_speech` or `interrupt_operation`
  - Cancel ongoing LLM+TTS turn.
- `manual_disconnect`
  - Graceful close acknowledgment.
- `start_dev_log` / `end_dev_log`
  - Enable/disable dev log streaming.
- `clear_agent_chat_history`
  - Clears server-side LLM history.
- `snapshot` (binary-framed recommended)
  - Frame: `{"type":"snapshot","contentType":"image/jpeg"}\n<jpeg-bytes>`
  - Server uses it for the next LLM call.
- `mic_audio` (fallback audio)
  - Base64 JSON: `{ "type": "mic_audio", "data": "<base64 pcm s16le 48k mono>" }`
  - Binary-framed: `{"type":"mic_audio","format":"s16le","sampleRate":48000,"channels":1}\n<pcm-bytes>`
- `start_video_chat` / `end_video_chat`
  - Turns on/off visual chat mode; optional `prompt` can be attached to `start_video_chat`.
- `ping`
  - `{ "type": "ping", "client_ts": <epoch_ms> }` — application-level keepalive. Server replies with `pong`. `client_ts` is optional; used for round-trip latency measurement.
- `update_meeting_participants`
  - Runtime roster management for ambient mode. No-op when ambient mode is not active.
  - join: `{ "type": "update_meeting_participants", "action": "join", "participant": { "name": "Ravi Sharma", "designation": "CTO", "speakerId": "speaker_0" } }` — `designation` and `speakerId` are optional.
  - leave: `{ "type": "update_meeting_participants", "action": "leave", "speakerId": "speaker_1" }`
  - sync: `{ "type": "update_meeting_participants", "action": "sync", "participants": [...] }` — full roster replacement.
  - Join/leave events are recorded in the ambient transcript so the engagement gate has contextual awareness of who is present.
- `presentation_started`
  - `{ "type": "presentation_started", ... }` — notify server that a presentation has begun. Used by AI Presenter Mode.
- `presentation_ready`
  - `{ "type": "presentation_ready", ... }` — notify server that the presentation is rendered and ready. Server replies with `ack_presentation_ready`.
- `presentation_page_changed`
  - `{ "type": "presentation_page_changed", ... }` — notify server of a page/slide navigation event.
- `presentation_ended`
  - `{ "type": "presentation_ended" }` — notify server that the presentation has ended.
- `plugin_enabled`
  - `{ "type": "plugin_enabled", "plugin": { "_id": "...", "title": "Spotify" } }`
  - Sent when the user accepts a JIT consent card (see `server_instruction` → `show_plugin`). The server connects the plugin, then **resumes the parked request automatically** — no re-prompt, no second confirmation. Wiring this is what makes a consent card more than decoration.
- `refresh_plugins`
  - `{ "type": "refresh_plugins" }` — optional `plugins` array to push a specific set.
  - Reloads the session's plugin/tool set mid-session without reconnecting; conversation context is preserved. Use after the user connects or disconnects a plugin in your own UI.
  - Server replies with `refresh_plugins_success` or `refresh_plugins_error`. A session started with `disable_tool: true` replies `{ toolsCount: 0, toolsDisabled: true }` and loads nothing — that is intentional and lasts the whole session.
- `dashboard_init`
  - `{ "type": "dashboard_init", "weather": "28°C, light rain" }` — `weather` optional.
  - Asks the server to generate home-screen content before any conversation starts. Server replies once with `dashboard_init_response`. Duplicate requests while one is in flight are ignored.
- `speaker_context`
  - `{ "type": "speaker_context", "speaker_name": "Ravi Sharma", "speaker_identity": "user_8821" }`
  - Active-speaker hint for meeting clients. The server receives a single **mixed** audio track, so STT cannot diarize it — this message tells the server who is currently talking so the ambient transcript and LLM context carry the right name. Send it on every active-speaker change (e.g. from LiveKit's `ActiveSpeakersChanged`). `speaker_identity` is optional.
- `update_sys_prompt`
  - `{ "type": "update_sys_prompt", "prompt": "You are assisting with a live product demo." }`
  - Appends client-supplied framing to the system prompt. Shares a handler with `start_video_chat`, so it also turns visual chat on — use `get_chat_completion` or `additional_current_context` if you want to add context *without* enabling vision.

## Server → client messages

- `session-created` — `{ type, clientId, auth, message }`
- `session-error` — `{ type, message }`
- `persona-data` — persona + tool config when fetched with credentials
- `text-stream` — `{ type, chunk, end?, text?, finish_reason? }`
- `ai_speech_start` — marks start of TTS playback
- `speech_end` — marks end/flush of TTS playback
- `user_speech_start` — `{ type, reason, interrupted }` — emitted when VAD or STT-interim detects the user has started speaking. `reason` is one of `"silero_vad"` | `"stt_interim"` | `"unknown"`. **`interrupted: true`** means the AI was actively playing TTS when the barge-in fired — clients should stop playback and discard their audio queue. **`interrupted: false`** means the AI was silent (e.g. user speaks before the AI's first response, or after `speech_end`) — clients should update their listening UI but must NOT discard a pending/queued AI response.
- `user_speech_end` — `{ type, text }` — emitted when STT produces a final (accepted) transcript, signalling the end of the user's utterance. `text` is the recognised speech. Paired with `user_speech_start` for each complete utterance.
- `speech_chunk` — `{ type, encoding: "base64", data }` (only when `receive_tts_chunks=true`)
- `dev_log` — `{ type, message, log_level }`
- `tool-activity` — `{ type, tool, tools, provider, path, phase, status_text, response_text, evidence?, citation_count? }` for server-side built-in tool usage such as browser search. Client-facing labels use Ava naming, for example `provider: "ava_live_tools"` and `path: "ava_legacy_tool_fast"`. `phase` is `requested` when the built-in-tool-backed request is dispatched and `completed` when usage is confirmed from the response. `status_text` carries the same conversational keep-alive phrase used for the spoken ephemeral progress update.
- `tool-activity` is distinct from `tool-calls`: it reports work the server/provider performs internally, so the client should treat it as status/telemetry rather than a request to execute anything.
- `server_instruction` — out-of-band UI instruction. Today the only action is `show_plugin`:
  `{ type: "server_instruction", action: "show_plugin", plugin: { _id, title, name, logo, human_description, creator, authentication_required, status, os?, isAuthenticated? } }`
  - Sent whenever the agent offers a plugin the user hasn't connected (JIT consent). The assistant's text carries a matching `[action: show_plugin | actionValue: <_id>]` tag — render the card in place of the tag; if the instruction arrives with no tag in the transcript, inject a standalone card.
  - **Tapping the card's toggle must send `plugin_enabled`.** The server has the user's original request parked behind that message and resumes it the moment it arrives — on every client and modality, with no further confirmation. A card with no `plugin_enabled` wired to it is a dead end.
  - `os` is present only when the plugin declares specific platforms; absent or empty means "runs everywhere". Disable the toggle when the list excludes this client rather than letting a tap fail.
  - `isAuthenticated` is present only when applicable; `authentication_required && !isAuthenticated` means the user must sign in before the plugin can work.
  - `status: "connected"` means already connected — show connected state, never a fresh "connect" prompt.
- **`response_id` on `text-stream` (all clients — read this before touching chat rendering)**

  Every `text-stream` frame now carries `response_id` (e.g. `"resp_3"`), monotonic per session.

  A **response** is one unit of assistant output: a stream of chunk frames terminated by exactly one `end: true`. Until now a turn produced exactly one response, so clients could safely treat *"the turn is over"* and *"this reply is over"* as the same event. **That equivalence is going away.**

  The server now runs multi-step work **concurrently with the conversation** — the user can keep talking while a task runs, and the agent reports back when it finishes. A session therefore has **more than one response in flight**: an acknowledgement, anything said while the work runs, then the task's own result. `end: true` terminates **that `response_id`**, not the turn.

  Concretely, one "play a song, take a selfie, text Sneha" request now looks like:

  ```
  resp_7  "On it — I'll let you know as soon as it's done."   end:true   ← turn closes here
  resp_8  "Kya kar rahi ho?" → "Bas isi pe lagi hoon!"        end:true   ← user talked mid-task
  resp_9  "Song chal raha hai, selfie bhi le li — Sneha ko    end:true   ← the task, reporting back
           message bhej diya hai, delivery confirm nahi kar
           paayi."
  ```

  `resp_9` arrives with **no user message before it**. That is rule 3, and it is the whole point.

  **Client rules**
  1. Key the assistant bubble on `response_id`. Chunks with a new id open a **new** bubble; do not append them to the bubble you were streaming into.
  2. `end: true` finalizes **only** the bubble for that `response_id`. Never use it to clear global "is the assistant replying" state — another response may still be streaming.
  3. A frame may arrive with **no preceding user message** (a task finishing, a proactive update). Render it as a normal assistant bubble; don't require a pending user turn.
  4. Ignore ordering assumptions between different `response_id`s. Within one id, chunks are ordered.
  5. `finish_reason` remains per-response.

  **Backwards compatible — by gating, not by luck.** Concurrency is enabled **per client source**. A client that has not adopted the rules above is never given more than one response per turn, so it behaves exactly as before. Nothing breaks by waiting.

  **Opting in.** Implement the five rules, then either ask for your `client_source` to be added server-side (`DETACHED_PLAN_EXECUTION_SOURCES`) or send the flag yourself in `initial_config` (or any later payload — it is read per turn, so it can be toggled mid-session):

  ```json
  { "multi_response": true }
  ```

  `multi_response: false` opts back out at any time; `supports_multi_response` and `multiResponse` are accepted spellings. Adopted and verified live: **ava_android**, **ava_desktop** — both are on by default server-side, so those clients need not send the flag at all. Pending: vidya, web.

  **How to verify you adopted it correctly.** Ask for something with two or three device actions in it. You should see the acknowledgement bubble finalize, be able to type or speak *while the work runs* and get an answer, and then see a **separate** final bubble with the outcome. Failure modes to look for: the result overwriting the acknowledgement (rule 1), the result never appearing (rule 2 — you cleared global replying state on the first `end`), or the result being dropped for having no user message (rule 3).

  See `V2/AGENTIC_CONCURRENCY_ROADMAP.md` for why this exists and what lands next.

- `ack_disconnect`, `ack_mute_speech`
- `ack_additional_current_context` — `{ type, status: "ok", has_context: boolean, length: number }`
- `ack_user_details` — `{ type, status: "ok", fields: string[], has_location: boolean }` — reply to `update_user_details`
- `self_disconnect` — server-initiated timeout
- `tool-calls` — `{ type: "tool-calls", tools: <toolCall> }` emitted by agent when a tool should run client-side
- `pong` — `{ type: "pong", client_ts: <echo>, server_ts: <epoch_ms> }` — reply to application-level `ping`
- `ack_presentation_ready` — `{ type: "ack_presentation_ready", status: "ok" }` — sent in response to `presentation_ready`
- `rolling_summary` — `{ type: "rolling_summary", summary, foundation, turn_count, timestamp }` — emitted after every N turns once the conversation compacts. Client should persist (keyed by persona ID, up to 3 entries FIFO) and send back as `pre_chat_context` on the next connect.
- `turn_suggestions` — `{ type: "turn_suggestions", suggestions: string[] }` — exactly three follow-up prompts, generated in the background after a persisted assistant turn. Render as tappable chips. Not emitted for a detached plan's acknowledgement ("on it, I'll let you know") — the real ones arrive with the plan's result a moment later. Generation failures are silent: no message, no error.
- `dashboard_init_response` — `{ type: "dashboard_init_response", thought_of_the_day: string, suggestions: string[3], error?: string }` — reply to `dashboard_init`. On failure the server still sends usable fallback content **plus** `error`, so render the payload either way rather than branching on `error`.
- `refresh_plugins_success` — `{ type: "refresh_plugins_success", status?: "success", message?: string, toolsCount: number, toolsDisabled?: true }` — reply to `refresh_plugins`.
- `refresh_plugins_error` — `{ type: "refresh_plugins_error", status: "error", message: string }` — the plugin set is unchanged; the session keeps the tools it already had.
- `speaker_identified` — `{ type: "speaker_identified", speakerId: "speaker_0", name: "Ravi Sharma", designation: string | null }` — ambient mode only. Emitted when a diarization label is auto-mapped to the next unassigned entry in the `meeting_participants` roster (appearance order). Use it to confirm or correct the mapping in a meeting UI; the mapping sticks for the rest of the session.

## Returning a plugin's result (READ THIS if your plugin produces an artifact)

Some plugins produce something — a photo, a scan, a file, a screenshot — and the natural thing to do is post it back into the conversation as an ordinary message. That is fine and supported. **But the artifact has to reach the server.**

### Why it matters

A device action the server dispatches but cannot verify is recorded as `dispatched_unconfirmed`: *"I sent it to your device, I can't confirm it worked."* That is deliberate — the server refuses to claim an outcome it has no evidence for. The **artifact is the evidence.** When it arrives, the server resolves that action to `confirmed`, and every downstream surface changes accordingly: the closing summary says it's done, a later "did that go through?" answers correctly, and any reply made while the work runs stops hedging.

If the artifact never arrives, none of that happens — and it is not a bug. The assistant saying *"I couldn't confirm the selfie"* while the photo sits in your chat window means **the photo was rendered locally and never sent to the server.**

### The contract

- Attach the artifact to the message you post back. For an image, `image` as base64 (without the data-URI prefix) on the same `get_chat_completion` payload that carries the caption. A binary frame works too.
- **Send it every time**, not only when convenient. Intermittent delivery is worse than none: the assistant confirms the action on some runs and hedges on others, for reasons the user cannot see.
- The caption alone is not evidence. A message reading "Captured the moment" with no attachment tells the server a user spoke, not that a capture succeeded.
- Late is fine. The server resolves the action whenever the artifact lands — a different turn, several seconds later, any route. It does not need to arrive in the same frame as the caption, and the server will not block waiting for it.

### How to check

Ask for something that produces an artifact. In the server log you want:

```
dispatch_confirmed_by_inbound_artifact   source: 'chat_message'
```

If that line is absent, the artifact did not reach the server, and the assistant will correctly report the action as unconfirmed no matter what your UI is showing.

## Rendering a running plan

With `multi_response` enabled a multi-step request unfolds over several responses, and a step has more states than "running" and "finished".

| State | What it means | What the user needs to see |
|---|---|---|
| executing | the step is running | progress |
| done | confirmed, or dispatched with its artifact returned | success |
| **waiting on you** | the step asked a question and **cannot proceed until it is answered** | the question, prominently |
| failed | it did not happen | plainly, without softening |

**"Waiting on you" is the one that matters.** It is not a slower kind of executing — nothing will happen until the user replies. Showing it as "Executing" is actively misleading: the user watches a spinner for a step that is waiting on *them*. If your task card has two states, this is the one to add.

The question itself arrives as a normal assistant response (its own `response_id`) and carries the options inline. If the user says something else in the meantime, the server re-asks it rather than dropping it — so expect the same question to legitimately appear more than once, and key any de-duplication on `response_id`, not on message text.

Steps whose dependency failed are reported as **not done**, not silently omitted. If your UI hides them, the user sees "1 of 4 succeeded" as an unqualified success.

## User config reference (patchable via `initial_config`)

The server maintains a per-session `userConfig`. Send the fields you care about; unspecified fields use defaults.

| Key                                     | Meaning                                                                      | Common values / notes                                                  |
| --------------------------------------- | ---------------------------------------------------------------------------- | ---------------------------------------------------------------------- |
| `system_prompt`                         | System prompt seed                                                           | Defaults to a friendly AvA persona                                     |
| `additional_current_context`            | Sticky client-provided latest app/user state appended to instructions        | Example: cart, selected product, checkout step; empty string clears it |
| `lang`                                  | Preferred client-facing STT language key                                     | `en-IN`, `hi-IN`, `unknown`; normalized server-side to `stt_lang`      |
| `tts_lang`                              | TTS language                                                                 | Matches voice; defaults to `en-IN`                                     |
| `voice_type`                            | `non-custom` or `custom`                                                     | `custom` routes to Cartesia; `tts_provider` overrides too              |
| `voice_id`                              | Voice selection                                                              | e.g., `en-IN-Chirp3-HD-Leda` or Cartesia voice IDs                     |
| `tts_provider`                          | `google` (Python WS) or `cartesia`                                           | `cartesia` also used when `voice_type=custom`                          |
| `tts_style_prompt`                      | Optional persona voice direction for Gemini TTS                              | e.g. "Speak in a soft, intimate tone"; ≤100 bytes (word-safe truncated) |
| `stt_provider`                          | STT provider                                                                 | `microsoft`, `sarvam`, `deepgram`, `openai_realtime`. **Send it explicitly** — the default is a server/env setting (`DEFAULT_STT_PROVIDER`), not a stable part of the API |
| `modalities`                            | Array of enabled outputs                                                     | `['text', 'audio']`                                                    |
| `use_vad`                               | Enable VAD on incoming audio                                                 | `true`/`false`                                                         |
| `vad_config`                            | VAD tuning. `threshold`: Silero speech probability. `minSpeechDuration` (s): sustained speech required to trigger barge-in **while the agent is speaking** (clamped 0.12–2.0s). `minSilenceDuration` (s): silence that re-arms the barge-in latch (clamped 0.2–3.0s). | `{ threshold: 0.75, minSpeechDuration: 0.2, minSilenceDuration: 0.4 }` |
| `receive_tts_chunks`                    | If true, server sends PCM frames (`speech_chunk`, base64 24kHz) instead of injecting into the WebRTC audio track. **Warning: audio you play yourself (e.g. Web Audio API) is invisible to the browser/OS echo canceller — on speaker setups the mic will pick the agent's voice back up. Keep this `false` unless you implement your own AEC-safe playback path.** | bool (default `false`)                                                 |
| `use_openai_realtime`                   | Route generation to OpenAI Realtime instead of internal agent                | bool                                                                   |
| `use_groq`, `groq_model`                | Groq routing flags                                                           | optional                                                               |
| `max_completion_tokens`, `temperature`  | LLM params                                                                   | numbers                                                                |
| `curr_user_token`                       | **Required.** Auth token for the session (see Authentication section)        | Alias: `api_key` also accepted                                         |
| `personaId`                             | Optional persona ID to fetch profile from API                                | Alias: `persona_id` also accepted; omit to use AvA defaults            |
| `client_source`                         | Optional client identifier                                                   | Default `"web"`. Values: `web`, `web_agent`, `ava_android`, etc.       |
| `agent_data`                            | Optional full persona + tools payload (pass-through if you have the encrypted blob) | When present, server uses it directly after validating token  |
| `start_prompt`                          | Auto-run a prompt after session create                                       | string — automatically overridden for returning users when `pre_chat_context` is present |
| `pre_chat_context`                      | Prior-session rolling summaries to restore on reconnect                      | Array of `{ summary, foundation, turn_count, timestamp }` objects sent in `initial_config` |
| `notification_context`                  | Simulated notification message clicked by the user to start/resume session   | Object containing `{ assistant_message, assistant_name, timestamp }` |
| `noise_supression`, `echo_cancellation` | **Server-side** DSP toggles (RNNoise denoise / NLMS echo filter on inbound mic audio). These do NOT replace client-side echo cancellation — real AEC must run on the device, where the speaker signal is known. See "Echo cancellation is a client requirement". | bool                                                                   |
| `mute_speech`                           | Soft-mute server TTS                                                         | bool                                                                   |
| `ephemeral_session`                     | **Shared-terminal mode.** Nothing this session says is read from or written to the persona's stored chat history — the history fetch is skipped at connect and persistence is refused. Persona, voice, tools and plugins are unaffected. Use on public kiosks/demo booths where the speaker is not the account holder. | bool or `"true"` (default `false`). Aliases: `ephemeral`, `no_chat_history`, `disable_chat_persistence` |
| `game_mode`                             | Isolate session for game play (disables persona plugins, restricts tools to client-supplied `tools`) | bool (default `false`)                                                 |
| `disable_tool` / `disable_plugin`       | Load persona + chat history but **no plugins/tools** — session runs a single lowest-latency fast-chat path (no validator, no JIT plugin discovery, no consent gate). Vision still works. Set once in `initial_config`. | bool or `"true"` (default `false`). Aliases `disable_tools` / `disable_plugins` also accepted |
| `plugin_search`                         | Enable/disable JIT plugin discovery (marketplace search + connect cards) for the session. Also accepted on any later message to toggle mid-session. | bool or `"true"`/`"false"` — default depends on `client_source` (ON for `ava_android`/`vidya_android`/`ava_desktop`, OFF for `web`/`web_agent`). Aliases: `pluginSearch`, `plugin_search_enabled`, `jit_plugin_search`, `enable_plugin_search` |
| `tools`                                 | Client-supplied custom tools (OpenAI function format) used when `game_mode` is true | Array of tool objects                                                  |
| `ambient_intelligence`                  | Enable ambient conversational mode (see below)                               | `false` (default)                                                      |
| `ambient_meeting_context`               | Short description of the meeting/call type fed to the engagement gate         | e.g. `"weekly sales team standup"` — optional                          |
| `ambient_trigger_names`                 | Reserved; currently unused — engagement gate uses self-identity from the persona system prompt instead of keyword lists | `[]` — kept for forward compatibility |
| `ambient_mode_config`                   | Per-session ambient tuning overrides (all fields optional, merged with server env defaults) | `{ "stt_provider": "deepgram", "silence_gap_ms": 400, "question_gap_ms": 150, "context_window_turns": 8, "max_tokens": 150 }` |
| `meeting_participants`                  | Participant roster for Deepgram diarization label resolution                  | `[{ "name": "Ravi Sharma", "designation": "CTO", "speakerId": "speaker_0" }]` — `speakerId` optional; auto-mapped by first-speech order when omitted |
| `capability_notes`                      | Free-text hints about client-side features injected into the ambient gate prompt | e.g. `"You can present URLs and PDFs as a screen share."` |

### STT configuration

Preferred client contract for speech recognition:

- `stt_provider`: STT backend to use — `microsoft`, `sarvam`, `deepgram`, or `openai_realtime`.
- `lang`: STT language.

**Send `stt_provider` explicitly.** When you omit it the session takes the server's `DEFAULT_STT_PROVIDER` (code default `microsoft`; the current deployment sets `deepgram`), which is an operational setting and can change under you without any client-visible API change.

Behavior by provider:

- `sarvam`: `lang` is optional. If omitted, the server lets Sarvam auto-detect the spoken language. You can also send `lang: "unknown"` to force auto-detection explicitly. Sarvam additionally runs a turn endpointer that merges a continued utterance into one turn instead of splitting it.
- `microsoft`: send `lang` explicitly, for example `en-IN`.
- `deepgram`: send `lang` explicitly. This is the provider that supports **speaker diarization**, so ambient/meeting sessions default to it.
- `openai_realtime`: send `lang` explicitly for predictable behavior.

Compatibility notes:

- The server still accepts legacy `stt_lang` in `initial_config`.
- The server also accepts legacy `language` in `update_stt_language`.

Examples:

```json
{
  "type": "initial_config",
  "value": {
    "stt_provider": "sarvam"
  }
}
```

```json
{
  "type": "initial_config",
  "value": {
    "stt_provider": "sarvam",
    "lang": "hi-IN"
  }
}
```

```json
{
  "type": "initial_config",
  "value": {
    "stt_provider": "microsoft",
    "lang": "en-IN"
  }
}
```

`agent_data` fields used by persona-aware flows:

- `curr_user_token`, `persona_id`, `persona_value`, `persona_languages`
- `tools_config`, `main_agent_tools`
- `legacy_tools`
- `is_expressive_persona` (enables richer prosody)
- `chat_history` (seeded to system prompt)

### Legacy built-in tools via `agent_data.legacy_tools`

Use `agent_data.legacy_tools` when you want the model provider to execute supported built-in tools directly instead of the server emitting client-side `tool-calls`.

- Supported values today: `browser_search`, `code_interpreter`
- Alias accepted on input: `code_execution` normalizes to `code_interpreter`
- Provider restriction: only used on Groq-backed GPT-OSS legacy-tool paths
- Precedence rule: legacy built-ins are attached only when no normal function tools are active for that turn
- Client contract: these tools do not generate `tool-calls`; instead the client may receive `tool-activity` events

Example:

```json
{
  "type": "initial_config",
  "value": {
    "use_groq": true,
    "groq_model": "openai/gpt-oss-120b",
    "agent_data": {
      "legacy_tools": ["browser_search", "code_execution"]
    }
  }
}
```

Behavior notes:

- If `main_agent_tools` or persona-provided function tools are present, those function tools take precedence and `legacy_tools` are ignored for that turn.
- If the selected model does not support a requested legacy tool, unsupported entries are dropped.
- Browser-search turns may emit an early `tool-activity` event with `phase: "requested"` and a conversational `status_text` before the final answer is ready.
- The keep-alive status phrase is spoken server-side but is not sent as normal `text-stream` assistant content.

## Recommended flows

### Additional current context (important for integration)

`additional_current_context` is the canonical way for clients to send current app state (cart, selected item, active screen, checkout step, permissions state, etc.) to the server-side agent.

- Scope: per WebSocket session (sticky until replaced/cleared).
- Semantics: **replace**, not merge. Sending a new value overwrites the previous context.
- Clear behavior: send empty string `""` (or non-string, which normalizes to empty) to clear.
- Limit: server trims whitespace and caps length to `MAX_ADDITIONAL_CURRENT_CONTEXT_CHARS` (default `8000`). Over-cap values are truncated at a word boundary and a `additional_current_context_truncated` warning is logged (originalLength/cappedLength).
- Prompt wiring: appended to the system instructions for direct agent and orchestrator paths; also re-applied to OpenAI Realtime session instructions when that mode is active.

Ways to send it:

1. In `initial_config`:

```json
{
  "type": "initial_config",
  "value": {
    "client_source": "web_agent",
    "additional_current_context": "cart_total=1499; currency=INR; selected_sku=ABC123"
  }
}
```

2. Mid-session update (recommended):

```json
{
  "type": "update_additional_current_context",
  "additional_current_context": "cart_total=1999; currency=INR; selected_sku=XYZ789"
}
```

3. Per-turn with `get_chat_completion` (also updates sticky context):

```json
{
  "type": "get_chat_completion",
  "text": "Can I apply available coupons?",
  "speech": true,
  "additional_current_context": "cart_total=1999; coupon_screen=true"
}
```

Server ack after context update:

```json
{
  "type": "ack_additional_current_context",
  "status": "ok",
  "has_context": true,
  "length": 47
}
```

Client best practices:

- Keep it compact and factual (state, IDs, flags), avoid long prose.
- Update only when state changes materially.
- Prefer stable keys (`selected_sku`, `cart_total`, `step`, `locale`) for predictable behavior.
- Do not put secrets/PII unless required; this context is used in model instructions.

### Server defaults (env)

- `DEFAULT_STT_PROVIDER`: STT provider used when the client omits `stt_provider`. Code default `microsoft`; the current deployment sets `deepgram`. Supported values: `microsoft`, `sarvam`, `deepgram`, `openai_realtime`. Treat it as deployment configuration, not API surface — clients that care should send `stt_provider`.

### Speech-to-speech (WebRTC-first)

1. Connect WS → send `initial_config` with `modalities: ['audio','text']`, `stt_provider` if you want to override the server default, and `lang` when you are not relying on Sarvam auto-detect.
2. WebRTC negotiate (`offer`/`answer`/`candidate`).
3. Start sending microphone audio on the WebRTC track (captured per the AEC requirements below). VAD runs server-side and feeds the configured STT provider + LLM.
4. Listen for `text-stream` for transcripts and `ai_speech_start` / `speech_end` for synthesized speech events.
5. Keep `receive_tts_chunks=false`; the server injects TTS into the WebRTC audio track, which is the only playback path the platform echo canceller can subtract. Setting it to true delivers PCM chunks you must play yourself — outside the echo-cancelled path.

### Echo cancellation is a client requirement

The agent's voice comes out of the device speaker and back into the device mic. The only place that loop can be cancelled properly is **on the device**, where the OS knows exactly what is being played. The server adds echo-resistant barge-in gating and self-echo transcript filtering as a safety net, but without client AEC a speakerphone setup will degrade (delayed barge-in commits, occasional swallowed first words). This is the same contract Gemini Live and OpenAI Realtime rely on.

**Browser (web) clients:**

```js
// 1. Capture the mic WITH processing enabled (do not disable these):
const mic = await navigator.mediaDevices.getUserMedia({
  audio: {
    echoCancellation: true,   // REQUIRED — cancels the agent's own voice
    noiseSuppression: true,
    autoGainControl: true,
    channelCount: 1,
  },
});
pc.addTrack(mic.getAudioTracks()[0], mic);

// 2. Play the agent's audio via the remote WebRTC track and a media element.
//    Do NOT decode/play agent audio through the Web Audio API
//    (AudioContext/AudioWorklet) — Chromium's AEC cannot see that path and
//    will not cancel it.
pc.ontrack = (ev) => {
  const audioEl = document.getElementById("agent-audio"); // <audio autoplay>
  audioEl.srcObject = ev.streams[0];
};
```

**Android (native):** record with `MediaRecorder.AudioSource.VOICE_COMMUNICATION` (platform AEC path), or attach `AcousticEchoCanceler` to the `AudioRecord` session. Play agent audio through the same audio session (`STREAM_VOICE_CALL` / WebRTC audio device module).

**iOS (native):** use `AVAudioSession` category `.playAndRecord` with mode `.voiceChat` (enables VoiceProcessingIO AEC), and keep agent playback inside that session.

**Rules of thumb:**

- Platform AEC only cancels audio it knows it is playing. Any playback path that bypasses the platform voice pipeline (raw PCM through your own audio graph) escapes cancellation.
- Headphones eliminate the loop entirely; speakerphone is the worst case — test there.
- If you must use `receive_tts_chunks=true` in a browser, route your playback through a loopback `RTCPeerConnection` and a media element so the AEC gets a reference signal.

### Text or text+vision to speech

1. Connect and send `initial_config` (may set `modalities` to `['audio','text']`).
2. If you have an image, send `snapshot` (binary) first, or include `image` Base64 in `get_chat_completion`.
3. Send `get_chat_completion` with `speech:true`.
4. Read `text-stream` and `ai_speech_start` / `speech_end`.

### Tool calling

1. Ensure your persona/tools are configured (via persona fetch or `agent_data.tools_config`).
2. When the server emits `tool-calls`, execute the described tool client-side.
3. Reply with `tool-result` including the `toolCallId` and structured `result`.
4. The server will resume the turn and continue streaming text/tts.

### Legacy built-in tool flow

1. Send `agent_data.legacy_tools` in `initial_config` and use a supported Groq model such as `openai/gpt-oss-120b`.
2. Send a normal `get_chat_completion` request.
3. If the provider-backed tool path is used, the client may receive `tool-activity` with `phase: "requested"` immediately.
4. Wait for the normal assistant response on `text-stream` / speech output.
5. Optionally handle a later `tool-activity` event with `phase: "completed"` when the server confirms usage from the provider response.

### Isolated Game Mode with Custom Tools

Use this flow to build playable games or sandbox sessions where you want to retain the assistant's voice and personality but isolate the toolset to only game-specific actions (e.g. check a visual riddle or register a score), bypassing standard persona plugins (like messaging or system apps).

1. In `initial_config`, set `game_mode: true` and pass your custom tools array:

```json
{
  "type": "initial_config",
  "value": {
    "curr_user_token": "your-jwt",
    "personaId": "your-persona-id",
    "game_mode": true,
    "tools": [
      {
        "type": "function",
        "function": {
          "name": "check_game_item",
          "description": "Evaluate if the physical item shown in the user's camera snapshot matches the target.",
          "parameters": {
            "type": "object",
            "properties": {
              "item_detected": { "type": "string" },
              "closeness": { "type": "string", "enum": ["cold", "warm", "hot", "correct"] },
              "verbal_hint": { "type": "string" }
            },
            "required": ["item_detected", "closeness", "verbal_hint"]
          }
        }
      }
    ]
  }
}
```

2. The server loads the persona's voice, prompt guidelines, and details, but overrides and restricts the active tools exclusively to the custom tools list.
3. Handle execution using the standard **Tool Calling Flow** (listening for `tool-calls` and responding with `tool-result`).

### Dev logging

- Send `start_dev_log` to subscribe to `dev_log` messages (per-session). Use this during integration; turn off with `end_dev_log`.

---

## Ambient Conversational Intelligence

Ambient mode changes the AI from reactive (it only responds when directly addressed) to ambient (it attends the full conversation and autonomously decides when — or whether — to speak).

### What it does

In normal mode the server responds to every finalized STT utterance. In ambient mode each utterance passes through a fast engagement gate before any LLM call is made:

1. **Backchannel filter** — before the LLM gate runs, a local classifier checks if the utterance is a passive acknowledgment (*"hmm"*, *"ok"*, *"right"*, *"haan"*) or continuation signal (*"go on"*, *"please continue"*). While the AI is speaking, these are dropped immediately so the AI is never interrupted by filler sounds.
2. **Transcript buffer** — the last 15 utterances are kept in a rolling window. When Deepgram diarization is active, utterances are labeled by speaker (`[speaker_0]`, `[speaker_1]`, etc.) and resolved to human-readable names if a `meeting_participants` roster is provided. AI responses are recorded as `[ai]`.
3. **Engagement gate** — a sub-300 ms Groq call evaluates the conversation using the AI's own persona system prompt as its identity (self-identity approach — no keyword lists). It returns one of four decisions:
   - `WAIT` — the AI stays silent. No generation is triggered.
   - `SPEAK` — the AI waits for a configurable silence gap (~900 ms) then speaks. Shorter gap (~450 ms) when urgency is high (urgency ≥ 7).
   - `SPEAK_DEFERRED` — the AI wants to speak but is currently mid-TTS. The generation is queued and fires automatically after the current `speech_end`, so the AI never talks over itself.
   - `INTERRUPT` — the AI fires immediately and prepends a natural softening phrase (e.g. *"Hold on, that's worth clarifying — "*). Requires urgency ≥ 9 while the AI is already speaking.
4. **Barge-in cancellation** — if new speech arrives while a SPEAK gap is pending, the timer is cancelled. The AI won't cut in mid-sentence.

The gate is **WAIT-biased by default**. It derives its own identity (name, domain, role) from the persona system prompt and decides autonomously. It speaks when there is a clear signal: direct name mention, unanswered open question in the window, or a high-urgency domain-relevant contribution (corrections, critical risks). Below urgency 4, even a `SPEAK` decision is suppressed.

### When to use it

- Meeting copilot or call-center supervisor scenario where the AI should listen and only jump in with value.
- Any experience where an automatic reply to every utterance would feel intrusive.

### How to enable

Add `ambient_intelligence: true` to your `initial_config`. Everything else is optional.

```json
{
  "type": "initial_config",
  "value": {
    "modalities": ["audio", "text"],
    "stt_provider": "sarvam",
    "ambient_intelligence": true,
    "ambient_meeting_context": "customer support call",
    "ambient_trigger_names": ["hey ava", "ava can you"]
  }
}
```

| Field                    | Required | Description                                                                           |
| ------------------------ | -------- | ------------------------------------------------------------------------------------- |
| `ambient_intelligence`   | Yes      | `true` to activate. `false` (default) = normal reactive mode.                        |
| `ambient_meeting_context`| No       | One-line description of the meeting type. Improves gate accuracy for domain matching. |
| `ambient_trigger_names`  | No       | Reserved; currently unused. The gate uses the persona's own identity for self-identification rather than keyword matching. |
| `ambient_mode_config`    | No       | Per-session ambient tuning overrides — all fields optional, merged with server env defaults. See sub-fields below. |

**`ambient_mode_config` sub-fields:**

| Sub-field               | Type   | Description                                                                 | Env var | Code default | Deployed today |
| ----------------------- | ------ | --------------------------------------------------------------------------- | ------- | ------------ | -------------- |
| `stt_provider`          | string | STT provider for ambient mode — `deepgram` is the one that diarizes          | —       | `deepgram`   | `deepgram`     |
| `silence_gap_ms`        | number | Silence pause (ms) before a SPEAK decision fires                            | `AMBIENT_SILENCE_GAP_MS` | `900` | `400` |
| `question_gap_ms`       | number | Shorter gap (ms) used when urgency ≥ 7 (direct questions / urgent signals)  | `AMBIENT_QUESTION_GAP_MS` | `450` | `150` |
| `context_window_turns`  | number | Rolling transcript window size fed to the engagement gate                   | `AMBIENT_CONTEXT_WINDOW_TURNS` | `15` | `8` |
| `max_tokens`            | number | Max output tokens for ambient voice generation. `0` / omit = session default | `AMBIENT_MAX_TOKENS` | `0` | `150` |

The last two columns differ on purpose: the deployment is tuned tighter than the code defaults for latency. **Neither is a contract** — if your experience depends on a specific gap, send `ambient_mode_config` and pin it per session.

Example — tighter timing for a fast-paced demo:

```json
{
  "type": "initial_config",
  "value": {
    "ambient_intelligence": true,
    "ambient_mode_config": {
      "stt_provider": "deepgram",
      "silence_gap_ms": 300,
      "question_gap_ms": 100,
      "context_window_turns": 6,
      "max_tokens": 120
    }
  }
}
```

| `meeting_participants`   | No       | Participant roster for diarization label resolution. Format: `[{ "name": "...", "designation": "...", "speakerId": "speaker_0" }]`. `speakerId` is optional — entries without one are auto-mapped in first-speech order. |
| `capability_notes`       | No       | Free-text capability hints injected into the gate prompt. Use to inform the gate about client features (e.g. screen sharing) it should consider when deciding to speak. |

### Client behaviour differences

| Aspect                 | Normal mode                        | Ambient mode                                                        |
| ---------------------- | ---------------------------------- | ------------------------------------------------------------------- |
| Response rate          | Every STT final → generation       | Only when gate decides SPEAK, SPEAK_DEFERRED, or INTERRUPT          |
| Response timing        | Immediate after STT final          | SPEAK: after 900 ms silence gap. SPEAK_DEFERRED: fires after current `speech_end`. INTERRUPT: immediate with prefix. |
| `text-stream` / speech | Always emitted                     | Only emitted on SPEAK/SPEAK_DEFERRED/INTERRUPT decisions            |
| `ai_speech_start`      | Always                             | Only on decided turns                                               |
| `stop_speech`          | Cancels current TTS                | Also cancels any pending gap timer and any queued SPEAK_DEFERRED     |

The WebSocket message types you already handle (`text-stream`, `ai_speech_start`, `speech_end`, `tool-calls`, `stop_speech`, `interrupt_operation`) work exactly the same. No new client-side message types are needed to use ambient mode.

### Engagement gate signals

The gate uses a **self-identity** approach — the AI's full persona system prompt is passed to the gate model as its identity. The model derives its own name, domain, and role from that prompt and decides autonomously whether to speak. No keyword lists or name-matching heuristics are used.

Inputs to the gate decision (all derived server-side — nothing extra from the client):

- The rolling transcript window (last 15 utterances, with speaker labels when diarization is active)
- Whether the AI is currently playing TTS audio (affects urgency thresholds — INTERRUPT requires urgency ≥ 9 while speaking; SPEAK requires urgency ≥ 8, with urgency 5–7 producing a `SPEAK_DEFERRED` instead)
- The `ambient_meeting_context` you supply and the persona's system prompt
- Any `capability_notes` you injected (client-side feature awareness)

### Speaker diarization

When `ambient_mode_config.stt_provider` is `"deepgram"` (the default for ambient mode), Deepgram assigns speaker labels (`speaker_0`, `speaker_1`, …) sequentially by first-speech appearance. These labels are resolved to human-readable names via the `meeting_participants` roster when provided, and are visible in the transcript window seen by the gate.

You can update the roster mid-session without reconnecting using the `update_meeting_participants` message.

### Interruption phrase

When the gate decides `INTERRUPT`, the AI's spoken response is automatically prefixed with a randomised softening phrase before the persona's answer. The prefix appears at the start of the `text-stream` content so you can display it in a transcript if needed.

Examples: *"Actually, let me add something here — "*, *"Hold on, that's worth clarifying — "*

### Server-side tuning (env vars)

For integration testing you can adjust gate behaviour via environment variables:

| Variable                      | Code default | Description                                            |
| ----------------------------- | ------------ | ------------------------------------------------------ |
| `AMBIENT_SILENCE_GAP_MS`      | `900`   | Silence gap before SPEAK fires (ms). Deployed: `400`        |
| `AMBIENT_QUESTION_GAP_MS`     | `450`   | Shorter gap for urgent/question signals (ms). Deployed: `150` |
| `AMBIENT_MIN_URGENCY_TO_SPEAK`| `4`     | Urgency score (1–10) below which SPEAK is suppressed   |
| `AMBIENT_INTERRUPT_ENABLED`   | `true`  | Set `false` to disable INTERRUPT action server-wide    |
| `AMBIENT_CONTEXT_WINDOW_TURNS`| `15`    | Number of recent utterances kept for gate context. Deployed: `8` |
| `AMBIENT_ENGAGEMENT_MODEL`    | `openai/gpt-oss-20b` | Groq model used for the engagement gate  |

---

## Cross-Session Memory & Rolling Summaries

AvA maintains a rolling summary of each conversation and carries that memory across reconnects. A returning user receives a personalised greeting that references their prior session — not a generic "Hello, how can I assist?"

### How it works

1. **Rolling summary generation** — after every N turns (default: 6) the server condenses the conversation into a compact summary (≤ 512 tokens). This summary grows with the session and is continuously refreshed.
2. **`rolling_summary` event** — once a summary is generated the server emits it as a WebSocket message. Your client should persist it (e.g. in `localStorage`) keyed by persona ID.
3. **On next connect** — when the user reconnects, send the stored summaries in `pre_chat_context` inside `initial_config`. The server injects them into the model's system prompt as `SESSION MEMORY` and overrides the opening `start_prompt` so Ava greets the user by name and references prior conversations naturally.

### Server → client: `rolling_summary`

```json
{
  "type": "rolling_summary",
  "summary": "User is Priya Nair, mobile app PM. Discussed feature prioritisation for Q3 sprint — delay widget + dark mode. Prefers bullet points. Next step: share mockup.",
  "foundation": "Priya Nair is a mobile app PM focused on Q3 sprint planning.",
  "turn_count": 8,
  "timestamp": 1718000000000
}
```

| Field | Description |
| ----------- | ----------- |
| `summary` | Full rolling summary (up to ~500 tokens) |
| `foundation` | Condensed core facts — kept stable across repeated summarisation cycles |
| `turn_count` | Number of turns covered by this summary |
| `timestamp` | Unix epoch ms when the summary was generated |

Store up to 3 entries per persona (FIFO). The oldest is dropped when a fourth arrives.

### Client → server: `pre_chat_context` in `initial_config`

On the next session connect, read the persisted summaries and send them in `initial_config`:

```json
{
  "type": "initial_config",
  "value": {
    "modalities": ["audio", "text"],
    "stt_provider": "sarvam",
    "agent_data": {
      "persona_id": "my-persona-id",
      "curr_user_token": "user-jwt"
    },
    "pre_chat_context": [
      {
        "summary": "User is Priya Nair, mobile app PM. Discussed feature prioritisation for Q3 sprint...",
        "foundation": "Priya Nair is a mobile app PM focused on Q3 sprint planning.",
        "turn_count": 8,
        "timestamp": 1718000000000
      }
    ]
  }
}
```

`pre_chat_context` is an array ordered oldest-first. The server merges all entries into a single `SESSION MEMORY` block and injects it into every LLM system prompt for the session.

### Simulated Notification Entry Point (`notification_context`)

In scenarios where a conversation starts because a user clicks a simulated notification (e.g. a message from assistant Sam: *"yoo supp rohan , remember we talked about burger yesterday , r u there?"*), the client can pass a `notification_context` object in `initial_config`:

```json
{
  "type": "initial_config",
  "value": {
    "modalities": ["audio", "text"],
    "agent_data": {
      "persona_id": "my-persona-id",
      "curr_user_token": "user-jwt"
    },
    "notification_context": {
      "assistant_message": "yoo supp rohan , remember we talked about burger yesterday , r u there?",
      "assistant_name": "sam",
      "timestamp": 1718000000000
    }
  }
}
```

The server will append a `NOTIFICATION ENTRY POINT` instruction to the `SESSION MEMORY` informing the assistant about the notification message clicked by the user, and will automatically direct the assistant to continue contextually from that message.

### Returning-user greeting

When `pre_chat_context` or `notification_context` is present, the server automatically replaces the default `start_prompt` with a context-aware instruction:

> "A returning user has reconnected. Their previous session context is in SESSION MEMORY above — read it before responding. Greet them warmly by name if their name is mentioned in the context. Reference something specific and relevant from prior conversations naturally. Do NOT say this is their first call. Do NOT introduce yourself from scratch."

If `notification_context` is present, the instruction also appends:
> "... Crucially, the conversation started via the notification message in NOTIFICATION ENTRY POINT. Be fully aware of that message and continue contextually."

This ensures the model does not fall back to a generic greeting even if the default `start_prompt` would normally trigger a fresh introduction.

### Recommended client-side localStorage pattern

```js
const MEMORY_KEY = `ava_summaries_${personaId}`;
const MAX_ENTRIES = 3;

// Save on rolling_summary events
ws.addEventListener("message", (evt) => {
  const msg = JSON.parse(evt.data);
  if (msg.type === "rolling_summary") {
    const entries = JSON.parse(localStorage.getItem(MEMORY_KEY) || "[]");
    entries.push({
      summary:    msg.summary,
      foundation: msg.foundation,
      turn_count: msg.turn_count,
      timestamp:  msg.timestamp,
    });
    if (entries.length > MAX_ENTRIES) entries.shift(); // keep newest MAX_ENTRIES
    localStorage.setItem(MEMORY_KEY, JSON.stringify(entries));
  }
});

// Read on next connect
function buildInitialConfig(personaId, token) {
  const stored = JSON.parse(localStorage.getItem(`ava_summaries_${personaId}`) || "[]");
  return {
    type: "initial_config",
    value: {
      modalities: ["audio", "text"],
      stt_provider: "sarvam",
      agent_data: { persona_id: personaId, curr_user_token: token },
      ...(stored.length > 0 ? { pre_chat_context: stored } : {}),
    },
  };
}
```

### Server-side tuning (env vars)

| Variable | Code default | Deployed today | Description |
| -------- | ------------ | -------------- | ----------- |
| `ROLLING_SUMMARY_TRIGGER_MESSAGES` | `9` | `6` | New transcript messages before the summary is regenerated |
| `ROLLING_SUMMARY_KEEP_LAST_MESSAGES` | `9` | `4` | Most recent messages kept verbatim after compaction |
| `SUMMARY_AGENT_MAX_TOKENS` | `512` | `512` | Max tokens for the summariser model response |

Do not build a client that assumes a fixed cadence — persist whatever `rolling_summary` events arrive and send the newest back.

---

## Payload sizing and limits

- WebSocket max payload: ~5 MB. Prefer binary framing for audio/images to avoid Base64 bloat.
- TTS PCM frames are 24 kHz mono, 16-bit. Server resamples to 48 kHz when injecting into WebRTC.
- Backpressure: when the WS `bufferedAmount` exceeds ~4 MB, non-critical `text-stream`/`speech_chunk` messages may be dropped until congestion clears (final `end:true` always sent).

## Error handling

- `session-error` is emitted when init fails (e.g., missing persona credentials). Client should close and retry.
- `self_disconnect` occurs on inactivity timeout (configurable via env). Reconnect to continue.
- On WebRTC failure, renegotiate by sending a fresh `offer`.

## Minimal client pseudocode

```js
const ws = new WebSocket("wss://avarealtime.pathor.ai/");
ws.onopen = () => {
  ws.send(
    JSON.stringify({
      type: "initial_config",
      value: { modalities: ["audio", "text"], stt_provider: "sarvam" },
    }),
  );
  // send offer next...
};

ws.onmessage = (evt) => {
  const msg = JSON.parse(evt.data);
  switch (msg.type) {
    case "session-created":
      console.log("session", msg.clientId);
      break;
    case "text-stream":
      if (msg.chunk) console.log(msg.chunk);
      if (msg.end) console.log("complete", msg.text);
      break;
    case "tool-calls":
      // run tool, then reply
      ws.send(
        JSON.stringify({
          type: "tool-result",
          toolCallId: msg.tools.id,
          result: { ok: true },
        }),
      );
      break;
  }
};
```

## Operational notes

- Inactivity timeout defaults to `appConfig.inactivityTimeout`; server sends `self_disconnect` then closes.
- For Android clients, provide `agent_data.curr_user_token` and `agent_data.persona_id` to authenticate persona fetch.
- When `use_openai_realtime=true`, the server streams text and (optionally) audio from the OpenAI Realtime endpoint; tool-calling flows may differ based on the configured deployment.
- Chat persistence runs when `personaId` + token are set and `client_source` is `web`/`ava_desktop`/`ava_android`/`web_agent` — **and** the session is not `game_mode` or `ephemeral_session`.
- **Shared terminals must send `ephemeral_session: true`.** Do not try to get isolation by inventing a `client_source` that is off the allow-list: the source also drives auth, plugin OS resolution and the JIT-search default, so an off-list value buys privacy by making the client lie about what it is, and silently starts persisting the day someone corrects it. `exclude_from_chat_history` is not a substitute either — it is per-`get_chat_completion`, so it covers typed turns only and voice turns still persist.
- Default non-vision Groq chat paths now use `openai/gpt-oss-120b`; vision turns continue to use the configured vision model.

## Glossary

- **VAD**: Voice Activity Detection on incoming audio.
- **Snapshot**: Latest image attached to the next LLM turn for visual reasoning.
- **Tool call**: LLM asks the client to execute a function and return data.
- **Dev log**: Server-side diagnostics pushed to the client for debugging.
- **Ambient mode**: Session mode where an engagement gate decides whether the AI speaks after each STT final, rather than always responding.
- **Engagement gate**: Sub-300 ms Groq inference that returns SPEAK / SPEAK_DEFERRED / WAIT / INTERRUPT for a given conversation context. Uses self-identity (persona system prompt) — no keyword lists.
- **Silence gap**: Configurable delay after a SPEAK decision before generation fires — allows natural conversation rhythm.
- **SPEAK_DEFERRED**: Gate decision when the AI wants to respond but is currently mid-TTS. The generation is queued and fires automatically after `speech_end`.
- **Backchannel filter**: Local pre-gate classifier that drops passive acknowledgments (*"hmm"*, *"ok"*, *"right"*) before the LLM gate runs, preventing the AI from interrupting itself when participants give listening cues.
- **Diarization**: Per-speaker labeling of the audio stream. In ambient mode, Deepgram assigns `speaker_0`, `speaker_1`, … labels resolved to participant names via the `meeting_participants` roster.
- **Rolling summary**: Compact (≤ 512 token) distillation of the conversation generated every N turns. `foundation` captures stable core facts (name, role, preferences); `summary` captures the latest narrative. Both survive disconnects via `rolling_summary` events.
- **pre_chat_context**: Array of prior-session rolling summaries sent in `initial_config`. The server merges them into `SESSION MEMORY` in the LLM system prompt and replaces the opening greeting instruction for returning users.
- **SESSION MEMORY**: The merged `pre_chat_context` block injected into every LLM system prompt for the session. Conflicts with live in-session messages are resolved in favour of the live messages.