Build realtime voice agents on one connection

Playground#

A real client, not a mock. Paste a key, hit connect, and you are talking to the same WebSocket and WebRTC endpoints your app will use. Every toggle takes effect on the live session, and the inspector on the right shows the exact frames going over the wire.

Idle Connect Disconnect

—ws rtt

—first token

0responses

0frames

Need a key? Sign up at console.ava.pathor.ai — create an assistant there, or just grab the token and drive it from here with your own system prompt.

Credentials

API key / bearer token *

Stays in this browser. Redacted in the inspector and never sent anywhere but the AvA server.

Remember on this devicelocalStorage — avoid on shared machines

Brain

Where the personality comes from Custom system prompt (no persona) Assistant from the console (personaId)

personaId

Voice, prompt, tools and history all resolve server-side. Everything below is then optional.

system_prompt You are Ava, a warm and quick-witted realtime voice assistant built by PathOr. Keep replies short and conversational — one or two sentences unless asked for detail. Never read out punctuation or emoji.

start_prompt (optional opener)

Speech

stt_provider microsoft sarvam deepgram openai_realtime

lang en-IN en-US hi-IN unknown (auto)

client_source web web_agent ava_desktop ava_android

Drives the defaults for JIT plugin search and concurrent responses.

Behaviour

multi_responseconcurrent responses, keyed by response_id plugin_searchJIT marketplace discovery + consent cards disable_toolfastest path — no tools, no validator ephemeral_sessionnothing here is written to stored history use_vadserver-side voice activity detection ambient_intelligencegate decides whether to speak at all

Barge-in tuning

threshold 0.75

minSpeech 0.20s

Raise minSpeech if room noise keeps interrupting the agent mid-sentence.

Live context

additional_current_context

Pushed with update_additional_current_context the moment you stop typing, while connected.

Copy this initial_config Reset to defaults

Paste a token and hit Connect.
Then talk, or type below — both go through the same session.

Mic off Camera Stop speech × Clear

Capture frame Close camera

Pause Clear

Every JSON frame in both directions lands here — click one to expand it. Binary audio frames are counted, not dumped.

Real session against wss://avarealtime.pathor.ai/. Audio plays through a hidden <audio> element fed by the remote WebRTC track — the AEC-safe path described in Audio & echo cancellation. Use headphones if you are on a speaker, or the agent will hear itself.

Things worth trying#

How it fits together#

Two channels, one session. Knowing which channel carries what removes most of the confusion when you start wiring things up.

A working voice client in five minutes#

Paste this into a page served over HTTPS (or localhost), drop in a token, and you have a full-duplex voice agent. Everything after this section is detail on the pieces used here.

1 · The page#

<!-- The agent speaks through this element. Keep it in the DOM. -->
<audio id="agent-audio" autoplay playsinline></audio>

<div id="transcript"></div>
<button id="talk">Start talking</button>
<script type="module" src="./ava-client.js"></script>

2 · The client#

const WS_URL = "wss://avarealtime.pathor.ai/";
const TOKEN     = "<your-jwt>";       // required
const PERSONA   = "<your-persona-id>"; // optional, but this is what gives it a voice

const ws  = new WebSocket(WS_URL);
const pc  = new RTCPeerConnection({ iceServers: [{ urls: "stun:stun.l.google.com:19302" }] });
const out = document.getElementById("transcript");

// Assistant bubbles are keyed by response_id, never by "the current turn".
const bubbles = new Map();

function bubbleFor(id) {
  let el = bubbles.get(id);
  if (!el) {
    el = document.createElement("p");
    el.dataset.responseId = id;
    out.appendChild(el);
    bubbles.set(id, el);
  }
  return el;
}

// ── 1. Session config ─────────────────────────────────────────────────────
ws.onopen = () => {
  send({
    type: "initial_config",
    value: {
      curr_user_token: TOKEN,
      personaId: PERSONA,
      client_source: "web",
      modalities: ["audio", "text"],
      stt_provider: "microsoft",
      lang: "en-IN",
      multi_response: true,   // opt in to concurrent responses
      plugin_search: true,    // opt in to JIT plugin discovery
    },
  });
};

// ── 2. Mic capture + WebRTC offer ─────────────────────────────────────────
async function connectAudio() {
  const mic = await navigator.mediaDevices.getUserMedia({
    audio: {
      echoCancellation: true,   // REQUIRED — cancels the voice coming out of the speaker
      noiseSuppression: true,
      autoGainControl: true,
      channelCount: 1,
    },
  });
  mic.getAudioTracks().forEach((t) => pc.addTrack(t, mic));

  // ── 3. Agent audio comes back on this track. Play it with a media element. ──
  pc.ontrack = (ev) => {
    document.getElementById("agent-audio").srcObject = ev.streams[0];
  };
  pc.onicecandidate = (ev) => {
    if (ev.candidate) send({ type: "candidate", candidate: ev.candidate });
  };

  const offer = await pc.createOffer({ offerToReceiveAudio: true });
  await pc.setLocalDescription(offer);
  send({ type: "offer", offer: pc.localDescription });
}
document.getElementById("talk").onclick = connectAudio;

// ── 4. Server events ──────────────────────────────────────────────────────
ws.onmessage = async (evt) => {
  if (typeof evt.data !== "string") return;   // binary frames: audio/image
  const msg = JSON.parse(evt.data);

  switch (msg.type) {
    case "session-created":
      console.log("session up:", msg.clientId);
      break;

    case "session-error":
      console.error("auth/init failed:", msg.message);
      ws.close();
      break;

    case "answer":
      await pc.setRemoteDescription(msg.answer ?? msg.sdp);
      break;

    case "candidate":
      if (msg.candidate) await pc.addIceCandidate(msg.candidate);
      break;

    case "text-stream": {
      const el = bubbleFor(msg.response_id ?? "legacy");
      if (msg.chunk) el.textContent += msg.chunk;
      if (msg.end)   el.dataset.done = "true";   // this response only, not the turn
      break;
    }

    case "user_speech_start":
      // interrupted === true means the agent was mid-sentence: drop queued audio.
      if (msg.interrupted) stopLocalPlayback();
      break;

    case "tool-calls":
      send({
        type: "tool-result",
        toolCallId: msg.tools.id,
        result: await runTool(msg.tools),
      });
      break;

    case "server_instruction":
      if (msg.action === "show_plugin") renderPluginCard(msg.plugin);
      break;

    case "rolling_summary":
      persistSummary(msg);   // hand it back as pre_chat_context next connect
      break;

    case "turn_suggestions":
      renderChips(msg.suggestions);
      break;
  }
};

// ── 5. Text turns work without any of the audio plumbing ──────────────────
function ask(text) {
  send({ type: "get_chat_completion", text, speech: true });
}

function send(obj) { ws.send(JSON.stringify(obj)); }

The handshake, in order#

Where to connect#

Tokens and personas#

Every session needs a bearer token. The server validates it against the PathOr persona API before it will set up an agent — there is no anonymous mode.

Accepted field names#

Send these at the top level of initial_config.value, or nested inside agent_data. Both forms are accepted; aliases exist for backwards compatibility.

What the server does#

{
  "type": "initial_config",
  "value": {
    "curr_user_token": "your-jwt",
    "personaId": "your-persona-id"
  }
}

{
  "type": "initial_config",
  "value": {
    "curr_user_token": "your-jwt",
    "agent_data": {
      "persona_id": "your-persona-id",
      "curr_user_token": "your-jwt",
      "tools_config": "<encrypted-blob>",
      "is_expressive_persona": true
    }
  }
}

{
  "type": "session-error",
  "message": "Authentication failed. Invalid or missing token."
}

Config reference#

The server keeps a per-session userConfig. Send only the fields you care about in initial_config.value — everything else falls back to a default or to the persona profile.

Speech recognition#

Two fields decide everything: stt_provider and lang. The language rules differ per provider, and picking the wrong pair is the most common source of "it does not hear me".

Examples#

{ "type": "initial_config", "value": { "stt_provider": "sarvam" } }

{ "type": "initial_config", "value": { "stt_provider": "sarvam", "lang": "hi-IN" } }

{ "type": "initial_config", "value": { "stt_provider": "microsoft", "lang": "en-IN" } }

{ "type": "update_stt_language", "lang": "en-IN" }

// Legacy alias, still accepted:
{ "type": "update_stt_language", "language": "en-IN" }

// Sarvam only — force auto-detection again:
{ "type": "update_stt_language", "lang": "unknown" }

Backwards compatibility: stt_lang is still accepted in initial_config, and language is still accepted in update_stt_language.

Persona and agent_data#

Sending a personaId is enough for most clients. agent_data is the escape hatch for when you already hold the resolved payload.

Fields used by persona-aware flows#

• curr_user_token, persona_id, persona_value, persona_languages
• tools_config, main_agent_tools
• legacy_tools — see below
• is_expressive_persona — enables richer prosody
• chat_history — seeded into the system prompt

Server-executed built-ins via legacy_tools#

Use these when you want the model provider to run a built-in tool directly, instead of the server emitting a client-side tool-calls request.

• Supported today: browser_search, code_interpreter
• code_execution is accepted on input and normalises to code_interpreter
• Only used on Groq-backed GPT-OSS paths
• Precedence: built-ins are attached only when no normal function tools are active for that turn. If main_agent_tools or persona function tools exist, legacy_tools is ignored for that turn.
• Unsupported entries for the selected model are silently dropped
• These never produce tool-calls. You may receive tool-activity instead.

{
  "type": "initial_config",
  "value": {
    "use_groq": true,
    "groq_model": "openai/gpt-oss-120b",
    "agent_data": {
      "legacy_tools": ["browser_search", "code_execution"]
    }
  }
}

A browser-search turn may emit an early tool-activity with phase: "requested" and a conversational status_text before the answer is ready. That keep-alive phrase is spoken server-side and is not sent as normal text-stream content.

Session modes#

Four flags change the shape of a session significantly. They compose, but each answers a different question.

Chat persistence runs when personaId and a token are set and client_source is one of web, ava_desktop, ava_android, web_agent — and the session is neither game_mode nor ephemeral_session.

Client → Server messages#

Everything is JSON unless noted. snapshot and mic_audio also accept a binary frame — a JSON header line, a newline, then the raw bytes — which avoids base64 overhead.

{
  "type": "initial_config",
  "value": {
    "curr_user_token": "your-jwt",
    "personaId": "your-persona-id",
    "client_source": "web",
    "modalities": ["audio", "text"],
    "stt_provider": "microsoft",
    "lang": "en-IN"
  }
}

{ "type": "offer",     "offer": { "type": "offer", "sdp": "..." } }
{ "type": "candidate", "candidate": { "candidate": "...", "sdpMid": "0", "sdpMLineIndex": 0 } }

{
  "type": "get_chat_completion",
  "text": "Can I apply available coupons?",
  "speech": true,
  "image": "<base64-jpeg>",
  "additional_current_context": "cart_total=1999; coupon_screen=true",
  "exclude_from_chat_history": false,
  "plugin_search": true
}

{
  "type": "update_additional_current_context",
  "additional_current_context": "cart_total=1999; currency=INR; selected_sku=XYZ789"
}

// Also accepted:
{ "type": "additional_current_context", "value": "..." }

{
  "type": "update_user_details",
  "user_details": {
    "latitude": "18.8047601",
    "longitude": "73.3204305",
    "location": "Pune",
    "devicePermissions": { "approved": ["location"], "not_approved": [] }
  }
}

// Also accepted:
{ "type": "user_details", "value": { ... } }

{ "type": "update_stt_language", "lang": "en-IN" }
{ "type": "update_stt_language", "language": "en-IN" }   // legacy alias
{ "type": "update_stt_language", "lang": "unknown" }     // Sarvam: back to auto-detect

{ "type": "tool-result", "toolCallId": "call_abc123", "result": { "ok": true }, "isDirect": false }

{ "type": "plugin_enabled", "plugin": { "_id": "6612...", "title": "Spotify" } }

{ "type": "refresh_plugins" }
{ "type": "refresh_plugins", "plugins": [ /* push a specific set */ ] }

{"type":"snapshot","contentType":"image/jpeg"}\n<jpeg-bytes>

// JSON + base64
{ "type": "mic_audio", "data": "<base64 pcm s16le 48k mono>" }

// Binary frame (preferred)
{"type":"mic_audio","format":"s16le","sampleRate":48000,"channels":1}\n<pcm-bytes>

{ "type": "start_video_chat", "prompt": "You are looking through the user's camera at a whiteboard." }
{ "type": "end_video_chat" }

{ "type": "update_sys_prompt", "prompt": "You are assisting with a live product demo." }

{ "type": "stop_speech" }

{ "type": "toggle_mute_speech", "mute": true }

{ "type": "ping", "client_ts": 1718000000000 }

{ "type": "dashboard_init", "weather": "28°C, light rain" }

{ "type": "update_meeting_participants", "action": "join",
  "participant": { "name": "Ravi Sharma", "designation": "CTO", "speakerId": "speaker_0" } }

{ "type": "update_meeting_participants", "action": "leave", "speakerId": "speaker_1" }

{ "type": "update_meeting_participants", "action": "sync", "participants": [ /* full replacement */ ] }

{ "type": "speaker_context", "speaker_name": "Ravi Sharma", "speaker_identity": "user_8821" }

{ "type": "presentation_started", ... }        // a presentation has begun
{ "type": "presentation_ready", ... }          // rendered and ready → ack_presentation_ready
{ "type": "presentation_page_changed", ... }   // slide navigation
{ "type": "presentation_ended" }

Server → Client messages#

Handle the ones your product needs and ignore the rest — unknown types are safe to skip. The four you almost certainly need are session-created, text-stream, user_speech_start and tool-calls.

{ "type": "session-created", "clientId": "...", "auth": true, "message": "..." }

{ "type": "session-error", "message": "Authentication failed. Invalid or missing token." }

{ "type": "text-stream", "chunk": "On it — ", "response_id": "resp_7" }
{ "type": "text-stream", "chunk": "give me a second.", "response_id": "resp_7" }
{ "type": "text-stream", "end": true, "text": "On it — give me a second.",
  "finish_reason": "stop", "response_id": "resp_7" }

{ "type": "user_speech_start", "reason": "silero_vad", "interrupted": true }

{ "type": "user_speech_end", "text": "play something by Ali Sethi" }

{ "type": "speech_chunk", "encoding": "base64", "data": "..." }

{ "type": "tool-calls", "tools": { "id": "call_abc123", "function": { "name": "...", "arguments": "{...}" } } }

{ "type": "tool-activity", "tool": "...", "tools": [...], "provider": "ava_live_tools",
  "path": "ava_legacy_tool_fast", "phase": "requested",
  "status_text": "Let me look that up…", "response_text": "...",
  "evidence": [...], "citation_count": 3 }

{
  "type": "server_instruction",
  "action": "show_plugin",
  "plugin": {
    "_id": "6612...", "title": "Spotify", "name": "spotify", "logo": "https://...",
    "human_description": "Play music and control playback",
    "creator": "PathOr", "authentication_required": true,
    "status": "not_connected", "os": ["android"], "isAuthenticated": false
  }
}

{ "type": "rolling_summary", "summary": "...", "foundation": "...",
  "turn_count": 8, "timestamp": 1718000000000 }

{ "type": "turn_suggestions", "suggestions": ["Summarize that", "Send it to Priya", "What did I miss?"] }

{ "type": "dashboard_init_response",
  "thought_of_the_day": "Keep your face to the sun ☀️.",
  "suggestions": ["Summarize my day.", "Teach me basics of python.", "Lets play some game"],
  "error": "optional — present only when generation failed" }

{ "type": "refresh_plugins_success", "status": "success",
  "message": "Plugins refreshed successfully", "toolsCount": 12 }

{ "type": "refresh_plugins_success", "toolsCount": 0, "toolsDisabled": true }   // disable_tool session

{ "type": "refresh_plugins_error", "status": "error", "message": "Failed to refresh plugins: ..." }

{ "type": "speaker_identified", "speakerId": "speaker_0", "name": "Ravi Sharma", "designation": "CTO" }

{ "type": "dev_log", "message": "...", "log_level": "info" }

{ "type": "ack_disconnect" }
{ "type": "ack_mute_speech" }
{ "type": "ack_additional_current_context", "status": "ok", "has_context": true, "length": 47 }
{ "type": "ack_user_details", "status": "ok", "fields": ["latitude", "longitude"], "has_location": true }
{ "type": "ack_presentation_ready", "status": "ok" }
{ "type": "pong", "client_ts": 1718000000000, "server_ts": 1718000000123 }

Concurrent responses#

The server now runs multi-step work while the conversation continues. The user can keep talking, independent actions run together, and the agent reports back when it finishes. That breaks one assumption almost every client makes: one turn, one response.

Adoption checklist#

In the order worth doing them. Nothing breaks if you do none of it — concurrency is enabled per client_source, so an unadopted client keeps getting exactly one response per turn.

What response_id means#

Every text-stream frame carries response_id (e.g. "resp_3"), monotonic per session. A response is one unit of assistant output: a stream of chunk frames terminated by exactly one end: true.

Until now a turn produced exactly one response, so clients could safely treat "the turn is over" and "this reply is over" as the same event. That equivalence is gone. A session can have several responses in flight: an acknowledgement, anything said while the work runs, then the result of the work itself.

One request — "play a song, take a selfie, text Sneha" — now looks like this:

resp_7  "On it — I'll let you know as soon as it is done."   end:true   ← the turn closes here
resp_8  "Kya kar rahi ho?" → "Bas isi pe lagi hoon!"          end:true   ← user talked mid-task
resp_9  "Song chal raha hai, selfie bhi le li — Sneha ko       end:true   ← the task, reporting back
         message bhej diya hai, delivery confirm nahi kar
         paayi."

resp_9 arrives with no user message before it. That is rule 3 below, and it is the whole point of the feature.

The five client rules#

Opting in#

Implement the five rules, then either ask for your client_source to be added server-side (DETACHED_PLAN_EXECUTION_SOURCES) or send the flag yourself. It is read per turn, so it can be toggled mid-session and does not have to live in initial_config.

{ "multi_response": true }

multi_response: false opts back out at any time. supports_multi_response and multiResponse are accepted spellings.

Rendering a running plan#

With multi_response on, a multi-step request unfolds over several responses — and a step has more states than "running" and "finished".

The question itself arrives as a normal assistant response with its own response_id, carrying the options inline. If the user says something else in the meantime, the server re-asks rather than dropping it — so expect the same question to legitimately appear more than once. Key any de-duplication on response_id, not on message text.

Steps whose dependency failed are reported as not done, never silently omitted. If your UI hides them, the user reads "1 of 4 succeeded" as an unqualified success.

Tools and plugins#

Three different things can execute work on a turn, and they behave differently on the wire. Knowing which one you are looking at removes most tool-integration confusion.

Client-executed tool flow#

JIT plugin discovery (plugin_search)#

When a turn needs a capability none of the session's connected tools cover, the server can semantically search the plugin marketplace and either run a plugin the user already enabled, or return a "connect this plugin?" consent card. This behaviour is per client source.

Any client can override the default explicitly:

{ "type": "initial_config", "value": { "client_source": "web", "plugin_search": true } }

// Toggle mid-session — on any later message, no reconnect needed:
{ "type": "get_chat_completion", "text": "book me a cab", "speech": true, "plugin_search": true }

• Accepts true/false, "true"/"false", "on"/"off", 1/0.
• Aliases: pluginSearch, plugin_search_enabled, jit_plugin_search, enable_plugin_search.
• The value is sticky until the client sends a different one, and applies from the next turn.

When disabled, the session is limited to its connected tools: no marketplace search, no consent cards, no JIT recovery after a tool failure — the agent answers honestly that it cannot do it. Explicit plugin management ("enable the Spotify plugin") still works either way, and disable_tool overrides everything.

The consent handshake#

Isolated game mode with custom tools#

Keep the voice and personality, replace the toolset. Useful for games and sandboxes where persona plugins (messaging, system apps) would be wrong.

{
  "type": "initial_config",
  "value": {
    "curr_user_token": "your-jwt",
    "personaId": "your-persona-id",
    "game_mode": true,
    "tools": [
      {
        "type": "function",
        "function": {
          "name": "check_game_item",
          "description": "Evaluate if the physical item shown in the user's camera snapshot matches the target.",
          "parameters": {
            "type": "object",
            "properties": {
              "item_detected": { "type": "string" },
              "closeness":     { "type": "string", "enum": ["cold", "warm", "hot", "correct"] },
              "verbal_hint":   { "type": "string" }
            },
            "required": ["item_detected", "closeness", "verbal_hint"]
          }
        }
      }
    ]
  }
}

The server loads the persona voice, prompt guidelines and details, then restricts the active tools to exactly your list. Execution uses the standard tool-calls → tool-result flow.

Returning a plugin artifact#

Read this if any of your plugins produce something — a photo, a scan, a file, a screenshot. Posting it back into the conversation is supported and expected. But the artifact has to reach the server.

Why it matters#

A device action the server dispatches but cannot verify is recorded as dispatched_unconfirmed: "I sent it to your device, I can't confirm it worked." That is deliberate — the server refuses to claim an outcome it has no evidence for.

The artifact is the evidence. When it arrives, that action resolves to confirmed, and every downstream surface changes: the closing summary says it is done, a later "did that go through?" answers correctly, and any reply made while the work runs stops hedging.

The contract#

• Attach the artifact to the message you post back. For an image, send image as base64 (no data-URI prefix) on the same get_chat_completion payload that carries the caption. A binary frame works too.
• Send it every time, not only when convenient. Intermittent delivery is worse than none: the assistant confirms on some runs and hedges on others, for reasons the user cannot see.
• The caption alone is not evidence. A message reading "Captured the moment" with no attachment tells the server that a user spoke, not that a capture succeeded.
• Late is fine. The server resolves the action whenever the artifact lands — a different turn, several seconds later, any route. It does not need to share a frame with the caption, and the server will not block waiting for it.

How to check#

Ask for something that produces an artifact. In the server log you want:

dispatch_confirmed_by_inbound_artifact   source: "chat_message"

If that line is absent, the artifact did not reach the server, and the assistant will correctly report the action as unconfirmed no matter what your UI is showing.

Vision#

The default chat model is vision-capable, so image turns need no special configuration — just get the pixels to the server.

Text plus vision to speech#

Audio and echo cancellation#

The agent's voice comes out of the speaker and back into the mic. The only place that loop can be cancelled properly is on the device, where the OS knows exactly what is being played. This is a client requirement, not a nice-to-have.

// 1. Capture the mic WITH processing enabled — do not disable these.
const mic = await navigator.mediaDevices.getUserMedia({
  audio: {
    echoCancellation: true,   // REQUIRED — cancels the voice coming back from the speaker
    noiseSuppression: true,
    autoGainControl: true,
    channelCount: 1,
  },
});
pc.addTrack(mic.getAudioTracks()[0], mic);

// 2. Play the agent's audio via the remote WebRTC track and a media element.
//    Do NOT decode/play agent audio through the Web Audio API
//    (AudioContext / AudioWorklet) — Chromium's AEC cannot see that path
//    and will not cancel it.
pc.ontrack = (ev) => {
  const audioEl = document.getElementById("agent-audio"); // <audio autoplay>
  audioEl.srcObject = ev.streams[0];
};

Rules of thumb#

• Platform AEC only cancels audio it knows it is playing. Any playback path that bypasses the platform voice pipeline — raw PCM through your own audio graph — escapes cancellation.
• Headphones eliminate the loop entirely. Speakerphone is the worst case — test there.
• If you must use receive_tts_chunks: true in a browser, route your playback through a loopback RTCPeerConnection and a media element so the AEC gets a reference signal.

Server-side DSP is a safety net, not a substitute#

noise_supression and echo_cancellation in the config toggle server-side RNNoise denoise and an NLMS echo filter on inbound mic audio. They help, but they run where the speaker signal is not known. Real AEC has to run on the device.

Barge-in and VAD tuning#

{ "vad_config": { "threshold": 0.75, "minSpeechDuration": 0.2, "minSilenceDuration": 0.4 } }

Formats#

mic_audio

PCM s16le, 48 kHz, mono

speech_chunk

PCM 24 kHz, mono, 16-bit, base64

WebRTC downlink

Server resamples TTS to 48 kHz before injecting it into the track

Ambient conversational intelligence#

Ambient mode changes the agent from reactive — it responds when addressed — to ambient: it attends the whole conversation and autonomously decides when, or whether, to speak.

How a turn is decided#

In normal mode the server responds to every finalized STT utterance. In ambient mode each utterance passes through a fast engagement gate before any generation happens.

The gate is WAIT-biased by default. It derives its identity — name, domain, role — from the persona system prompt and decides autonomously. It speaks on a clear signal: a direct name mention, an unanswered open question in the window, or a high-urgency domain-relevant contribution such as a correction or a critical risk. Below urgency 4, even a SPEAK decision is suppressed.

Enabling it#

{
  "type": "initial_config",
  "value": {
    "modalities": ["audio", "text"],
    "ambient_intelligence": true,
    "ambient_meeting_context": "customer support call",
    "ambient_mode_config": {
      "stt_provider": "deepgram",
      "silence_gap_ms": 300,
      "question_gap_ms": 100,
      "context_window_turns": 6,
      "max_tokens": 120
    },
    "meeting_participants": [
      { "name": "Ravi Sharma", "designation": "CTO", "speakerId": "speaker_0" },
      { "name": "Priya Nair",  "designation": "PM" }
    ],
    "capability_notes": "You can present URLs and PDFs as a screen share."
  }
}

ambient_mode_config sub-fields

The last two columns differ on purpose — the deployment is tuned tighter than the code defaults for latency. Neither is a contract. If your experience depends on a specific gap, send ambient_mode_config and pin it per session.

What changes for your client#

Speaker diarization#

With stt_provider: "deepgram", labels (speaker_0, speaker_1, …) are assigned sequentially by first-speech appearance and resolved to names through the meeting_participants roster. Each auto-mapping emits a speaker_identified event so a meeting UI can confirm or correct it. Update the roster mid-session with update_meeting_participants, without reconnecting.

If your client receives a mixed audio track — one blended stream for the whole room — diarization cannot work at all. Send speaker_context on every active-speaker change instead.

The interruption prefix#

On an INTERRUPT decision the spoken response is prefixed with a randomised softening phrase — "Actually, let me add something here — ", "Hold on, that's worth clarifying — ". It appears at the start of the text-stream content, so you can display it in a transcript.

Server-side tuning#

Cross-session memory#

The server condenses each conversation into a rolling summary and streams it to you. Hand it back on the next connect and a returning user gets a greeting that references their last session — not a generic "Hello, how can I assist?"

The rolling_summary event#

{
  "type": "rolling_summary",
  "summary": "User is Priya Nair, mobile app PM. Discussed feature prioritisation for Q3 sprint — delay widget + dark mode. Prefers bullet points. Next step: share mockup.",
  "foundation": "Priya Nair is a mobile app PM focused on Q3 sprint planning.",
  "turn_count": 8,
  "timestamp": 1718000000000
}

Sending it back as pre_chat_context#

{
  "type": "initial_config",
  "value": {
    "modalities": ["audio", "text"],
    "stt_provider": "microsoft",
    "lang": "en-IN",
    "agent_data": { "persona_id": "my-persona-id", "curr_user_token": "user-jwt" },
    "pre_chat_context": [
      {
        "summary": "User is Priya Nair, mobile app PM. Discussed feature prioritisation for Q3 sprint...",
        "foundation": "Priya Nair is a mobile app PM focused on Q3 sprint planning.",
        "turn_count": 8,
        "timestamp": 1718000000000
      }
    ]
  }
}

The array is ordered oldest first. The server merges every entry into one SESSION MEMORY block and injects it into every LLM system prompt for the session. Where session memory conflicts with live in-session messages, the live messages win.

A working localStorage pattern#

const MAX_ENTRIES = 3;
const keyFor = (personaId) => `ava_summaries_${personaId}`;

// Save whenever a rolling_summary arrives
ws.addEventListener("message", (evt) => {
  const msg = JSON.parse(evt.data);
  if (msg.type !== "rolling_summary") return;

  const entries = JSON.parse(localStorage.getItem(keyFor(personaId)) || "[]");
  entries.push({
    summary:    msg.summary,
    foundation: msg.foundation,
    turn_count: msg.turn_count,
    timestamp:  msg.timestamp,
  });
  if (entries.length > MAX_ENTRIES) entries.shift();   // keep the newest three
  localStorage.setItem(keyFor(personaId), JSON.stringify(entries));
});

// Read on the next connect
function buildInitialConfig(personaId, token) {
  const stored = JSON.parse(localStorage.getItem(keyFor(personaId)) || "[]");
  return {
    type: "initial_config",
    value: {
      modalities: ["audio", "text"],
      stt_provider: "microsoft",
      lang: "en-IN",
      agent_data: { persona_id: personaId, curr_user_token: token },
      ...(stored.length > 0 ? { pre_chat_context: stored } : {}),
    },
  };
}

Starting from a notification#

When the conversation starts because the user tapped a simulated notification, pass what they tapped. The server appends a NOTIFICATION ENTRY POINT instruction to SESSION MEMORY and directs the assistant to continue contextually from that message.

{
  "type": "initial_config",
  "value": {
    "agent_data": { "persona_id": "my-persona-id", "curr_user_token": "user-jwt" },
    "notification_context": {
      "assistant_message": "yoo supp rohan, remember we talked about burger yesterday, r u there?",
      "assistant_name": "sam",
      "timestamp": 1718000000000
    }
  }
}

The returning-user greeting#

When pre_chat_context or notification_context is present, the server replaces the default start_prompt with a context-aware instruction:

"A returning user has reconnected. Their previous session context is in SESSION MEMORY above — read it before responding. Greet them warmly by name if their name is mentioned in the context. Reference something specific and relevant from prior conversations naturally. Do NOT say this is their first call. Do NOT introduce yourself from scratch."

With notification_context it also appends an instruction to continue from the notification message. This is what stops the model falling back to a generic greeting even when the configured start_prompt would normally trigger a fresh introduction.

Server-side tuning#

Recipes#

The common shapes, end to end. Each one assumes you have already sent initial_config with a valid token.

Speech to speech (WebRTC first)#

Feeding live app state#

additional_current_context is the canonical channel for current app state — cart, selected item, active screen, checkout step, permission state. Session-scoped and sticky until replaced.

{
  "type": "initial_config",
  "value": {
    "client_source": "web_agent",
    "additional_current_context": "cart_total=1499; currency=INR; selected_sku=ABC123"
  }
}

{
  "type": "update_additional_current_context",
  "additional_current_context": "cart_total=1999; currency=INR; selected_sku=XYZ789"
}

{
  "type": "get_chat_completion",
  "text": "Can I apply available coupons?",
  "speech": true,
  "additional_current_context": "cart_total=1999; coupon_screen=true"
}

Practices worth keeping

• Keep it compact and factual — state, ids, flags. Avoid long prose.
• Update only when state changes materially.
• Prefer stable keys (selected_sku, cart_total, step, locale) so behaviour stays predictable.
• Do not put secrets or unnecessary PII in it — this text goes into model instructions.

Server-executed built-in tools#

Dev logging#

Send start_dev_log to subscribe this session to dev_log messages. Use it during integration; turn it off with end_dev_log.

Limits, errors and operations#

Payload sizing#

• WebSocket max payload is about 5 MB. Prefer binary framing for audio and images to avoid base64 bloat.
• TTS PCM frames are 24 kHz mono 16-bit. The server resamples to 48 kHz when injecting into WebRTC.
• Backpressure: when the socket bufferedAmount exceeds roughly 4 MB, non-critical text-stream and speech_chunk messages may be dropped until congestion clears. The final end: true is always sent.
• additional_current_context is capped at 8000 characters (MAX_ADDITIONAL_CURRENT_CONTEXT_CHARS) and truncated at a word boundary above that.
• tts_style_prompt is capped at 100 bytes, also word-safe truncated.

Error handling#

Operational notes#

• Inactivity timeout is configured by INACTIVITY_TIMEOUT; the server sends self_disconnect and then closes.
• Android clients should provide agent_data.curr_user_token and agent_data.persona_id so the persona fetch authenticates.
• With use_openai_realtime: true the server streams text and optionally audio from the OpenAI Realtime endpoint. Tool-calling flows may differ depending on the configured deployment.
• Chat persistence runs when personaId and a token are set and client_source is web, ava_desktop, ava_android or web_agent — and the session is neither game_mode nor ephemeral_session.
• Default non-vision Groq chat paths use openai/gpt-oss-120b; vision turns use the configured vision model.

Minimum viable client#

Everything you strictly need, with no audio plumbing at all.

const ws = new WebSocket("wss://avarealtime.pathor.ai/");

ws.onopen = () => {
  ws.send(JSON.stringify({
    type: "initial_config",
    value: {
      curr_user_token: TOKEN,
      modalities: ["audio", "text"],
      stt_provider: "microsoft",
      lang: "en-IN",
    },
  }));
};

ws.onmessage = (evt) => {
  const msg = JSON.parse(evt.data);
  switch (msg.type) {
    case "session-created":
      console.log("session", msg.clientId);
      break;

    case "text-stream":
      if (msg.chunk) render(msg.response_id, msg.chunk);
      if (msg.end)   finalize(msg.response_id, msg.text);
      break;

    case "tool-calls":
      ws.send(JSON.stringify({
        type: "tool-result",
        toolCallId: msg.tools.id,
        result: { ok: true },
      }));
      break;
  }
};

Glossary#