Streaming into your app

The Quickstart shows the smallest working client. This page covers the production shape: a mint proxy on your server, LiveKit room join on the client, agent-track subscription, chat over the LiveKit data channel, reconnect, and the common failure modes.

Most consumers don't write this directly

Every 2wai SDK (Flutter, Web / TypeScript, iOS native) handles the client side of this for you. Reach for the SDK first; the raw walkthrough below is for understanding what the SDK does, and for integrators in languages no SDK targets yet.

Shape

+------------------+          +----------------+          +---------------------+
|   Your client    |          |  Your server   |          |     2wai API        |
|                  |          |  (mint proxy)  |          |  (api.2wai.ai)      |
+--------+---------+          +--------+-------+          +----------+----------+
         |                             |                             |
         | 1. POST /mint-proxy         |                             |
         | ------------------------->  | 2. POST /v1/sessions        |
         |                             | --------------------------> |
         |                             |                             |
         |                             | <-- livekit_url, token      |
         | <-- livekit_url, token      |                             |
         |                                                           |
         | 3. join room (livekit_url, participant_token)             |
         | ----------------------------------------------------------|
         |                                                           |
         | <-- agent participant appears, video + audio tracks       |
         |                                                           |
         | 4. send user text over data channel topic "lk.chat"       |
         | ----------------------------------------------------------|
         |                                                           |
         | <-- agent replies: audio speaks, data-channel reply text  |
         |                                                           |

Three distinct hops:

1. Mint proxy (your server) — holds the sk_, exchanges it for a per-session participant_token.
2. LiveKit room join (your client → LiveKit cloud) — WebRTC transport, standard LiveKit SDK.
3. Chat envelope over data channel — user turns go out as JSON on topic lk.chat, reply chunks come back the same way.

Mint proxy

The simplest Node mint proxy:

mint-proxy.js

import express from 'express';
const app = express();
app.use(express.json());

app.post('/mint', async (req, res) => {
  const r = await fetch('https://api.2wai.ai/v1/sessions', {
    method: 'POST',
    headers: {
      Authorization: `Bearer ${process.env.TWAI_SECRET_KEY}`,
      'Content-Type': 'application/json',
    },
    body: JSON.stringify({
      avatar_id: req.body.avatar_id,  // or hardcode on the proxy
      voice_id: req.body.voice_id,
      mode: 'video',
      transport: 'livekit',
    }),
  });
  if (!r.ok) return res.status(r.status).json(await r.json());

  const s = await r.json();
  // Return only the fields the client needs to join the room.
  res.json({
    session_id: s.session_id,
    livekit_url: s.livekit_url,
    participant_token: s.participant_token,
    room_name: s.room_name,
  });
});

app.listen(3000);

Reference impls for Node, Python, and Cloudflare Worker are in 2waiorg/2wai-toolkit-public.

Client — join + render

client.html

<video id="avatar" autoplay playsinline style="max-width: 480px"></video>
<script src="https://cdn.jsdelivr.net/npm/livekit-client/dist/livekit-client.umd.min.js"></script>
<script type="module">
  const { Room, RoomEvent, DataPacket_Kind } = LivekitClient;
  const video = document.getElementById('avatar');

  async function connect() {
    const res = await fetch('/mint', {
      method: 'POST',
      headers: { 'Content-Type': 'application/json' },
      body: JSON.stringify({ avatar_id: 'avt_...' }),
    });
    const { livekit_url, participant_token } = await res.json();

    // Mandatory for 2wai: adaptiveStream + dynacast MUST be off.
    const room = new Room({ adaptiveStream: false, dynacast: false });

    room.on(RoomEvent.TrackSubscribed, (track) => {
      if (track.kind === 'video') track.attach(video);
      else if (track.kind === 'audio') track.attach();
    });

    room.on(RoomEvent.DataReceived, (payload, participant, kind, topic) => {
      if (topic === 'lk.chat') {
        const frame = JSON.parse(new TextDecoder().decode(payload));
        if (frame.text) console.log('avatar reply:', frame.text, 'final:', frame.final);
      }
    });

    await room.connect(livekit_url, participant_token);
    return room;
  }

  // Send a user text turn over the data channel.
  async function send(room, text) {
    const envelope = {
      type: 'user_text',
      turn_id: crypto.randomUUID(),
      text,
    };
    const data = new TextEncoder().encode(JSON.stringify(envelope));
    await room.localParticipant.publishData(data, { reliable: true, topic: 'lk.chat' });
  }

  const room = await connect();
  window.send = (text) => send(room, text);
</script>

The chat envelope (data channel topic lk.chat)

User-text (client → agent):

{"type":"user_text","turn_id":"<uuid>","text":"Hi, tell me about yourself.","extra":{}}

Reply-frame (agent → client, one or more per turn):

{"type":"reply","turn_id":"<uuid>","text":"Hi — I'm Dr. Alman.","final":false}

The final: true frame marks the end of a reply turn. The agent’s audio track plays the voiced version of the same text; the data-channel reply is for your UI (showing the reply text alongside the video, or feeding analytics / logs).

Speaking events

LiveKit’s RoomEvent.ActiveSpeakersChanged fires when the agent participant starts/stops producing audio. Use it to gate your Send button (don’t let the user interrupt themselves) and to animate the UI (e.g. “listening” vs “speaking” state).

room.on(RoomEvent.ActiveSpeakersChanged, (speakers) => {
  const agentActive = speakers.some(p => p.identity.startsWith('livealman') || p.isRemote);
  // update UI
});

Driving the conversation server-side

Your mint proxy routes /say to the gateway’s /speak endpoint using your sk_:

app.post('/say', async (req, res) => {
  const { session_id, text } = req.body;
  const r = await fetch(`https://api.2wai.ai/v1/sessions/${session_id}/speak`, {
    method: 'POST',
    headers: {
      Authorization: `Bearer ${process.env.TWAI_SECRET_KEY}`,
      'Content-Type': 'application/json',
    },
    body: JSON.stringify({ text }),
  });
  res.status(r.status).json(await r.json());
});

Call /interrupt the moment you detect new user input to cut off a stale reply — the LiveKit audio track mutes, the agent gets ready for the next turn.

Reconnect

LiveKit SDKs handle network blips automatically. For longer drops (token expired, session ended), re-mint via POST /v1/sessions and re-connect the room with the fresh token:

async function reconnect() {
  await room.disconnect();
  const { livekit_url, participant_token } = await (await fetch('/mint', { method: 'POST' })).json();
  await room.connect(livekit_url, participant_token);
}

Exponential backoff on retry (1s, 2s, 4s, cap at 30s). SDKs ship this; raw clients should implement it.

Common failure modes

Symptom	Likely cause	Fix
Mint returns 401	sk_ wrong / revoked	Verify curl ... /v1/sessions with the sk_ directly (not via proxy)
Mint 429	Tenant cap hit	Back off per retry-after header
Room connects but no video appears	adaptiveStream not disabled	Pass new Room({ adaptiveStream: false, dynacast: false })
Avatar silent after speak	TTS voice unknown on your tenant	See Voices — register a voice first
First user turn clipped mid-sentence	Sent before agent warm-up window	SDKs honor a 2-4s SEND_GRACE_MS window after agent-ready; raw clients should wait a beat after first subscribe before sending

Reach for an SDK

Every item in the failure table above is handled inside the 2wai SDKs — Flutter’s TwaiTransport0Player, Web’s @twai/core engine, iOS’s TwaiPlayerView. The raw path is good for understanding; the SDKs are the production path.

Next

Reference integration End-to-end code walk-through with every piece wired up.

Bring your own brain Swap the gateway's default brain for your own LLM.

Sessions Lifecycle, pausing, reconnect, cleanup.