Streaming into your app

Once your server has a client_token and websocket_url (see Authentication), the browser’s whole job is: connect the WebSocket, feed what it receives into a <video> element, and forward a couple of control messages. This page builds a small reusable AvatarPlayer class that does that properly. The Quickstart version is a simplified inline version of the same thing.

The wire protocol

Once the WebSocket is open:

1. Client to server, one JSON text frame to start: {"type": "session.ready"}. Send this once your MediaSource and SourceBuffer are set up and ready to receive data.
2. Server to client, binary frames: an fMP4 initialization segment first, then a stream of fMP4 media segments. Append each one, in order, to your SourceBuffer.
3. Server to client, JSON text frames for control: {"type": "speaking", "value": "start"} and {"type": "speaking", "value": "stop"} bracket each utterance, so you can drive a speaking indicator, disable/enable your input box, and so on.

Note

Treat unrecognized type values as safe to ignore. New control frame types may be added over time, so don’t assume speaking and session.ready are the only ones you’ll ever see.

The exact codec string

Both MediaSource.isTypeSupported() and addSourceBuffer() need this precise MIME/codecs string:

const MIME_CODEC = 'video/mp4; codecs="avc1.42C01F, mp4a.40.2"';

That’s H.264 (Baseline, level 3.1) video and AAC-LC audio, muxed as fragmented MP4. Check support before you connect:

if (!MediaSource.isTypeSupported(MIME_CODEC)) {
  throw new Error('This browser cannot play the avatar stream.');
}

Why mode = 'sequence'

Set sourceBuffer.mode = 'sequence' right after creating it. Segments are appended in the order they arrive over the WebSocket (a single connection preserves ordering), and 'sequence' mode places each new segment immediately after the previous one on the timeline, which is what you want for a live stream that doesn’t carry meaningful absolute timestamps across segments.

Backpressure

SourceBuffer.appendBuffer() is asynchronous and throws if you call it while a previous append is still updating. Queue incoming segments and drain the queue on updateend:

AvatarPlayer.js

export class AvatarPlayer {
  static MIME_CODEC = 'video/mp4; codecs="avc1.42C01F, mp4a.40.2"';

  constructor(videoEl) {
    this.video = videoEl;
    this.queue = [];
    this.mediaSource = null;
    this.sourceBuffer = null;
    this.ws = null;

    // Assign these from the outside if you want to react to them.
    this.onSpeaking = null; // (value: "start" | "stop") => void
    this.onClose = null; // (event: CloseEvent) => void
    this.onError = null; // (event: Event) => void
  }

  connect(websocketUrl) {
    if (!MediaSource.isTypeSupported(AvatarPlayer.MIME_CODEC)) {
      throw new Error('This browser cannot play the avatar stream.');
    }

    this.mediaSource = new MediaSource();
    this.video.src = URL.createObjectURL(this.mediaSource);

    this.mediaSource.addEventListener('sourceopen', () => {
      this.sourceBuffer = this.mediaSource.addSourceBuffer(AvatarPlayer.MIME_CODEC);
      this.sourceBuffer.mode = 'sequence';
      this.sourceBuffer.addEventListener('updateend', () => this._drain());

      // websocket_url already carries the client_token as a query param.
      this.ws = new WebSocket(websocketUrl);
      this.ws.binaryType = 'arraybuffer';

      this.ws.addEventListener('open', () => {
        this._send({ type: 'session.ready' });
      });

      this.ws.addEventListener('message', (event) => this._onMessage(event));
      this.ws.addEventListener('close', (event) => this.onClose?.(event));
      this.ws.addEventListener('error', (event) => this.onError?.(event));
    });
  }

  _onMessage(event) {
    if (typeof event.data === 'string') {
      const message = JSON.parse(event.data);
      if (message.type === 'speaking') {
        this.onSpeaking?.(message.value);
      }
      return;
    }

    // Binary frame: an init segment or a media segment. Either way, queue
    // it and let _drain() append it once the SourceBuffer is free.
    this.queue.push(event.data);
    this._drain();
  }

  _drain() {
    if (!this.sourceBuffer || this.sourceBuffer.updating || this.queue.length === 0) return;
    this.sourceBuffer.appendBuffer(this.queue.shift());
  }

  _send(payload) {
    if (this.ws?.readyState === WebSocket.OPEN) {
      this.ws.send(JSON.stringify(payload));
    }
  }

  close() {
    this.ws?.close();
    if (this.mediaSource?.readyState === 'open') {
      try {
        this.mediaSource.endOfStream();
      } catch {
        // Ignore: the source may already be closing.
      }
    }
  }
}

Usage:

const player = new AvatarPlayer(document.querySelector('#avatar'));
player.onSpeaking = (value) => console.log('avatar speaking:', value);
player.onClose = () => console.log('session stream closed');
player.connect(websocketUrl);

// later, e.g. on component unmount
player.close();

Cleanup

Always call close() (or equivalent) when the viewer navigates away or the session ends. This closes the WebSocket and finalizes the MediaSource so the browser can release decoder resources. Leaving stale players open across route changes is the most common cause of “why are there five video decoders running” bug reports.

Reconnecting

WebSockets drop: proxies restart, wifi hiccups, laptops sleep. A dropped connection doesn’t have to mean a lost session. Request a fresh client_token/websocket_url pair from your server (which calls POST /v1/sessions/{id}/token) and call connect() again. See Sessions for the full reconnect flow and backoff guidance.