# `@animated-waffle/client`

Framework-independent browser SDK for embedding a published Narya talking
Agent: connection lifecycle, microphone input, text input, state, and
transcripts. React applications should use
[`@animated-waffle/react`](../waffle-react), which re-exports this entire API
and adds the Avatar renderer. For plain JavaScript or another browser framework, use
[`@animated-waffle/web`](../waffle-web) and `mountWaffleAvatar`. This package is
headless; it does not render a character.

Install the current npm release without copying a mutable registry version:

```bash
pnpm add @animated-waffle/client@latest
```

The managed-session identity and `sessionContext` APIs described here were
introduced in version 0.3.1. The workspace [`package.json`](./package.json) is
the source of truth for this repository's version; `@latest` resolves the
current npm release.

The Agent owns its avatar, voice, and prompt. Media runtime, API routes, and
avatar asset URLs are SDK implementation details. Production is the default
API origin; tests and non-production integrations may pass `apiOrigin`.

## Connect

Keep the permanent workspace API key on your server. Exchange it for a
short-lived Agent session token and give the browser a callback that obtains
one for each connection:

```ts
import { WaffleClient } from '@animated-waffle/client'

const agentId = '22222222-2222-4222-8222-222222222222'
const endUserId = '55555555-5555-4555-8555-555555555555'
const waffle = new WaffleClient({
  agentId,
  // Set false for text-triggered, output-only applications.
  inputAudioEnabled: false,
  sessionContext: {
    userName: ' Space é ',
    timeZone: 'Asia/Shanghai',
  },
  sessionToken: async () => {
    const response = await fetch('/api/narya-session-token', {
      method: 'POST',
      headers: { 'Content-Type': 'application/json' },
      body: JSON.stringify({ agentId, endUserId }),
    })
    const body = await response.json()
    return body.token
  },
})

waffle.on('transcript', ({ id, role, text, final }) => {
  // Completed Narya conversation rows use the same id as persisted history.
  if (final) console.log(id, role, text)
})
waffle.on('transcriptProgress', ({ segmentId, role, text, complete }) => {
  // Ephemeral ASR/playout caption; never persist segmentId as a message id.
})

await waffle.connect()
```

`endUserId` is one stable bare UUID per end user. An anonymous application can
generate it once with `crypto.randomUUID()` and persist it on that device.

## Creator Developer Feedback

Developer Feedback is opt-in and unavailable unless the application supplies
a callback to its own creator-authenticated backend. The SDK invokes it lazily
when feedback is opened or saved, after the API has returned the persisted
source Session id:

```ts
const waffle = new WaffleClient({
  agentId,
  sessionToken: () => getSessionToken(agentId),
  developerFeedback: ({ agentId, sourceSessionId, signal }) =>
    fetch('/api/narya-developer-feedback-grant', {
      method: 'POST',
      headers: { 'Content-Type': 'application/json' },
      body: JSON.stringify({ agentId, sourceSessionId }),
      signal,
    }),
})
```

That backend response uses the public grant shape
`{ endpoint, token, expires_at, source_session_id }`. It must be protected as a
creator development route and mint the grant with a server-side workspace key;
never expose that key to the browser. The feedback endpoint receives only the
five-minute grant with `credentials: 'omit'`. It cannot use the ordinary Agent
session token. Non-loopback SDK development origins must be included in the
API's configured origin allowlist.

`loadDeveloperFeedback()` reads the current private draft.
`submitDeveloperFeedback()` appends one Preserve/Change observation for the
whole Session or a persisted Assistant message. It never submits the round,
generates a revision, or publishes the Agent. Reuse an explicit feedback `id`
when retrying after an ambiguous network failure.

To target a non-production environment:

```ts
const waffle = new WaffleClient({
  agentId,
  sessionToken: () => getSessionToken(agentId),
  apiOrigin: 'https://animated-waffle-dev.narya.ai',
})
```

## API surface

- `sessionContext?: { userName?: string; timeZone?: string }` — available only
  with the default `sessionToken` starter. Nonblank values are sent exactly as
  provided; blank values are omitted. It is mutually exclusive with the
  advanced `startSession` callback, so token-path consumers do not recreate
  the raw managed-session request.
- `reportedSource?: 'Web' | 'SDK' | 'Client' | 'Unknown'` — self-reported
  integration family for observability only. The SDK defaults to `SDK`.
- `connect({ inputMode?, sessionId?, overrides? })` — `'push-to-talk'`
  (default) or `'automatic'`; resolves only when the Agent runtime is ready.
- `session.agentStudioUrl?` — server-authored dashboard target for the exact
  Agent and source Session. Development UI may show “Open Agent in Studio”
  when present; do not rebuild it from `avatarId`. Older protocol-v2 servers
  may omit it during rolling upgrades.
- `inputAudioEnabled: false` — never requests or enables the browser microphone;
  text input, Agent audio, transcripts, and reconnects continue to work. Push-
  to-talk controls fail closed with `input_audio_disabled`. The default is
  `true`, enabled only after the Agent is ready.
- `muted` — silences the Agent's voice while it keeps performing: the session,
  the transcript and the Avatar's cues carry on, only playback goes quiet.
- `audioPlayback: false` — the client builds no audio element of its own. Play
  `remoteAudioTrack` yourself; an application that routes the voice through its
  own element, mixer or Web Audio graph must turn this off, or the voice is
  heard twice. `muted` then has nothing to do.
- `disconnect()`, `sendText(text, { audioResponse? })`. `audioResponse` defaults
  to `true`; `false` keeps the typed turn and transcript but suppresses reply
  audio for that turn.
- `loadDeveloperFeedback()`, `submitDeveloperFeedback(input)` — available only
  with the explicit creator BFF callback above.
- `startPushToTalk()` / `commitPushToTalk()` / `cancelPushToTalk()`.
- `on('connectionState' | 'session' | 'remoteAudioTrack' | 'transcriptProgress' | 'transcript' |
  'agentSpeaking' | 'declined' | 'error', fn)` — `declined` carries the
  operator-facing reason for deliberate silence. The call returns
  the unsubscribe function.
- Conversation transcripts are completed persisted rows, carry the same required
  `id` as history, and currently always have `final: true`. Provider
  sentence/progress framing never defines a public conversation row.
- Transcript progress is ephemeral and keyed by a runtime `segmentId`; user ASR
  and assistant playout update it incrementally until `complete` closes it.
- `connectionState`, `session`, `remoteAudioTrack` getters; `status` is the
  deprecated 0.1 alias. Failures throw `WaffleError` with `code` and a typed
  `stage`: `session`, `transport`, `audio`, `transcript`, `avatar-load`, or
  `avatar-renderer`. A failed `connect()` emits and rejects with the same error
  instance.

`./internal` is a private contract consumed by `@animated-waffle/react`; it is
not part of the public API and may change without notice.

See the [SDK documentation](https://animated-waffle.narya.ai/docs/sdk) for the
complete integration guide.

## Application sessions

For an application-owned conversation, use `application: true` with a
`startSession` callback returning the media session created by
`WaffleServer.startApplicationSession`. The backend retains the separate control
grant; the browser never executes tools or forwards privileged callbacks.
See the [application-session contract](../../docs/public/sdk.md#application-owned-tools-and-conversation-state).

## Development diagnostics

`client.diagnostics.server.waitDetails` splits input handoff to LLM start (negative when preemptive), first token to TTS request (sentence buffering plus connection), TTS connection, first PCM to worker audio sink, and the original PCM's leading-silence estimate. Automatic ASR finalization and end-of-turn decision are separate `server.asrFinalizationMs` and `server.endOfTurnDecisionMs` fields. Older workers omit these fields.

For RTC playback, `client.diagnostics.turn.transport` contains the selected browser-to-LiveKit ICE RTT, jitter-buffer/target and device-playout averages over the first 250 ms after detected playback, and `sampleWindowMs`. Sampling completes asynchronously and preserves the original playback timestamp; audio never waits for diagnostics. Unsupported, reset or stale counters stay `null`. These are overlapping diagnostic samples, not additive one-way network latency; server and client observations may describe different inputs. They contain no audio, transcripts, device identifiers or network addresses. The built-in Web/React diagnostics panel renders the same fields.

Read `client.diagnostics` and subscribe with `client.on('diagnostics', render)`.
React returns the same snapshot as `waffle.diagnostics`. The React `useWaffle` and
web `mountWaffleAvatar` adapters show a collapsed floating diagnostics panel in
development builds; production defaults off. Their `diagnostics` option overrides
visibility. For an inline view, use `WaffleDiagnosticsPanel` or
`mountWaffleDiagnostics(host, client)` and disable the automatic overlay.
The headless client never adds UI. Mounting a panel never starts a connection or microphone.

The panel shows connection/media/Avatar state, connection phases, latest client
input-end-to-playback estimate, safe errors and the current Session Studio link.
Updated workers additionally report ASR, LLM, TTS and worker-output timings using
Studio's reducer. Client and server rows have separate correlation IDs and clocks;
do not sum the stages or treat a worker output as browser playback. Missing data
remains unavailable. Voice input activity and RTC/paired output timing are estimates,
not calibrated acoustic measurements. Snapshots exclude text, audio, tokens and raw
error messages. The panel is optional; existing timing telemetry is independent of it.

## Microphone capture

`audioCapture` accepts native LiveKit capture constraints (excluding custom processors).
They apply before the first microphone opens, and after reconnecting. They stay local
to the client; authentication, billing and server voice routing are unchanged.

```ts
const client = new WaffleClient({
  agentId,
  sessionToken: () => fetchSessionToken(),
  // Only for an isolated virtual input or audio that already has echo processing.
  audioCapture: { echoCancellation: false },
})
```

Keep echo cancellation enabled for normal speaker/microphone setups. Disabling it
can feed the Agent's voice back into recognition and trigger unwanted interruptions.
No device-name detection or automatic processing changes are applied.
Inspect `client.microphoneTrack?.getSettings()` after connecting to check what the
browser actually enabled; changing an already-open track may be rejected.
`useWaffle` accepts the same option. Equivalent inline constraints retain the
connection; changing their values recreates the client. Capture options do not
change LiveKit's server-side interruption warmup or end-of-turn settings.