--- name: write-fixtures description: Use when writing test fixtures for @copilotkit/aimock — mock LLM responses, tool call sequences, error injection, multi-turn agent loops, embeddings, structured output, sequential responses, or debugging fixture mismatches --- # Writing aimock Test Fixtures ## What aimock Is aimock is a zero-dependency mock infrastructure for AI apps. Fixture-driven. Multi-provider (OpenAI, Anthropic, Gemini, Gemini Interactions, AWS Bedrock, Azure OpenAI, Vertex AI, Ollama, Cohere, OpenRouter). Multimedia endpoints (image generation, text-to-speech, audio transcription, video generation). MCP, A2A, AG-UI, and vector DB mocking. Runs a real HTTP server on a real port — works across processes, unlike MSW-style interceptors. WebSocket support for OpenAI Responses/Realtime and Gemini Live APIs. Record-and-replay for all endpoints including multimedia. Chaos testing and Prometheus metrics. ## Core Mental Model - **Fixtures** = match criteria + response - **First-match-wins** — order matters - All providers share one fixture pool (provider adapters normalize to `ChatCompletionRequest`) - Fixtures are live — mutations after `start()` take effect immediately - Sequential responses are supported via `sequenceIndex` (match count tracked per fixture) ## Match Field Reference | Field | Type | Matches Against | | -------------------- | ----------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `userMessage` | `string` | Substring of last `role: "user"` message text | | `userMessage` | `RegExp` | Pattern test on last `role: "user"` message text | | `systemMessage` | `string` | Substring of the concatenated text of every `role: "system"` message in the request. Use to gate a fixture on host-supplied context (persona, agent-context entries) so changes to that context cause the fixture to fall through instead of returning a stale baked response | | `systemMessage` | `string[]` | Array of substrings — ALL must be present in the joined system text (AND semantics). Use when the gate must combine multiple non-adjacent tokens whose serialisation order isn't stable | | `systemMessage` | `RegExp` | Pattern test on the concatenated system-message text | | `inputText` | `string` | Substring of embedding input text (concatenated if multiple inputs) | | `inputText` | `RegExp` | Pattern test on embedding input text | | `embeddingRequest` | `{ operation, input, dimensions? }` | Exact operation (`/api/embed`, `/api/embeddings` or `/v1/embeddings`), input shape and requested dimensions. Recordings set it alongside `inputText` and `model` so batches and widths stay distinct; omit it for broad embedding fixtures | | `toolName` | `string` | Exact match on any tool in request's `tools[]` array (by `function.name`). On the OpenAI Responses API, with `responsesTools: "extended"`, this also includes tools inside a `namespace` tool, `custom` tools (by `name`, in `req.customTools`), and tools in an `additional_tools` or `tool_search_output` input item; by default only top-level function tools, as in 1.44.0. Other Responses tool types (`web_search`, `mcp`, ...) are not visible | | `toolNamespace` | `string` | Exact OpenAI Responses tool namespace; needs `responsesTools: "extended"` (ignored otherwise). Alone: any offered tool sits in it. With `toolName`: one offered tool must carry both (Codex routes by this exact pair). No default namespace is assumed. Chat Completions passes request tools through, so a client-sent top-level `namespace` there also matches. With `"extended"` (`aimock validate --responses-tools extended`), `aimock validate` and `--validate-on-load` warn about a file fixture with `toolNamespace` and an `endpoint` other than `"chat"`. A Responses `namespace` tool with an empty or missing `name` is dropped. Codex names an MCP server's namespace `mcp__` (no trailing `__`). aimock-pytest needs aimock 1.45.0+, its default pin from 0.7.0 | | `toolCallId` | `string` | Exact match on `tool_call_id` of last `role: "tool"` message | | `toolResultContains` | `string` | Substring of the last tool message's text content, gated on that message being the request's LAST message (same rule as `toolCallId`). Discriminates resume paths that share a `tool_call_id` and differ only inside the tool-result payload (e.g. approve `{"chosen_time": …}` vs cancel `{"cancelled": true}`) | | `model` | `string` | Exact match on `req.model` | | `model` | `RegExp` | Pattern test on `req.model` | | `responseFormat` | `string` | Exact match on `req.response_format.type` (`"json_object"`, `"json_schema"`) | | `sequenceIndex` | `number` | Matches only when this fixture's match count equals the given index (0-based) | | `turnIndex` | `number` | Stateless conversation-depth matching. Counts `role: "assistant"` messages in the request; matches when that count equals the value. `turnIndex: 0` = first turn (no prior assistant messages). Use instead of `sequenceIndex` for shared/deployed instances where stateful counters break under concurrency | | `hasToolResult` | `boolean` | Stateless tool-message presence matching, scoped to the CURRENT turn (messages after the last `role: "user"` message). `true` matches when a `role: "tool"` message appears after the last user message; `false` matches when none does. (If the request has no user message, the whole conversation is scanned.) Provider-consistent across all aimock handlers (OpenAI, Claude, Gemini, Bedrock, Ollama, Cohere) | | `endpoint` | `string` | Restrict to endpoint type: `"chat"`, `"image"`, `"speech"`, `"transcription"`, `"video"`, `"embedding"` | | `predicate` | `(req: ChatCompletionRequest) => boolean` | Custom function — full access to request | **AND logic**: all specified fields must match. Empty match `{}` = catch-all. Multi-part content (e.g., `[{type: "text", text: "hello"}]`) is automatically extracted — `userMessage` matching works regardless of content format. ### When to Use Each Multi-turn Matching Approach | Approach | Stateless? | Best For | | -------------------- | ---------- | ------------------------------------------------------------------------------------------------------------------------- | | `turnIndex` | Yes | Shared/deployed instances; matches on conversation depth (count of assistant messages in request) | | `hasToolResult` | Yes | Simplest option for 2-step tool flows — boolean: does the current turn (after the last user message) carry a tool result? | | `sequenceIndex` | No | Single-client unit tests with repeated identical requests (server-side counter, breaks under concurrency) | | `toolCallId` | Yes | Matching specific tool result IDs in the conversation history | | `toolResultContains` | Yes | Same tool call id, different outcomes — match on the tool-result payload (approve vs cancel legs) | **Prefer stateless approaches** (`turnIndex`, `hasToolResult`, `toolResultContains`) for shared aimock instances (deployed via Docker, used by multiple test runners). Use `sequenceIndex` only in isolated single-client unit tests where the counter won't be corrupted by concurrent requests. ### Multi-turn fixture examples ```jsonc // 2-step HITL with turnIndex {"match": {"userMessage": "trip to mars", "turnIndex": 0}, "response": {"toolCalls": [{"id": "call_001", "name": "generate_steps", "arguments": "{}"}]}} {"match": {"userMessage": "trip to mars", "turnIndex": 1}, "response": {"content": "Great choices! Proceeding."}} // Same thing with hasToolResult (simpler for 2-step) {"match": {"userMessage": "trip to mars", "hasToolResult": false}, "response": {"toolCalls": [{"id": "call_001", "name": "generate_steps", "arguments": "{}"}]}} {"match": {"userMessage": "trip to mars", "hasToolResult": true}, "response": {"content": "Great choices!"}} // HITL suspend tool where approve and cancel resume with the SAME tool call id — // discriminate on the tool-result payload; put the cancel leg first (first match wins) {"match": {"toolCallId": "call_001", "toolResultContains": "\"cancelled\""}, "response": {"content": "No problem — nothing was booked."}} {"match": {"toolCallId": "call_001"}, "response": {"content": "Booked: Monday 9:00 AM confirmed."}} ``` ## Response Types ### Text ```typescript { content: "Hello!"; } ``` ### Tool Calls ```typescript // Preferred: object form (auto-stringified by the fixture loader) { toolCalls: [{ name: "get_weather", arguments: { city: "SF" } }]; } // Also accepted: JSON string form (backward compatible) { toolCalls: [{ name: "get_weather", arguments: '{"city":"SF"}' }]; } ``` **Both object and string forms are accepted** for `arguments`. The fixture loader auto-stringifies objects via `JSON.stringify()`. Object form is preferred for readability. **OpenAI Responses API only** — namespaced and custom tool calls (used by Codex for MCP tools, sub-agents and `apply_patch`). Start the server with `responsesTools: "extended"` (`--responses-tools extended`): it makes namespaced, custom and `additional_tools` tools visible to `toolName` and predicates, counts custom tool rounds for `hasToolResult` / `toolCallId` / `turnIndex`, emits a `toolCalls` entry's `namespace`, and records namespaces and custom calls. The default (`"legacy"`) behaves exactly like 1.44.0. `toolNamespace`, `customToolCalls` and `responsesBlocks` also need `"extended"`; the default ignores them. ```typescript const response = { toolCalls: [ // function call to a tool offered inside a `namespace` tool { name: "spawn_agent", namespace: "collaboration", arguments: JSON.stringify({ message: "..." }), }, ], // custom (freeform) tool calls: free-text `input`, never parsed or stringified, // emitted after the function calls. A custom-only turn sets `toolCalls: []`. customToolCalls: [{ name: "apply_patch", input: "*** Begin Patch\n...\n*** End Patch\n" }], }; ``` A `toolCalls` entry is always a function call; a `type` or `input` on it is ignored (as in 1.44.0). To interleave custom calls with text or function calls, use `responsesBlocks` (the `blocks` entries plus `{ "type": "customToolCall", "name", "input", "id"?, "namespace"? }`); a fixture may not set both `blocks` and `responsesBlocks`. Every other API (Chat Completions, Anthropic Messages, Gemini, Gemini Interactions, Bedrock, Ollama, Cohere, Realtime, Gemini Live) ignores `namespace` and rejects a fixture with custom calls before any content with code `aimock_unsupported_tool_call` (Ollama `/api/generate` keeps its HTTP 400 for every tool-call fixture); a `responsesBlocks` without custom calls is served there as `blocks`. A malformed `customToolCalls` entry or `responsesBlocks` tool block fails validation (`aimock validate`, `--validate-on-load`, `addFixturesFromJSON()`, `POST /__aimock/fixtures`); in a programmatic, factory or unvalidated file fixture it fails the Responses request with code `aimock_invalid_fixture_tool_call`. Fixtures that 1.44.0 accepted get no new validation finding and no new error. Where the code goes: `error.code` (OpenAI Chat / Responses, Azure, Cohere, Anthropic, Gemini Interactions); `error.metadata.reason` (OpenRouter Chat); a `google.rpc.ErrorInfo` detail (Gemini, Vertex AI, Gemini Live error frame with `code` 13); `reason` (Bedrock); `code` next to `error` (Ollama `/api/chat`); an `error` event's `error.code` (Responses WebSocket); `response.done` with `status: "failed"` and `status_details.error.code` (Realtime). For Codex, gate the turn-0 fixture with `hasToolResult: false` and the follow-up with `hasToolResult: true`, or the turn-0 fixture matches again and the agent loops. A fixture that matches only on `toolName` / `toolNamespace` loops the same way, because Codex offers the same tools on every request. ### Blocks (ordered text / tool-call streaming) The optional `blocks` array expresses an explicit, ordered sequence of stream entries — something plain `content` + `toolCalls` cannot, since those imply text-then-tools. Each entry is `{ "type": "text", "text": "..." }`, `{ "type": "toolCall", "name": "...", "arguments": "...", "id"?: "...", "namespace"?: "..." }`, streamed in array order (OpenAI Responses `responsesBlocks` also take `customToolCall` blocks). This enables tool-first ordering (a tool call before any text) and interleaved text/tool ordering. ```typescript // Tool-first: tool call streams before the text { blocks: [ { type: "toolCall", name: "get_weather", arguments: { city: "SF" } }, { type: "text", text: "Checking the weather for you…" }, ]; } ``` When `blocks` is present it takes **precedence over `content`/`toolCalls`** for stream order; when absent, legacy behavior is unchanged. blocks-only fixtures are first-class — a response may be just `{ blocks: [...] }` with no `content` and no `toolCalls`, and builders derive the aggregate `content`/`tool_calls` from the blocks. A toolCall block's `arguments` may be a JSON object or a string (objects auto-stringify), exactly like top-level `toolCalls`. Replay caveat: block order is observable on some providers and not others — see the [per-provider observability matrix](../../docs/fixtures/index.html#ordered-blocks). ### Embedding ```typescript { embedding: [0.1, 0.2, 0.3, -0.5, 0.8]; } ``` The embedding vector is returned for each input in the request. If no embedding fixture matches, deterministic embeddings are auto-generated from the input text hash — you only need fixtures when you want specific vectors. ### Image ```typescript // Single image { image: { url: "https://example.com/generated.png" } } // Multiple images { images: [{ url: "https://example.com/1.png" }, { b64Json: "iVBOR..." }] } ``` Use `match: { endpoint: "image" }` to prevent cross-matching with chat fixtures. ### Speech (TTS) ```typescript { audio: "base64-encoded-audio-data" } // With explicit format (default: mp3) { audio: "base64-data", format: "opus" } ``` ### Transcription ```typescript // Simple { transcription: { text: "Hello world" } } // Verbose with timestamps { transcription: { text: "Hello world", language: "en", duration: 2.5, words: [...], segments: [...] } } ``` ### Video ```typescript { video: { id: "vid-1", status: "completed", url: "https://example.com/video.mp4" } } ``` Video uses async polling — `POST /v1/videos` creates, `GET /v1/videos/{id}` checks status. ### Error ```typescript { error: { message: "Rate limited", type: "rate_limit_error" }, status: 429 } ``` ### Raw passthrough (provider events as written) Prefer the typed responses above: they work on every provider and are drift-checked. Use `raw` only for an event, field or error shape aimock does not model yet (for example a new Responses event type). aimock sends `events` (streaming) or `json` (non-streaming) exactly as written and adds only the named wire's framing. ```typescript { raw: { wire: "openai-responses", // required; the fixture matches only this wire events: [ { type: "response.output_text.annotation.added", annotation: { type: "url_citation", url: "https://example.com" }, }, ], json: { id: "resp_1", object: "response", status: "completed", output: [] }, status: 200, // optional, 200-599, not 204/304 headers: { "x-request-id": "req_1" }, // optional; no content-length/transfer-encoding/connection/content-encoding }, } ``` - `wire` is one of `openai-responses`, `openai-chat`, `anthropic`, `gemini`, `gemini-interactions`, `cohere`, `ollama` (`/api/chat`), `ollama-generate` (`/api/generate`), `bedrock-invoke`, `bedrock-converse`. The router skips a raw fixture on every other wire and keeps matching; only a factory that returns a raw fixture for another wire gets a 500. - Events on `openai-responses`, `anthropic` and `cohere` need a string `type`; a `bedrock-converse` event is `{ "": { …payload } }`. - `done: false` (openai-chat only, a boolean) omits `data: [DONE]`. - Unless `headers` sets `Content-Type`, a stream gets the typed stream's default headers for its wire (`text/event-stream`, `application/x-ndjson` on Ollama, `application/vnd.amazon.eventstream` on Bedrock) and a non-streaming reply gets `application/json` on every wire. An authored header replaces the default of the same name in any letter case. - An error reply (`status` >= 400) is never delayed, streaming or not. With `json`, it answers JSON even to a streaming request. - A request whose mode has no body (`events` for streaming, `json` for non-streaming) gets the wire's 500 with `aimock_raw_mode_unavailable`. An invalid raw fixture registered with `on()` or a factory is not checked when added; it gets the same 500 with `aimock_raw_fixture_invalid`, as does a factory result for another wire. Every raw fixture is checked again when it is served. `--validate-on-load` reports invalid fixtures at load. Also invalid: events that are not plain JSON (holes, `undefined`, non-finite numbers, Dates, cycles), a CR or LF in a `type`, `customToolCalls` or `responsesBlocks` next to `raw`, and a raw fixture with its own `misbehavior`. Unknown keys inside `raw` are a warning. - `chunkSize` does not apply (each event is one frame). Misbehavior faults skip raw fixtures: a server-level fault is journaled as skipped, an `X-AIMock-Misbehavior` header returns 501 `aimock_misbehavior_not_applicable`, and a fixture's own `misbehavior` makes it invalid (load error, 500 `aimock_raw_fixture_invalid`). Streaming timing, `truncateAfterChunks`, `disconnectAfterMs` and chaos apply. - Non-streaming timing: `nonStreamingTiming: true` delays a raw `stream: false` reply on `openai-chat` only; `nonStreamingTimingAllProviders: true` on every raw wire. Either needs `streamingProfile` or `recordedTimings`; the wait is sized from the served JSON. - OpenRouter `models` fallback: an `openai-chat` raw fixture with `status` >= 400 fails over like a typed error fixture. - Not served over WebSocket Responses yet (error event with code `aimock_raw_mode_unavailable`). Docs: `/raw-passthrough`. ### Chaos (Failure Injection) The optional `chaos` field on a fixture enables probabilistic failure injection: ```typescript { chaos?: { dropRate?: number; // Probability (0-1) of returning a 500 error malformedRate?: number; // Probability (0-1) of returning malformed JSON disconnectRate?: number; // Probability (0-1) of disconnecting mid-stream } } ``` Rates are evaluated per-request. When triggered, the chaos failure replaces the normal response. ### Bad model output and retry recovery Use `misbehavior` to change a valid fixture response. Keep `chaos` for transport failures. Semantic fault support varies by wire, fault, and mode; consult the [support matrix](../../docs/model-misbehavior/index.html#support) before authoring a fixture. The example below uses `openai-chat`, including Azure and OpenRouter chat endpoints. Local support does not establish native fidelity; preserve the [modeled contracts and native capture limits](../../docs/model-misbehavior/index.html#fidelity). Save this fixture as `bad-tool.json`. Its first matching request receives invalid tool JSON; the next receives valid arguments. ```json { "fixtures": [ { "match": { "userMessage": "weather", "endpoint": "chat" }, "response": { "toolCalls": [{ "id": "call_weather", "name": "weather", "arguments": { "city": "Paris" } }] }, "misbehavior": { "seed": 42, "faults": [ { "fault": "tool-args-invalid-json", "style": "trailing-comma", "times": 1, "providers": ["openai-chat"] } ] } } ] } ``` Check application parsing and recovery explicitly. An HTTP 200 with bad arguments does not trigger the SDK's HTTP retry logic. For Vitest, use the real local server and official SDK: ```typescript import OpenAI from "openai"; import { LLMock } from "@copilotkit/aimock"; import { expect } from "vitest"; const mock = new LLMock(); mock.loadFixtureFile("bad-tool.json"); await mock.start(); try { const client = new OpenAI({ apiKey: "local", baseURL: `${mock.url}/v1`, maxRetries: 0, defaultHeaders: { "X-Test-Id": "retry-tool" }, }); const request = { model: "gpt-4", messages: [{ role: "user", content: "weather" }], } as const; const first = await client.chat.completions.create({ ...request, messages: [...request.messages], }); const bad = first.choices[0].message.tool_calls?.[0].function.arguments; expect(bad).toBeDefined(); expect(() => JSON.parse(bad!)).toThrow(SyntaxError); const second = await client.chat.completions.create({ ...request, messages: [...request.messages], }); expect(JSON.parse(second.choices[0].message.tool_calls![0].function.arguments)).toEqual({ city: "Paris", }); expect(mock.journal.getAll()[0].response.misbehavior).toMatchObject({ applied: true, source: "fixture", wire: "openai-chat", fault: "tool-args-invalid-json", }); } finally { await mock.stop(); } ``` Keep the same test ID for both attempts. `times` counts firings per source, entry and test ID, not matching requests. Use `mock.resetMatchCounts("retry-tool")` to restart that test's counters. Clearing journal entries alone preserves fault counters. A configuration accepts a fault ID string or `{ seed, faults }`. Use `{ faults: [] }` for an explicit opt-out. Only `POST /__aimock/misbehavior` accepts bare `{}` as an opt-out. Each entry accepts `rate` from 0 to 1, positive integer `times`, a `tool` selector and a `providers` wire list. The default rate is 1. An omitted `times` has no firing limit. The default seed is 0; `"random"` selects one process seed. | Fault | Fault-specific configuration | | ---------------------------- | ----------------------------------------------------------------------------------------------------------------------------- | | `tool-args-invalid-json` | `style`: `truncated` (default), `trailing-comma`, `single-quotes` | | `tool-args-schema-violation` | `violation`: `missing-required` (default), `wrong-type`, `extra-property`, `enum-mismatch`, `not-object`; optional `property` | | `tool-unknown-name` | Optional `name` | | `tool-call-id-duplicate` | No extra parameters; requires at least one call on an ID-emitting wire; duplicates the sole call when only one exists | | `stop-length-mid-tool` | `at` strictly between 0 and 1; default 0.5; emits a clean `length` stop with partial arguments | | `empty-response` | No extra parameters | | `refusal` | Optional `message`; keep `category` null or omitted on OpenAI Chat | | `content-filter` | No extra parameters | | `reasoning-only` | Optional `reasoning`; OpenRouter emits `reasoning` with `reasoning_details`; native OpenAI exposes no reasoning | For schema violations, send the tool schema in the request. Use direct schema constraints; do not assume `$ref` or combinator support. Selection order is HTTP header, fixture, runtime scope, then server baseline. Selected configurations replace one another; they do not merge. Fixture and header faults that cannot apply fail loudly. Runtime and server faults that cannot apply are skipped. Bad fixture configuration throws `FixtureLoadError` with a `misbehavior/` rule, including CLI loads without `--validate-on-load`. Bad server configuration or setter input throws a rule-prefixed `TypeError`. Check `response.misbehavior.evaluations` in the journal for skips, exhausted budgets and provider exclusions. Check actual SDK output too: `servedToolCalls` describes the prepared response, not proof of network delivery. See the [control API](https://aimock.copilotkit.dev/control-api#misbehavior) for runtime scope, validation errors and header grammar. ## Common Patterns ### Basic text fixture ```typescript mock.onMessage("hello", { content: "Hi there!" }); ``` ### Tool call → tool result → final response (3-step agent loop) The most common pattern. Fixture 1 triggers the tool call, fixture 2 handles the tool result. ```typescript // Step 1: User asks about weather → LLM calls tool mock.onMessage("weather", { toolCalls: [{ name: "get_weather", arguments: { city: "SF" } }], }); // Step 2: Tool result comes back → LLM responds with text mock.addFixture({ match: { predicate: (req) => req.messages.at(-1)?.role === "tool" }, response: { content: "It's 72°F in San Francisco." }, }); ``` **Why predicate, not userMessage?** After a tool call, the client replays the same conversation with the tool result appended. The user message hasn't changed — `userMessage: "weather"` would match the SAME fixture again, creating an infinite loop. ### Embedding fixture ```typescript // Match specific input text mock.onEmbedding("search query", { embedding: [0.1, 0.2, 0.3, 0.4, 0.5], }); // Match with regex mock.onEmbedding(/product.*description/, { embedding: [0.9, -0.1, 0.5, 0.3, 0.2], }); ``` ### Structured output / JSON mode ```typescript // onJsonOutput auto-sets responseFormat: "json_object" and stringifies objects mock.onJsonOutput("extract entities", { entities: [ { name: "Acme Corp", type: "company" }, { name: "Jane Doe", type: "person" }, ], }); // Equivalent manual form: mock.addFixture({ match: { userMessage: "extract entities", responseFormat: "json_object" }, response: { content: '{"entities":[...]}' }, }); ``` ### Sequential responses (same match, different responses) ```typescript // First call returns tool call, second returns text mock.on( { userMessage: "status", sequenceIndex: 0 }, { toolCalls: [{ name: "check_status", arguments: {} }] }, ); mock.on({ userMessage: "status", sequenceIndex: 1 }, { content: "All systems operational." }); ``` Match counts are tracked per fixture group. Use `resetMatchCounts()` between tests to reset counts while keeping loaded fixtures. `reset()` also clears the fixture pool, so avoid it between tests that share a loaded fixture set. ### Streaming physics (realistic timing) ```typescript mock.onMessage( "tell me a story", { content: "Once upon a time..." }, { streamingProfile: { ttft: 200, // 200ms before first token tps: 30, // 30 tokens per second after that jitter: 0.1, // ±10% random variance }, }, ); ``` ### Predicate-based routing (same user message, different context) Common in supervisor/orchestrator patterns where the system prompt changes: ```typescript mock.addFixture({ match: { predicate: (req) => { const sys = req.messages.find((m) => m.role === "system")?.content ?? ""; return typeof sys === "string" && sys.includes("Flights found: false"); }, }, response: { toolCalls: [{ name: "search_flights", arguments: {} }] }, }); ``` ### Catch-all (always add one) Prevents unmatched requests from returning 404 and crashing the test: ```typescript mock.addFixture({ match: { predicate: () => true }, response: { content: "I understand. How can I help?" }, }); ``` ### Tool result catch-all with prependFixture Must go at the front so it matches before substring-based fixtures: ```typescript mock.prependFixture({ match: { predicate: (req) => req.messages.at(-1)?.role === "tool" }, response: { content: "Done!" }, }); ``` ### Stream interruption simulation (v1.3.0+) ```typescript mock.onMessage( "long response", { content: "This will be cut short..." }, { truncateAfterChunks: 3, // Stop after 3 SSE chunks disconnectAfterMs: 500, // Or disconnect after 500ms }, ); ``` ### Chaos testing (probabilistic failures) ```typescript mock.addFixture({ match: { userMessage: "flaky" }, response: { content: "Sometimes works!" }, chaos: { dropRate: 0.3 }, }); ``` 30% of requests matching this fixture will get a 500 error instead of the response. Can also use `malformedRate` (garbled JSON) or `disconnectRate` (connection dropped mid-stream). Server-level chaos applies to ALL requests: ```typescript mock.setChaos({ dropRate: 0.1 }); // 10% of all requests fail mock.clearChaos(); // Remove server-level chaos ``` ### Error injection (one-shot) ```typescript mock.nextRequestError(429, { message: "Rate limited", type: "rate_limit_error" }); // Next request gets 429, then fixture auto-removes itself ``` ### JSON fixture files ```json { "fixtures": [ { "match": { "userMessage": "hello" }, "response": { "content": "Hi!" } }, { "match": { "userMessage": "weather" }, "response": { "toolCalls": [ { "name": "get_weather", "arguments": { "city": "SF", "units": "fahrenheit" } } ] } }, { "match": { "inputText": "search query" }, "response": { "embedding": [0.1, 0.2, 0.3] } }, { "match": { "userMessage": "status", "sequenceIndex": 0 }, "response": { "content": "First response" } } ] } ``` **JSON auto-stringify**: In JSON fixture files, `arguments` and `content` can be objects — the loader auto-stringifies them with `JSON.stringify()`. This also applies to a `blocks` entry's `arguments` — object form auto-stringifies just like top-level `toolCalls`. The escaped-string form (`"{\"city\":\"SF\"}"`) still works but objects are preferred for readability. JSON files cannot use `RegExp` or `predicate` — those are code-only features. `streamingProfile` is supported in JSON fixture files. Load with `mock.loadFixtureFile("./fixtures/greetings.json")` or `mock.loadFixtureDir("./fixtures/")`. ## API Endpoints All providers share the same fixture pool — write fixtures once, they work for any endpoint. | Endpoint | Provider | Protocol | | ---------------------------------------------------------------------------------------- | ------------- | --------- | | `POST /v1/chat/completions` | OpenAI | HTTP | | `POST /v1/responses` | OpenAI | HTTP + WS | | `POST /v1/messages` | Anthropic | HTTP | | `POST /v1/messages/count_tokens` | Anthropic | HTTP | | `POST /v1/embeddings` | OpenAI | HTTP | | `POST /v1beta/models/{model}:{method}` | Google Gemini | HTTP | | `POST /model/{modelId}/invoke` | AWS Bedrock | HTTP | | `POST /openai/deployments/{id}/chat/completions` | Azure OpenAI | HTTP | | `POST /openai/deployments/{id}/embeddings` | Azure OpenAI | HTTP | | `GET /health` | — | HTTP | | `GET /ready` | — | HTTP | | `POST /model/{modelId}/invoke-with-response-stream` | AWS Bedrock | HTTP | | `POST /model/{modelId}/converse` | AWS Bedrock | HTTP | | `POST /model/{modelId}/converse-stream` | AWS Bedrock | HTTP | | `POST /v1/projects/{p}/locations/{l}/publishers/google/models/{m}:generateContent` | Vertex AI | HTTP | | `POST /v1/projects/{p}/locations/{l}/publishers/google/models/{m}:streamGenerateContent` | Vertex AI | HTTP | | `POST /api/chat` | Ollama | HTTP | | `POST /api/generate` | Ollama | HTTP | | `GET /api/tags` | Ollama | HTTP | | `POST /api/embed` · `/api/embeddings` | Ollama | HTTP | | `POST /v2/chat` | Cohere | HTTP | | `POST /api/v1/chat/completions` | OpenRouter | HTTP | | `GET /api/v1/models` · `/api/v1/key` · `/api/v1/credits` | OpenRouter | HTTP | | `GET /metrics` | — | HTTP | | `GET /v1/models` · `/v1/models/{id}` | OpenAI-compat | HTTP | | `GET /v1beta/models` · `/v1beta/models/{model}` | Google Gemini | HTTP | | `WS /v1/responses` | OpenAI | WebSocket | | `WS /v1/realtime` | OpenAI | WebSocket | | `WS /ws/google.ai...BidiGenerateContent` | Gemini Live | WebSocket | | `POST /v1/images/generations` | OpenAI | HTTP | | `POST /v1beta/models/{model}:predict` | Gemini Imagen | HTTP | | `POST /v1/audio/speech` | OpenAI | HTTP | | `POST /v1/audio/transcriptions` | OpenAI | HTTP | | `POST /v1/videos` | OpenAI | HTTP | | `GET /v1/videos/{id}` | OpenAI | HTTP | `GET /v1/models` returns the OpenAI-format list by default. With `policy.modelCatalog: "native"` (or `--policy modelCatalog=native`, or the `X-AIMock-Policy: modelCatalog=native` header), it returns the Anthropic list (native paging, `after_id` / `before_id`) when the request has `anthropic-version`, or `x-api-key` without `Authorization`; other callers still get the OpenAI format. `GET /v1/models/{id}` uses the same header selection without the opt-in. Catalogs list the fixture model IDs. Catalog and count routes (`count_tokens`, Gemini `:countTokens`) never proxy or record. Counts are a deterministic estimate of about 1 token per 4 characters of messages (plus tool definitions), minimum 1, and ignore fixture `usage`. ## Response Template Overrides Fixture responses can include optional override fields to control auto-generated envelope values. These are merged into the provider-specific response format (OpenAI, Claude, Gemini, Responses API). | Field | Type | Default | Description | | -------------------- | -------------------------------- | ------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `id` | string | auto-generated | Override response ID (e.g., `chatcmpl-custom`) | | `created` | number | `Date.now()/1000` | Override Unix timestamp | | `model` | string | echoes request | Override model name in response | | `usage` | object | zeroed | Override token counts: `{ prompt_tokens, completion_tokens, total_tokens }`. OpenAI Chat includes usage in response body; Responses API uses `response.usage`. When omitted, auto-computed from content length | | `finishReason` | string | `"stop"` / `"tool_calls"` | Override finish reason. Mappings: `stop` -> `end_turn` (Claude), `STOP` (Gemini); `tool_calls` -> `tool_use` (Claude), `FUNCTION_CALL` (Gemini); `length` -> `max_tokens` (Claude), `MAX_TOKENS` (Gemini); `content_filter` -> `SAFETY` (Gemini), `failed` (Responses API) | | `role` | string | `"assistant"` | Override message role | | `phase` | `"commentary" \| "final_answer"` | (omitted) | OpenAI Responses API (HTTP and WebSocket) only: labels each assistant `message` output item. Preserved in streaming added/done items, the completed response, and non-streaming output. Omitted by default; never inferred from content or finish reason. Does not label reasoning or tool-call items | | `systemFingerprint` | string | (omitted) | Add `system_fingerprint` to response | | `provider` | string | slug author | OpenRouter only: top-level serving-provider display name (default = the winning model slug's author). Override to assert who served the request | | `nativeFinishReason` | string | mirrors `finishReason` | OpenRouter only: the raw upstream `native_finish_reason` alongside the normalized `finish_reason` | | `usage.cost` | number | (omitted) | OpenRouter only: per-request `usage.cost` (scriptable — powers budget-guard tests). When set, `usage.cost_details` is emitted too. Never fabricated when omitted | | `usage.is_byok` | bool | (omitted) | OpenRouter only: emit `usage.is_byok`. Also `usage.prompt_tokens_details`, `usage.completion_tokens_details` — emitted only when set | Claude Messages and Bedrock Claude invoke also emit `usage.cache_read_input_tokens`, `usage.cache_creation_input_tokens` and a `usage.cache_creation` object (`ephemeral_5m_input_tokens`, `ephemeral_1h_input_tokens`) when a fixture supplies them. ### Example ```typescript mock.onMessage("hello", { content: "Hi!", model: "gpt-4-turbo-2024-04-09", usage: { prompt_tokens: 10, completion_tokens: 5, total_tokens: 15 }, systemFingerprint: "fp_abc123", }); ``` ### In JSON fixtures ```json { "match": { "userMessage": "hello" }, "response": { "content": "Hi!", "model": "gpt-4-turbo-2024-04-09", "usage": { "prompt_tokens": 10, "completion_tokens": 5, "total_tokens": 15 }, "systemFingerprint": "fp_abc123" } } ``` These fields map correctly across all provider formats — for example, `finishReason: "stop"` becomes `finish_reason: "stop"` in OpenAI, `stop_reason: "end_turn"` in Claude, and `finishReason: "STOP"` in Gemini. ## OpenRouter (chat / router) A request whose path starts with `/api/v1/` (point the OpenAI SDK at a `baseURL` ending `/api/v1`) is shaped as OpenRouter: `gen-` id, top-level `provider`, per-choice `native_finish_reason`, always-present `system_fingerprint`/`service_tier` (null by default), an always-present `message.reasoning` (null unless a fixture supplies reasoning and the model is reasoning-capable), and a rich `usage`. Requests on the plain `/v1/...` base are untouched OpenAI. Same fixture pool — the fields above are the only additions. - **Scriptable cost / provider / finish reason**: set `provider`, `nativeFinishReason`, and `usage.cost` on the response (see the overrides table). `cost`/`cost_details` are emitted only when a fixture supplies `cost` — aimock never fabricates a cost. - **`models[]` fallback (router failover)**: when the request body carries `models: [m1, m2, ...]`, aimock walks `[model, ...models]` in order and serves the first fixture that returns a NON-error response. A `429`/`503` error fixture on a candidate simulates a RUNTIME provider failure and falls through to the next candidate; the winning slug is echoed back as the top-level `model` (assert failover via `response.model`). Model the primary's "failure" as a 429/503 — an unknown/invalid model is just a fixture miss (aimock does not replicate OpenRouter's up-front invalid-model 400). - **Terminal (non-failover) error class — `fallthrough: false`**: real OpenRouter fails over inconsistently by error class (a `403 budget-exceeded` / generic "provider returned error" is served as terminal and does NOT advance to the next candidate, while 429/503 usually do). Set `fallthrough: false` on an error fixture to make it terminal: the fallback loop stops and serves that error even when a good candidate follows. Absent / `true` keeps the default fall-through. Composes with `provider.allow_fallbacks`: fall-through happens only when both allow it (if either says stop, the error is terminal). Use it to reproduce the exact provider error a dev's app must handle itself. `{ error: { message: "budget exceeded" }, status: 403, fallthrough: false }` - **Keepalive**: set the fixture option `openRouterProcessing: true` to emit one `: OPENROUTER PROCESSING` SSE comment before the first data frame (opt-in, default off). - **Error envelope**: OpenRouter errors are `{ error: { message, code } }` (numeric `code` == HTTP status), with optional free-form `metadata`. ```typescript // primary is a runtime 429, fallback answers mock.on( { model: "openai/gpt-4o", userMessage: "route" }, { error: { message: "rate limited" }, status: 429 }, ); mock.on( { model: "anthropic/claude-3.5-sonnet", userMessage: "route" }, { content: "served by the fallback", provider: "Anthropic", usage: { cost: 0.0021 }, }, ); // POST /api/v1/chat/completions { model: "openai/gpt-4o", models: [...], ... } // → response.model === "anthropic/claude-3.5-sonnet" ``` ## Provider Support Matrix | Feature | OpenAI Chat | OpenAI Responses | Claude | Gemini | Gemini Int. | Bedrock | Azure | Ollama | Cohere | OpenRouter | | -------------------- | ----------- | ---------------- | ------ | ------ | ----------- | ------- | ----- | ------ | ------ | ---------- | | Text | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | | Tool Calls | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | | Content + Tool Calls | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | | Streaming | SSE | SSE | SSE | SSE | SSE | Binary | SSE | NDJSON | SSE | SSE | | Reasoning | Yes | Yes | Yes | Yes | -- | Yes | Yes | -- | -- | Yes | | Web Searches | -- | Yes | -- | -- | -- | -- | -- | -- | -- | -- | | Response Overrides | Yes | Yes | Yes | Yes | Yes | -- | Yes | -- | -- | Yes | | Custom Tool Calls | Error | Yes | Error | Error | Error | Error | Error | Error | Error | Error | Custom tool calls (`customToolCalls` / `customToolCall` blocks in `responsesBlocks`): "Error" means the request fails before any content with code `aimock_unsupported_tool_call`. The Ollama column is `/api/chat`; `/api/generate` rejects every tool-call fixture, custom calls included, with HTTP 400 (`invalid_request_error`) and no aimock code. The Azure column is the deployment route (`/openai/deployments/{id}/chat/completions`) and the OpenRouter column is `/api/v1/chat/completions`. aimock serves `/openai/v1/responses` and OpenRouter's `/api/v1/responses` with its OpenAI Responses handler, so custom tool calls work there. Realtime and Gemini Live WebSockets also reject them. ## Critical Gotchas 1. **Order matters** — first match wins. Specific fixtures before general ones. Use `prependFixture()` to force priority. 2. **`arguments` accepts both objects and strings** — `"arguments": {"key":"value"}` (preferred, auto-stringified) or `"arguments": "{\"key\":\"value\"}"` (legacy). The same applies to `content` fields that contain JSON. The fixture loader detects `typeof === "object"` and calls `JSON.stringify()` automatically. 3. **Latency is per-chunk, not total** — `latency: 100` means 100ms between each SSE chunk, not 100ms total response time. It never delays a non-streaming (`stream: false`) response, also with `nonStreamingTiming: true` or `nonStreamingTimingAllProviders: true`. Similarly, `truncateAfterChunks` and `disconnectAfterMs` are for simulating stream interruptions (added in v1.3.0). 4. **`streamingProfile` takes precedence over `latency`** — when both are set on a fixture, `streamingProfile` controls timing. Use one or the other. By default, `streamingProfile` and `recordedTimings` delay only streamed replies: a `stream: false` reply returns immediately, also for recorded fixtures. Set `nonStreamingTiming: true` on the fixture to also delay `stream: false` OpenAI Chat Completions replies (text, tool calls, combined content and tool calls, and `blocks`) by the sum of the per-chunk delays a streamed reply at the same `chunkSize` would get. `nonStreamingTiming` does not delay other providers. To delay `stream: false` replies with the same formula on OpenAI Chat Completions, Anthropic Messages, OpenAI Responses, Gemini and Vertex AI `generateContent`, Gemini Interactions (`stream: false`), Cohere v2 chat, Bedrock InvokeModel and Converse, and Ollama `/api/chat` and `/api/generate`, set `nonStreamingTimingAllProviders: true` instead. Audio transcription replies are not delayed. Error replies are never delayed. If the client disconnects during the wait, nothing is written and the journal entry gets `interrupted: true`, `interruptReason: "client aborted"` and `status: 0`. 5. **Tool result messages don't change the user message** — after a tool call, the client sends the same conversation + tool result. Matching on `userMessage` will hit the SAME fixture again → infinite loop. Always use `predicate` checking `role === "tool"` for tool results. Note: a whole-conversation `role === "tool"` check (e.g. `req.messages.some((m) => m.role === "tool")`) diverges from `hasToolResult`'s current-turn scoping in multi-turn flows — the built-in `hasToolResult` matcher only looks after the last user message, so a later turn whose history carries an earlier tool result still reads `false`. 6. **`clearFixtures()` preserves the array reference** — uses `.length = 0`, not reassignment. The running server reads the same array object. 7. **Journal records everything** — including 404 "no match" responses. Use `mock.getLastRequest()` to debug mismatches. 8. **All providers share fixtures** — a fixture matching "hello" works whether the request comes via `/v1/chat/completions` (OpenAI), `/v1/messages` (Anthropic), Gemini, Bedrock, or Azure endpoints. 9. **WebSocket uses the same fixture pool** — no special setup needed for WebSocket-based APIs (OpenAI Responses WS, Realtime, Gemini Live). 10. **Embeddings auto-generate if no fixture matches** — deterministic vectors are generated from the input text hash. You don't need a catch-all for embedding requests. 11. **Sequential response counts are tracked per fixture** — use `resetMatchCounts()` between tests to reset counts while keeping loaded fixtures; `reset()` also clears the fixture pool, so don't use it between tests that share a loaded fixture set. The count increments after each match of that fixture group (all fixtures sharing the same non-`sequenceIndex` match fields). 12. **Bedrock uses Anthropic Messages format internally** — the adapter normalizes Bedrock requests to `ChatCompletionRequest`, so the same fixtures work. Bedrock supports both non-streaming (`/invoke`, `/converse`) and streaming (`/invoke-with-response-stream`, `/converse-stream`) endpoints. 13. **Azure OpenAI routes through the same handlers** — `/openai/deployments/{id}/chat/completions` maps to the completions handler, `/openai/deployments/{id}/embeddings` maps to the embeddings handler. Fixtures work unchanged. 14. **Ollama defaults to streaming** — opposite of OpenAI. Set `stream: false` explicitly in the request for non-streaming responses. 15. **Ollama tool call `arguments` is an object, not a JSON string** — unlike OpenAI where `arguments` is a JSON string, Ollama sends and expects a plain object. 16. **Bedrock streaming uses binary Event Stream format** — not SSE. The `invoke-with-response-stream` and `converse-stream` endpoints use AWS Event Stream binary encoding. 17. **Vertex AI routes to the same handler as consumer Gemini** — the same fixtures work for both Vertex AI (`/v1/projects/.../models/{m}:generateContent`) and consumer Gemini (`/v1beta/models/{model}:generateContent`). 18. **Cohere requires `model` field** — returns 400 if `model` is missing from the request body. ## Mount & Composition ### mount() API Mount additional mock services onto a running LLMock server. All services share one port, one health endpoint, and one request journal. ```typescript const llm = new LLMock({ port: 5555 }); llm.mount("/mcp", mcpMock); // MCP tools at /mcp llm.mount("/a2a", a2aMock); // A2A agents at /a2a llm.mount("/vector", vectorMock); // Vector DB at /vector await llm.start(); ``` Any object implementing the `Mountable` interface (a `handleRequest` method that returns `boolean`) can be mounted. Path prefixes are stripped before the service sees the request — `/mcp/tools/list` arrives as `/tools/list`. ### createMockSuite() Unified lifecycle for LLMock + mounted services: ```typescript import { createMockSuite } from "@copilotkit/aimock"; const suite = createMockSuite({ port: 0, fixtures: "./fixtures", services: { "/mcp": mcpMock, "/a2a": a2aMock }, }); await suite.start(); // suite.llm — the LLMock instance // suite.url — base URL afterEach(() => suite.llm.resetMatchCounts()); // reset sequence counts, keep fixtures afterAll(() => suite.stop()); ``` ### aimock CLI config file The `aimock` CLI reads a JSON config and serves all services on one port: ```bash aimock --config aimock.json --port 4010 ``` Config format: ```json { "llm": { "fixtures": "./fixtures", "latency": 0, "metrics": true }, "services": { "/mcp": { "type": "mcp", "tools": "./mcp-tools.json" }, "/a2a": { "type": "a2a", "agents": "./a2a-agents.json" } } } ``` ## VectorMock Mock vector database server for testing RAG pipelines. Supports Pinecone, Qdrant, and ChromaDB API formats. ```typescript import { VectorMock } from "@copilotkit/aimock"; const vector = new VectorMock(); // Create a collection and register query results vector.addCollection("docs", { dimension: 1536 }); vector.onQuery("docs", [ { id: "doc-1", score: 0.95, metadata: { title: "Getting Started" } }, { id: "doc-2", score: 0.87, metadata: { title: "API Reference" } }, ]); // Upsert vectors vector.upsert("docs", [ { id: "v1", values: [0.1, 0.2, ...], metadata: { title: "Intro" } }, ]); // Dynamic query handler vector.onQuery("docs", (query) => { return [{ id: "result", score: 1.0, metadata: { topK: query.topK } }]; }); // Standalone or mounted const url = await vector.start(); // Or: llm.mount("/vector", vector); ``` ### VectorMock endpoints | Provider | Endpoints | | -------- | ---------------------------------------------------------------------------------------------------------------------------------------- | | Pinecone | `POST /query`, `POST /vectors/upsert`, `POST /vectors/delete`, `GET /describe-index-stats` | | Qdrant | `POST /collections/{name}/points/search`, `PUT /collections/{name}/points`, `POST /collections/{name}/points/delete` | | ChromaDB | `POST /api/v1/collections/{id}/query`, `POST /api/v1/collections/{id}/add`, `GET /api/v1/collections`, `DELETE /api/v1/collections/{id}` | ## Service Mocks (Search / Rerank / Moderation) Built-in mocks for common AI-adjacent services. Registered on the LLMock instance directly — no separate server needed. ### Search (Tavily-compatible) ```typescript // POST /search — matches request `query` field mock.onSearch("weather", [ { title: "Weather Report", url: "https://example.com", content: "Sunny today" }, ]); mock.onSearch(/stock\s+price/i, [ { title: "ACME Stock", url: "https://example.com", content: "$42", score: 0.95 }, ]); ``` ### Rerank (Cohere-compatible) ```typescript // POST /v2/rerank — matches request `query` field mock.onRerank("machine learning", [ { index: 0, relevance_score: 0.99 }, { index: 2, relevance_score: 0.85 }, ]); ``` ### Moderation (OpenAI-compatible) ```typescript // POST /v1/moderations — matches request `input` field mock.onModerate("violent", { flagged: true, categories: { violence: true, hate: false }, category_scores: { violence: 0.95, hate: 0.01 }, }); // Catch-all — everything passes mock.onModerate(/.*/, { flagged: false, categories: {} }); ``` ### Pattern matching All three services use the same matching logic: - **String patterns** — case-insensitive substring match - **RegExp patterns** — full regex test - **First match wins** — register specific patterns before catch-alls ## Debugging Fixture Mismatches When a fixture doesn't match: 1. **Inspect what the server received**: `mock.getLastRequest()` → check `body.messages` array 2. **Check fixture order**: `mock.getFixtures()` returns fixtures in registration order 3. **For `userMessage`**: match is against the LAST `role: "user"` message only, substring match (not exact) 4. **Check the journal**: `mock.getRequests()` shows all requests including which fixture matched (or `null` for 404) ## E2E Test Setup Pattern ```typescript import { LLMock } from "@copilotkit/aimock"; // Setup — port: 0 picks a random available port const mock = new LLMock({ port: 0 }); mock.loadFixtureDir("./fixtures"); await mock.start(); process.env.OPENAI_BASE_URL = `${mock.url}/v1`; // Per-test cleanup — reset sequence match counts, keep the loaded fixtures afterEach(() => mock.resetMatchCounts()); // Teardown afterAll(async () => await mock.stop()); ``` ### Static factory shorthand ```typescript const mock = await LLMock.create({ port: 0 }); // creates + starts in one call ``` ## API Quick Reference | Method | Purpose | | ---------------------------------------- | ------------------------------------------- | | `addFixture(f)` | Append fixture (last priority) | | `addFixtures(f[])` | Append multiple | | `prependFixture(f)` | Insert at front (highest priority) | | `clearFixtures()` | Remove all fixtures | | `getFixtures()` | Read current fixture list | | `on(match, response, opts?)` | Shorthand for `addFixture` | | `onMessage(pattern, response, opts?)` | Match by user message | | `onEmbedding(pattern, response, opts?)` | Match by embedding input text | | `onJsonOutput(pattern, json, opts?)` | Match by user message with `responseFormat` | | `onToolCall(name, response, opts?)` | Match by tool name in `tools[]` | | `onToolResult(id, response, opts?)` | Match by `tool_call_id` | | `onTurn(turn, pattern, response, opts?)` | Match by turn index + user message | | `nextRequestError(status, body?)` | One-shot error, auto-removes | | `loadFixtureFile(path)` | Load JSON fixture file | | `loadFixtureDir(path)` | Load all JSON files in directory | | `start()` | Start server, returns URL | | `stop()` | Stop server | | `reset()` | Clear fixtures + journal + match counts | | `resetMatchCounts()` | Clear sequence match counts only | | `getRequests()` | All journal entries | | `getLastRequest()` | Most recent journal entry | | `clearRequests()` | Clear journal only | | `setChaos(opts)` | Set server-level chaos rates | | `clearChaos()` | Remove server-level chaos | | `onSearch(pattern, results)` | Match search requests by query | | `onRerank(pattern, results)` | Match rerank requests by query | | `onModerate(pattern, result)` | Match moderation requests by input | | `onImage(pattern, response)` | Match image generation by prompt | | `onSpeech(pattern, response)` | Match TTS by input text | | `onTranscription(response)` | Match audio transcription | | `onVideo(pattern, response)` | Match video generation by prompt | | `mount(path, handler)` | Mount a Mountable (VectorMock, etc.) | | `url` / `baseUrl` | Server URL (throws if not started) | | `port` | Server port number | Between tests that share a loaded fixture set, use `resetMatchCounts()` (not `reset()`, which also clears fixtures). For a `MockSuite`, call `suite.llm.resetMatchCounts()` — the suite itself has no `resetMatchCounts()`. Sequential responses use `on()` with `sequenceIndex` in the match — there is no dedicated convenience method. ## Record-and-Replay (VCR Mode) aimock supports a VCR-style record-and-replay workflow for ALL endpoints including multimedia (image, TTS, transcription, video): unmatched requests are proxied to real provider APIs, and the responses are saved as standard aimock fixture files for deterministic replay. Binary TTS responses are base64-encoded with format derived from Content-Type. Multimedia fixtures automatically include `endpoint` in their match criteria for correct routing on replay. ### CLI usage ```bash # Record mode: proxy unmatched requests to real OpenAI and Anthropic APIs aimock --record \ --provider-openai https://api.openai.com \ --provider-anthropic https://api.anthropic.com \ -f ./fixtures # Strict mode: fail on unmatched requests (no proxying, no catch-all 404) aimock --strict -f ./fixtures ``` - `--record` enables proxy-on-miss. Requires at least one `--provider-*` flag. - `--strict` returns a 503 error when no fixture matches AND no proxy is configured (or the proxy attempt fails), instead of silently returning a 404. The proxy is still tried first when `--record` is set. Use this in CI to prevent unmatched requests from slipping through as silent 404s. - Provider flags: `--provider-openai`, `--provider-anthropic`, `--provider-gemini`, `--provider-vertexai`, `--provider-bedrock`, `--provider-azure`, `--provider-ollama`, `--provider-cohere`. ### How it works 1. **Existing fixtures are served first** — the router checks all loaded fixtures before considering the proxy. 2. **Misses are proxied** — if no fixture matches and recording is enabled, the request is forwarded to the real provider API. Upstream URL path prefixes are preserved (e.g., `https://gateway.company.com/llm/v1` correctly proxies to `/llm/v1/chat/completions`). 3. **All request headers are forwarded (auth headers NOT saved)** — all client request headers are passed through to the upstream provider, except hop-by-hop headers and `host`/`content-length`/`cookie`/`accept-encoding`. Auth headers (`Authorization`, `x-api-key`, `api-key`) are forwarded but stripped from the recorded fixture. 4. **Responses are saved as standard fixtures** — recorded files land in `{fixturePath}/recorded/` and use the same JSON format as hand-written fixtures. Nothing special about them. 5. **Streaming responses are collapsed** — SSE streams are collapsed into a single text or tool-call response for the fixture. The original streaming format is preserved in the live proxy response. 6. **Base64 embedding decoding** — when the upstream returns base64-encoded embeddings (the default `encoding_format` in Python's openai SDK), the recorder decodes them into float arrays so fixtures contain readable numeric data instead of opaque base64 strings. 7. **Loud logging** — every proxy hit logs at `warn` level so you can see exactly which requests are being forwarded. ### Programmatic API ```typescript const mock = new LLMock({ port: 0 }); await mock.start(); // Enable recording at runtime mock.enableRecording({ providers: { openai: "https://api.openai.com", anthropic: "https://api.anthropic.com", }, fixturePath: "./fixtures/recorded", }); // ... run tests that hit real APIs for uncovered cases ... // Disable recording (back to fixture-only mode) mock.disableRecording(); ``` ### Workflow 1. **Bootstrap**: Run your test suite with `--record` and provider URLs. All requests that don't match existing fixtures are proxied and recorded. 2. **Review**: Check the recorded fixtures in `{fixturePath}/recorded/`. Edit or reorganize as needed. 3. **Lock down**: Run your test suite with `--strict` to ensure every request hits a fixture. No network calls escape. 4. **Maintain**: When APIs change, delete stale fixtures and re-record. ## MCP Recording and Fake Reports (1.45.0+) ### Record a live MCP server ```bash npx -p @copilotkit/aimock llmock -f ./fixtures \ --mcp-record /mcp=http://localhost:3001/mcp ``` - `--mcp-record =` (repeatable) is on the `llmock` bin and the Docker image, not on `aimock --config`. It needs a local `--fixtures` path. `--mcp-proxy-only =` forwards without writing and needs no `--fixtures`. - With `aimock --config`, set `llm.enableMcpRecording: true` and use `llm.record.mcp`: `{ "/mcp": "http://localhost:3001/mcp" }`, or an object with `upstream`, `fixturePath`, `proxyOnly`, `upstreamAuth`, `secretValues`, `strict`, `maxRecordBufferBytes`. Without the opt-in, `llm.record.mcp` is ignored with a warning. `llm.record.mcp` alone does not turn on LLM recording. - In code: `mcpMock.enableRecording({ upstream, ... })` and `disableRecording()`. - Every MCP request needs a test id (`X-Test-Id`, percent-encoded, or `?testId=`) or a context. Without one, the call is forwarded but not written. - The recording lands at `/recorded//mcp.json`. Replay it by starting aimock with the same `--fixtures` and no `--mcp-record`. A changed call then fails with `MCP_FAKE_MISMATCH`. - A call the file already answers is answered from the file, not forwarded. Delete the file to re-record. - Secrets: headers are never written; known secrets become `[REDACTED]` with a `_warnings` pointer. Add custom secret values with `AIMOCK_RECORD_SECRET_VALUES` (newline-separated, each at least 8 characters). Set an upstream credential with `AIMOCK_MCP_UPSTREAM_AUTH="Name: value"`. OAuth is not supported through the recorder. - Recorded `mcpFakes` files may hold `list`, `recorded` and `timing` (block) and `notifications` and `durationMs` (call entry). Older aimock versions reject them. Replay sends recorded progress notifications (as SSE) at the recorded timing, scaled by `--replay-speed`. Recorded log notifications replay only after the client calls `logging/setLevel`. ### Fail a test on a swallowed or unused fake - `GET /__aimock/mcp/fakes/report?testId=` returns `{ ok, served, unconsumed, failures, unfaked, sharedUnconsumed, evicted }` for one test id. - `useAimock({ fixtures, fakesReport: "fail" })` (Vitest or Jest) checks each test's report in `afterEach` and throws `AimockFakesReportError` when a fake error was swallowed by the agent or a declared entry was never used. `"warn"` prints instead. - `mock().fakesFor()` returns `{ testId, mcpUrl, headers }` for the current test. The default test id is ` › › ` (Vitest; Jest joins the names with spaces). Scope the `mcpFakes` block to that id. Pass an explicit id for concurrent tests and, in Jest, for duplicate test names. - `assertFakesReport(report)` does the same check anywhere. In pytest: `aimock.fakes_for()`, `aimock.fakes_report()` and `aimock.assert_fakes_report()`; the default id is the pytest node id. - Fakes work for any caller over MCP HTTP: agents, Mastra Workflow steps, LangGraph nodes and plain functions.