--- name: ironbee-run-scenario description: "Run a saved IronBee verification scenario end-to-end using the ibdt_scenario-* MCP tools. Drives the pause → act → resume loop: realizes any `llm-action` steps with the platform tools and caches the realization. NOT a verification cycle — it submits no verdict and does not gate completion." disable-model-invocation: true --- # IronBee — Run scenario > **⚠️ Run this inline — do NOT delegate.** Do every step yourself, in this conversation; > do **not** use the `Task` tool or spawn a sub-agent. A sub-agent runs as a separate session, > so its tool calls are recorded elsewhere and the work won't be tracked correctly. Run a saved verification **scenario** end-to-end. Most steps execute themselves inside the devtools sandbox; you step in only when the run **pauses**. This is **NOT a verification cycle** — it opens no cycle, submits no verdict, and does not satisfy the completion gate (use `/ironbee-verify` for that). ## Resolve the arguments Peel these off the front of the argument string, then the remainder is the scenario: - **`step`** → step-by-step mode: pause after EVERY step, not just `llm-action` ones. - **`refresh:[,…]`** (or **`refresh:all`**) → re-resolve those `llm-action` steps from scratch, ignoring their cached realizations. (`ironbee scenario status --verbose` lists the step names and which are stale.) - The **remainder is the scenario reference** — an exact name OR a semantic description — up to an optional ` args:` boundary. A ` args:` token (a space, then `args:`) splits it: text BEFORE is the reference, text AFTER is a JSON object of **override arguments** for the scenario's `params`. With no ` args:` the whole remainder is the reference and the scenario runs with its captured defaults. ## Step model — what you may be driving A scenario is EITHER one `script` (the original shape) or an ordered **`steps`** array of: `script` (sandbox JS with `callTool` / `args` / the shared `ctx` / `assert`), `command` (a shell command run on the host; its `{ stdout, stderr, exitCode }` goes to `ctx[assignTo]`), `llm-action` (a natural-language `intent` that YOU realize — see below), and `include` (splices another scenario's steps in). Steps share one mutable `ctx`; `args` is re-injected fresh per step. An `llm-action`'s cached realization lives in the SIBLING `.cache.json` (`byStepName[""]`), committed and diff-reviewable. **Never edit that file directly** — the only write channel is the `cache` payload on a `ibdt_scenario-run` resume. > **Never drive platform tools to DIAGNOSE a run.** Poking the browser (or any platform tool) > between runs — to see why one failed, or to check the app's state before re-running — is outside > the scenario contract and wastes turns and cost. A failed or paused run already tells you what you > need: its response carries the failing/pending step, its error and the `ctx` snapshot, and > `ibdt_scenario-recorded-actions` replays the calls made during the current pause. Read those and > resume — don't improvise around the run loop. ## The run loop 1. **Resolve** the reference (`ibdt_scenario-search` semantic + exact-name `ibdt_scenario-list`). Several plausible matches → ask which; none → say so and stop. 2. **Start it.** `ibdt_scenario-run({ name, args?, scope: 'project' })` — plus `stepByStep: true` when `step` was given and `refresh: true | ["", …]` when `refresh:` was given. These are latched at start; a resume never changes them. **Pass them as real JSON values, never stringified** (`refresh: true` / `refresh: ["do-login"]`, NOT `refresh: "true"`): a stringified value is silently ignored, so a step you asked to refresh comes back `source: 'cache'` instead of pausing. If that happens, re-send it as a real boolean/array before concluding anything. 3. **Branch on `status` in every response.** - `completed` → done → report (§Reporting). - `failed` → done → report the failing step + its error, and **STOP**. Do NOT then drive the browser yourself to "investigate" the app. Distinguish two failures — they get opposite responses: a BROKEN mechanic (a tool errored, an element wasn't found, a cached `llm-action`'s replay threw) is DRIFT → for a cached `llm-action` heal it (step 4), for a `script`/`command` step just report it; a business assertion FAILING on values read fine (e.g. `orderTotal 1999 != cartTotal 1713.43` — tools succeeded, numbers came back, relationship doesn't hold) is a real app DEFECT → report the assertion + its values and STOP, don't edit the scenario and don't open a browser session to diagnose the app. For a script assertion whose expected value legitimately moved, ask the user — never auto-edit it. Staleness NEVER reclassifies this. `ironbee scenario status` is a STATIC guess (it diffs author-declared `coveredPaths` since a baseline) and cannot tell a behaviour-changing edit from an inert one — adding `module.exports` to a service trips it as hard as rewriting checkout. Use it only to explain a BROKEN MECHANIC ("did the cached realization drift?"), and read its basis: `[replay failed]` is a run's own evidence, `[static guess]` is not. A business assertion that failed on values the tools read fine is a DEFECT even when every covered file changed — a cache that truly drifted breaks mechanically, it does not return coherent wrong numbers. - `paused` → look at `pendingStep.reason` and continue below. 4. **`reason: 'llm-action'` → YOU realize the intent — UNLESS it is already cached.** - **Decide on `pendingStep.reason`, NEVER on `pendingStep.type`.** This is the trap. In `stepByStep` mode the run pauses before EVERY step, and a pause before an `llm-action` step carries `type: 'llm-action'` AND its `intent` text — but `reason: 'step-by-step'`. That is NOT a request to act: it means "just resume." The `intent` is only yours to realize when **`reason` is `'llm-action'`**. Seeing `type: 'llm-action'` + an intent and driving the browser is exactly the mistake to avoid — the run never asked you to. - **Is this step cached? Let the RUN tell you — never guess from files.** The ONLY signals are the pause itself: `pendingStep.reason` and `pendingStep.healReason`. On a `step-by-step` pause with no `healReason`, the cache is intact and will replay on resume — **do NOT drive the browser. Just resume** (step 7); the cached script does the work. Re-realizing a step that already has a good cache doubles the work and burns turns. - **Do NOT `cat` a `.cache.json` to decide cached-vs-uncached** — that is how this goes wrong. An INCLUDED step's cache lives in the INCLUDED scenario's file (`signin.cache.json`, `add-first-product.cache.json`), NOT the parent's (`checkout-happy-path.cache.json`), so looking only at the parent makes a cached step look uncached and you drive the browser for nothing. You never need to check: just resume. If the next step is genuinely UNCACHED, the run pauses AGAIN with `reason: 'llm-action'` (not `step-by-step`) — realize it THEN, and only then. You realize an `llm-action` yourself in exactly two cases: `reason: 'llm-action'` (uncached) or `healReason` present (the cache broke). - Read `pendingStep.intent` and `ctxSnapshot` (what earlier steps produced). **`ctxSnapshot` is scoped**: when `pendingStep.ctxPath` is non-empty the step is bound to a NESTED ctx root (an `include` with `as:`) and the snapshot IS that root — refer to `ctx.` relative to it, never to an outer key. - **You do NOT need a verification cycle for this.** A scenario run is gate-orthogonal: never call `ironbee hook verification-start` to unblock yourself. IronBee recognises the pause from the `ibdt_scenario-run` response and allows your platform calls for as long as the run is parked on it. - Achieve the intent with this project's platform tools (navigate / click / fill / call / inspect …) — ordinary tool calls. **Cap the effort at ~20 tool calls**; if it still isn't achieved, stop and report a failure instead of looping. - **Self-heal:** when `pendingStep.healReason` is present a previously cached script broke, and `pendingStep.priorCacheScript` is that script — **repair THAT script**, don't start over. `healReason` separates an assertion/expectation failure (the flow ran but the outcome was wrong — suspect a real defect) from a tool/selector failure (the mechanics moved). - **Before you call it a DEFECT, check whether the code actually moved.** Run `ironbee scenario status --verbose`: if IronBee reports this step's cache STALE because a covered file changed since its baseline, the mechanics moved — that is drift, repair the script. Untouched covered files + a misbehaving app = a real defect: STOP and report, don't rewrite the cache around it. 5. **Author the cache script** — the deterministic replay of what you just did. - Sandbox code: `await callTool('', { … })`, plus `assert.*` where a replay should check something. - **Tool-name mapping is mandatory.** You invoke the tool as `MCP:bdt_navigation_go-to`, but `callTool` takes the BARE name `bdt_navigation_go-to` — strip the `MCP:` wrapper. When unsure, run `ironbee scenario tool-name ` in the terminal and use exactly what it prints. - **What you actually did** is the source. When the realization took many steps and you are not certain of the exact sequence, call **`ibdt_scenario-recorded-actions`** — it returns the platform calls you made during THIS pause, in order, with their inputs. Turn that into a CLEAN script: drop exploration and failed attempts, keep the path that worked, preserve the order. - **Never hardcode volatile values** (order ids, timestamps, generated emails) — read them from `ctx` or from a tool's return value so the script re-runs on fresh data. - Write `ctx` **relative to your own root** (see `ctxPath` above). - **Write into `ctx` exactly what LATER steps read — this is the #1 way a cache breaks the run.** Before writing the script, look at the steps AFTER this one in the scenario (and, for an `include`, in the including scenario): every `ctx.` they reference must be set BY YOUR SCRIPT. `resumeOutput` is NOT a substitute — a warm replay never sends one, so anything that only arrives that way is missing on every later run. If a later step asserts `ctx.loggedIn`, your realization must set `ctx.loggedIn` — not some near-miss like `ctx.login = {…}`. 6. **Stamp the baseline — do NOT skip this.** Put both in the `cache` payload: - `resolvedCommit` — the current commit (`git rev-parse HEAD`). - `coveredPaths` — the source paths this step exercises (routes / components / handlers you drove, or files you had to read to do it). Best-effort globs are fine (`src/auth/**`). These are what lets `ironbee scenario status` later notice the code moved under this cached step. Without them the cache is invisible to staleness detection and can only self-heal AFTER it breaks. 7. **Resume.** `ibdt_scenario-run({ name, resume: true, resumeOutput?, cache: { script, resolvedCommit, coveredPaths } })`. `resumeOutput` is optional: a plain JSON **object** is merged into `ctx` — send a real object, NOT a stringified one; a string (or array / scalar) lands on `ctx._resume` instead and the next steps won't see your keys. Do NOT re-send `args` / `stepByStep` / `refresh`. If the response says the cache was rejected (e.g. a syntax error), fix the script and resume again. > **Call resume DIRECTLY — top-level, one per turn. NEVER wrap a resume in `execute` or a > `callTool` loop.** A resume drives the top-level pause/resume state machine; wrapping it makes > the run NESTED, and a nested run's step timeline is buried in the parent tool response and never > recorded — the whole resumed body silently vanishes from the step events. The tool now REJECTS a > nested resume with an error. Loop by calling `ibdt_scenario-run({ resume: true })` again at the > top level, not by batching resumes inside one `execute`. 8. **`reason: 'step-by-step'` → do NOTHING, just resume**: `ibdt_scenario-run({ name, resume: true })` with no `cache`. (If the engine then reaches an `llm-action` it pauses again with `reason: 'llm-action'` — THAT is when you act.) 9. **Loop** back to 3 until `completed` or `failed`. > **ONE run, kept going by RESUME — never restart from the top.** A `paused` run is an invitation to > `resume: true`, NOT a reason to fire a fresh `ibdt_scenario-run({ name })`. Starting the whole > scenario over to "get past" a step throws away everything the run has done: it re-realizes every > already-cached step from scratch, opens a brand-new span, and multiplies the cost — a scenario that > would resume to the end in ONE run instead sprawls into N restarts, each re-doing the last one's > work and stopping a little further along. Author the pending step's cache and **resume the SAME > run**; the engine carries you forward. Start fresh ONLY when a run has genuinely TERMINATED > (`status: completed` / `failed`) and you want a clean re-attempt, or you're changing `refresh:` / > `step`. (This is the "don't improvise around the run loop" rule above, applied to the run itself.) > **A genuine defect is a STOP, not a workaround.** If the intent can't legitimately succeed because > the app is broken, do NOT cache a script that dodges the bug or asserts a weaker outcome. Stop, > report the defect, and leave the scenario as the honest failing check it is. ## Showing progress Every response carries `progress` (`total`, `completedCount`, `currentIndex`) plus the cumulative `completedSteps`. - If the response has **`renderHint`, print it VERBATIM** — don't rewrite or reformat it. It's a ready-made markdown block and the harness renders markdown. - If it doesn't, build the checklist yourself from `progress` + `completedSteps` (`✓` ok, `✗` failed, `·` skipped; the next item is `pendingStep`). Re-print the whole block each turn — never `\r` or in-place terminal tricks. - **Decide from the structured fields** (`status` / `verdict` / `progress` / `completedSteps`), never from `renderHint` prose. ## Reporting Report from `verdict` + `assertionSummary`, not `status` alone: - `verdict: 'pass'` → passed. - `verdict: 'fail'` → failed; name the failing step and its error. - **`verdict: 'soft-fail'` (or `assertionSummary.softFailed > 0`) → NOT a clean pass.** Soft assertions record a failure without stopping the run, so `status` can still be `completed`. Say so explicitly — never report a soft-fail run as green. Also say which steps ran `source: 'cache'` (replayed) vs `source: 'live'` (freshly resolved), and whether you authored or repaired a cache. ## include / `as:` — reusable parts An `include` step splices the referenced scenario's steps into this run, so an `llm-action` inside a reusable part pauses and caches exactly like a top-level one. Its cache lives in the INCLUDED scenario's own `.cache.json` and is therefore **shared by every scenario that includes it**. `include.args` is resolved BEFORE the run starts, so it carries only STATIC values (a literal, or `{{args.}}` referring to the including scenario's own params) — never data a previous step produced. `as: ""` gives the part its own `ctx` root (`ctx.`): use it for a **self-contained** part, never for one that reads runtime data an earlier step wrote into the shared `ctx` (an isolated root can't see it and the cached script would fail on every replay). (To author/change a scenario use `/ironbee-manage-scenario`, to find one `/ironbee-search-scenario`, to re-validate + repair stale ones `/ironbee-sync-scenario`. To just *detect* staleness — including per cached step — run `ironbee scenario status --verbose`.)