---
name: analyze-traces
description: >
  Analyze OpenTelemetry traces and structured logs from a running autotel service to debug errors,
  investigate latency, follow requests across services, and surface cardinality / attribute hygiene
  problems. Works with traces from any OTLP backend (Honeycomb, Grafana Tempo, Datadog, Jaeger,
  Sentry, Axiom, HyperDX, …) plus the local `.autotel/spans/` dump and `InMemorySpanExporter` in tests.
license: MIT
---

# Analyze traces

This skill teaches an AI assistant how to read and reason about OpenTelemetry traces produced by autotel. Whether they're sitting in a backend, exported to a local JSON dump, or captured in a test.

## When to use

- Debugging a failing endpoint after deploy
- Investigating latency regressions (p50 / p95 / p99 spike)
- Following a single request across browser → server → queue → worker
- Auditing attribute hygiene (cardinality, PII leak risk, noise)
- Spot-checking that a new instrumentation actually produces the spans you expected

## Input formats

| Source                         | How to access                                                      |
| ------------------------------ | ------------------------------------------------------------------ |
| Local debug dump               | `.autotel/spans/*.ndjson`: one span per line, OTLP JSON shape      |
| `InMemorySpanExporter` (tests) | `exporter.getFinishedSpans()`                                      |
| Backend (interactive)          | Jaeger / Tempo / Honeycomb UI; Datadog Trace Search; etc.          |
| Backend (programmatic)         | Honeycomb Query API, Tempo `/api/search`, Datadog Logs / Trace API |

## The shape of an autotel span

```json
{
  "name": "POST /api/checkout",
  "context": { "traceId": "…", "spanId": "…" },
  "parentSpanId": "…",
  "kind": "SERVER",
  "startTimeUnixNano": "…",
  "endTimeUnixNano": "…",
  "status": { "code": "OK" },
  "attributes": {
    "service.name": "checkout",
    "http.request.method": "POST",
    "url.full": "https://api.example.com/api/checkout",
    "http.response.status_code": 200,
    "user.id": "usr_123",
    "user.plan": "enterprise",
    "cart.items": 3,
    "cart.total": 14999,
    "_correlationId": "01J…"
  },
  "events": [
    {
      "name": "log.emit.manual",
      "attributes": { "level": "info", "stage": "validated" }
    }
  ],
  "links": [],
  "resource": {
    "service.name": "checkout",
    "deploy.id": "v2025.05.04-1"
  }
}
```

Key conventions to recognise:

- `service.name` distinguishes services in a multi-service trace.
- `_correlationId` (autotel-specific) is stable within a logical unit of work even across forked child spans (`_parentCorrelationId` ties them).
- `gen_ai.*` attributes follow OpenTelemetry gen-ai semantic conventions for LLM calls.
- `exception.*` attributes (auto-set by `createStructuredError`) carry `type`, `message`, `stacktrace`.

## Common investigations

### "Why is endpoint X failing?"

1. Find the trace: filter `service.name = "<svc>" AND http.route = "<route>" AND status = error` for the last hour.
2. Open the slowest / latest matching trace.
3. Inspect the root span's `exception.message` and `exception.stacktrace`.
4. Walk down the child spans: the deepest span with `status.code = ERROR` is usually the culprit.
5. If using `createStructuredError`, look for `code`, `why`, `internal.*` attributes. They usually answer the "why" without you reading code.

### "Why is endpoint X slow?"

1. Find a slow trace: `service.name = "<svc>" AND http.route = "<route>" AND duration > p99(duration)`.
2. View the waterfall: pinpoint the longest child span by self-time (not wall time).
3. Common offenders:
   - **Sequential awaits that should be parallel**: sibling spans run end-to-end instead of overlapping.
   - **N+1 queries**: many short same-named spans (`SELECT * FROM …`) under one parent.
   - **Cold starts**: `faas.coldstart=true` in Workers or Lambda.
   - **Tool retries**: gen-ai spans with `gen_ai.response.finish_reasons` containing `error` followed by another call.

### "Which requests are affected?"

A waterfall explains one slow request. It cannot tell you what the slow ones
share. Run the core analysis loop instead: split the population on the symptom,
then score every recorded field by how much the two groups disagree.

```typescript
import { compareCohorts } from 'autotel/analysis';

const isSlow = (event) => Number(event['checkout.duration_ms']) >= 800;

const ranked = compareCohorts({
  outlier: events.filter(isSlow),
  baseline: events.filter((event) => !isSlow(event)),
});
// [{ field: 'payment.provider', value: 'bank-beta', difference: 0.67, … }]
```

Feed it any array of flat records: `span.attributes` from a collector, wide
events from `getRequestLogger()`, or rows returned by a backend query.

The loop skips fields whose values never repeat. A request id takes a fresh
value on every event, so it would bury the answer under noise. Bucket durations
and payload sizes when you instrument them if you want the loop to rank them.

The top row is a hypothesis. A field can move with the problem without causing
it. You are done when you have named the cohort and confirmed the mechanism in
two traces drawn from it.

### "Follow this user across services"

Use `_correlationId` (or `user.id` if you have it):

```
service.name in (web, api, worker) AND _correlationId = "01J…"
ORDER BY startTime
```

Each service contributes spans with the same `traceId` (W3C trace context propagation handles this automatically with autotel's global fetch instrumentation).

### "Did the new instrumentation actually fire?"

In a test, dump the in-memory exporter:

```typescript
import { InMemorySpanExporter } from 'autotel/exporters';
const exporter = new InMemorySpanExporter();
// … run the code under test
const spans = exporter.getFinishedSpans();
console.log(spans.map((s) => ({ name: s.name, attrs: s.attributes })));
```

Or live, point the SDK at a local file dump:

```typescript
init({ service: 'my-app', debug: 'pretty', spanDumpPath: '.autotel/spans' });
```

…then `tail -f .autotel/spans/*.ndjson | jq` while exercising the feature.

## Cardinality / hygiene audits

| Check                         | Query / heuristic                                                                                               |
| ----------------------------- | --------------------------------------------------------------------------------------------------------------- |
| **Span name cardinality**     | Top-K distinct `name` per service. Anything > a few hundred is a red flag: likely an unnormalised URL.          |
| **Per-attribute cardinality** | `unique(attribute_value)` per `attribute_key`. UUIDs / emails / `Date.now()` ids in attributes blow up storage. |
| **Missing `service.name`**    | Spans where the resource attribute is empty or `"app"`: fix at the SDK init.                                    |
| **PII smell**                 | Look for raw `@`, leading digit-runs of length 16, or `eyJ` prefixes: your redactor is off.                     |
| **Health-check noise**        | Spans with `http.route in (/healthz, /ready)`. Drop with `FilteringSpanProcessor`.                              |

## Reading gen-ai traces

LLM calls produce a parent span (kind `CLIENT`) with children for each tool call:

| Attribute                                                              | Meaning                                  |
| ---------------------------------------------------------------------- | ---------------------------------------- |
| `gen_ai.provider.name` (legacy: `gen_ai.system`)                       | Provider (`openai`, `anthropic`, …)      |
| `gen_ai.request.model`                                                 | Model id                                 |
| `gen_ai.usage.input_tokens` / `output_tokens`                          | Token count                              |
| `gen_ai.usage.cache_read.input_tokens` / `cache_creation.input_tokens` | Cache hits                               |
| `gen_ai.response.finish_reasons`                                       | `stop`, `tool_calls`, `length`, `error`  |
| `gen_ai.tool.name`                                                     | Tool invoked (on tool-call child spans)  |
| `gen_ai.usage.cost.usd`                                                | Estimated cost (if pricing map provided) |

These canonical `gen_ai.*` attributes are emitted by the `autotel-genai` package (third-party instrumentations may still emit the deprecated `gen_ai.system`).

Common findings:

- High `gen_ai.usage.input_tokens` with low `gen_ai.usage.cache_read.input_tokens` → enable prompt caching.
- Many sequential tool-call spans → consider parallel tool calls if the model supports it.
- `gen_ai.response.finish_reasons` contains `length` → bump `max_tokens`.

## When the trace is missing

If you expected a span and there isn't one:

1. **Sampling.** Did head sampling drop it? Check `sampling.rates` and `recordedSpans` in any subscriber.
2. **Workers without `waitUntil`.** Did the request return before the exporter flushed? Move to `defineWorkerFetch` / `wrapModule`.
3. **`instrumentation.disabled = true`**: check env-conditional config.
4. **Exporter rejected.** Check service logs for `OTLP exporter` 4xx / 5xx: bad token, wrong dataset.

## Output format

When summarising an investigation, lead with the **decision-changing fact**, then the supporting evidence:

```
Failure cause: payment.declined (Stripe code: insufficient_funds)
- Trace: 9d3a…b21
- 38 / 412 checkout requests in the last hour failed with status=402.
- All in eu-west-1, all on plan=free.
- exception.cause.stripeChargeId starts with ch_3M…
- Suggest: surface the structured `fix` field to the client; current 402 body returns generic message.
```

Keep raw span dumps out of the summary; link to the trace ID instead.