LLM Observability: Monitoring AI Agents in Production

Traditional monitoring assumes failures look like failures: errors, timeouts, 500s. LLM systems break that assumption. A misrouted request, a hallucinated client name, a confidently wrong draft — all of them return 200 OK in 800ms. Your uptime dashboard glows green while the agent quietly erodes user trust.

Observability for AI agents means making quality failures as visible as availability failures. Here's the stack we run on production systems handling 500+ actions per week across 30 action types.

Trace Everything, End to End

The unit of observability is not the API call — it's the full journey from trigger to outcome. Every request gets a correlation ID at entry, and every stage appends a span:

1. Trigger received (message, email, webhook) — source and timestamp
2. Intent classification — matched action, confidence, tier, latency (our budget: under 200ms)
3. Graph traversal — every node visited, every edge taken, every clarification asked
4. Context retrieval — which of the 9 connected sources were queried, hit counts from the 38,000+ embedding index
5. Model calls — model, prompt version, tokens, latency, cost
6. Tool execution or proposal — payload, validation results, approval-queue decision
7. Outcome — approved, rejected, edited, reverted, or user correction downstream

When a user says "the agent did something weird Tuesday," the answer is one trace lookup, not an archaeology dig across five log systems. This is the same discipline as distributed tracing in microservices — the spans are just weirder.

The Metrics That Matter

Note how many of these come free if you built the system right: the approval queue produces labels, the audit trail produces traces, the confidence scores produce calibration data. Observability isn't a separate project — it's the exhaust of good architecture.

Drift: The Slow Failure

The failures that hurt most aren't incidents; they're drifts. A model provider updates a version and your extraction gets subtly worse. A new client type changes the input distribution. A prompt edit two weeks ago shifted routing on one action. None of these page anyone.

Our defenses:

• A frozen evaluation set: several hundred labeled real requests, re-run against the live pipeline nightly. Score changes localize the drift to a stage.
• Shadow comparisons on every change: model upgrades and prompt rewrites run in shadow against the current version before cutover — the same shadow-mode machinery from initial deployment, kept forever
• Per-action baselines: aggregate metrics hide local failures; one action collapsing from 98% to 80% approval barely moves a 30-action average

If your only alert is on error rate, your AI system has been broken before and you didn't know. The users knew.

The Weekly Review Ritual

Dashboards don't improve systems; the loop does. Every week, 45 minutes, same agenda:

1. Every rejected proposal — why, and does it become an anti-pattern example?
2. Every misroute flagged by users — new positive example on the correct action
3. Top 5 highest-edit-distance approvals — what did the human fix, and can the prompt or context fetch absorb it?
4. Calibration check — is the 0.9+ tier still above 97% actual accuracy?
5. One metric trend discussed, one concrete change shipped

This ritual is how our routing accuracy got to 95% and stays there. Not a bigger model — a tighter loop.

Starting from Zero

1. Add a correlation ID and log every model call with prompt version, tokens, latency, and cost — today, before anything else
2. Capture outcomes: approvals, rejections, edits, user corrections. Labels are the scarce resource; harvest them from day one.
3. Build the frozen eval set from your first 200 real requests
4. Set alerts on distributions, not just errors
5. Schedule the weekly review before you think you need it

You can't manage what you can't see, and with LLMs you can't see anything by default. Instrument first; the intelligence is only as good as your ability to notice when it stops being intelligent.