Intent Classification with LLMs: Sub-200ms Routing for AI Agents

Every AI agent has a first decision to make: what is the user actually asking for? Get this wrong and everything downstream is wasted — the best drafting model in the world can't help if it's drafting the wrong thing. Get it slow and the agent feels broken before it does anything.

We route requests across a catalog of 30 automated actions in under 200ms, at 95% accuracy, at a volume of 500+ actions per week. Here is the architecture.

Why Not Just Ask the Big Model?

The naive approach — send the message to your flagship LLM with the full action catalog in the prompt and ask "which one?" — works in demos and fails in production for three reasons:

• Latency: 2–5 seconds before the agent even starts working. Users perceive this as the whole system being slow.
• Cost: a full-context call on every message, including the 60% of messages that are casual chat needing no action at all
• Drift: prompt-based classification is unstable — a catalog rewrite silently shifts routing behavior with no way to measure it

The Two-Stage Architecture

Stage 1: Embedding similarity (~50ms)

Every action in the catalog has a set of example phrasings — real user requests, not invented ones. We embed all of them (our production index holds 38,000+ vector embeddings across actions and knowledge). An incoming message is embedded once and compared against the action space. This gives a ranked shortlist of candidate actions with cosine similarity scores.

Stage 2: Lightweight verification (~100ms)

A small, fast model receives the message plus only the top 3–5 candidates and picks one — or none. Because the candidate set is tiny, this call is cheap and fast. The output is a single action ID and a confidence score.

Total budget: under 200ms. The user never waits on classification; they wait only on the actual work.

Confidence Tiers: The Part Everyone Skips

The classifier's job is not to always pick an action. It's to know when it shouldn't. We route every message through four tiers:

The 0.7–0.89 band is where trust is won or lost. A wrong execution at 0.8 confidence destroys more goodwill than ten clarifying questions. The clarification step also produces gold: every user click on "yes, I meant X" is a new labeled training example.

A classifier that says "I'm not sure" 15% of the time and is right the other 85% beats one that always answers and is right 90% of the time. Users forgive questions. They don't forgive confident mistakes.

Building the Example Set

Classification quality is 80% data, 20% architecture. Our process per action:

1. Start with 10–15 phrasings written by the people who'll use it — not by engineers
2. Run shadow mode and harvest real messages that should have matched
3. Add negative examples: messages that look similar but must NOT match (see anti-patterns — "email John the contract" vs "did John email the contract?")
4. Review misroutes weekly; every misroute becomes an example on the correct side

Actions with fewer than 20 examples misroute noticeably. Past 50, gains flatten. Budget accordingly.

Measuring It Honestly

Track three numbers per week, per action:

• Routing accuracy: did the matched action equal the correct action? Our floor is 95% overall.
• Coverage: what fraction of actionable messages got matched at all? Accuracy is easy to game by matching nothing.
• Calibration: within the 0.9+ tier, accuracy should exceed 97%. If it doesn't, the tier thresholds are lying and need to move up.

Latency Checklist

1. Pre-compute and cache all action embeddings; never embed the catalog at request time
2. Use the smallest embedding model that holds accuracy — measure, don't assume
3. Run stage 1 and any per-user context loading in parallel
4. Keep the stage-2 prompt under 1,000 tokens; candidates only, no full catalog
5. Set a hard timeout — if classification exceeds 500ms, fall back to the Q&A tier rather than blocking

Fast, calibrated intent classification is the difference between an agent that feels like a colleague and one that feels like a phone menu. Spend the engineering effort here first; every other component depends on it.