LLM Cost Optimization: Small Models for Routing, Big Models for Judgment

Teams discover their LLM bill the way they discover a water leak: weeks late and all at once. The instinct is to negotiate pricing or cap usage. Both miss the point — in every system we've audited, the bill was an architecture problem.

Where the Money Actually Goes

Three patterns cause most of the waste:

1. Flagship models doing intern work — classification, extraction, and yes/no routing sent to the most expensive model available
2. Context stuffing — entire documents and chat histories resent on every turn
3. Unbounded outputs — no max_tokens, so a routing call that needs 50 tokens returns 2,000

The Tiering Table

In our production agent, roughly 70% of calls are routing and extraction on the fast tier. The expensive consensus path — three models in parallel — runs on under 10% of traffic, triggered only by analytical queries, long documents, or image analysis. Net effect: the blended cost per query is about a tenth of the naive "one big model" design, while the answers on hard questions got better.

Six Levers, In Order of Payoff

1. Route by difficulty. A cheap classifier deciding which model handles each request is the single highest-ROI component you can build.
2. Cap output tokens per call type. Routing: 200. Confirmation: 300. Only open-ended generation gets a large budget.
3. Use prompt caching. System prompts and action catalogs are identical across calls — cached input tokens cost a fraction of fresh ones.
4. Retrieve, don't stuff. Vector search pulls the 5 relevant chunks instead of resending the whole knowledge base. Our 38,000-vector index costs pennies to query.
5. Precompute the predictable. Morning briefings, recurring reports, and analytics summaries render once on a schedule — not per user, per view.
6. Batch the background work. Nightly embedding and enrichment jobs tolerate latency; batch APIs run them at half price.

Spend More Where It Matters

Optimization is not minimization. Some queries deserve a 3x premium: we run Claude, GPT and Gemini in parallel on contract review and compliance questions precisely because a wrong answer there costs more than a thousand routed messages. The discipline is spending 3x on the 10% that matters instead of 10x on everything.

Make the Bill Observable

• Log tokens and model per request, tagged by feature and user
• Track cost per action completed, not cost per API call — it's the number your CFO actually cares about
• Alert on per-feature cost spikes; a prompt regression can triple output length silently

Our benchmark: a completed automated action — researched, drafted, approved, executed — costs less than the first 90 seconds of the employee time it replaces. Once that's true, the LLM bill stops being a debate.

Right-size the model, bound the output, cache the static, retrieve the relevant, and reserve the premium path for premium problems. The bill takes care of itself.