Building Agents with the Claude API: Routing, Vision, and Tool Use

Most teams use one model for everything and wonder why their agent is slow, expensive, or both. The Claude family is a range — and production agents work best when each tier does the job it's priced for.

The Three-Tier Pattern

Every user message hits the fast tier first. The router returns an intent, extracted parameters, and a confidence score. Only when real generation is needed does a bigger model get involved. This single decision cuts cost by an order of magnitude versus routing everything to the largest model.

Routing: Ask for JSON, Verify Everything

The routing prompt gives Haiku the action catalog (names, intent patterns, anti-patterns, required parameters) and demands a strict JSON response: { action, confidence, params }. Two rules keep this reliable:

• Validate the JSON against a schema. If parsing fails, retry once, then fall back to general Q&A. Never crash on model output.
• Treat confidence as a first-class output. We execute at 0.9+, ask a clarifying question at 0.7–0.89, and fall back to search below that. The thresholds came from reviewing hundreds of routed messages, not from intuition.

Vision: The Underrated Workhorse

Claude's vision handles what OCR pipelines used to: signed contracts as scans, screenshots of error messages, photos of whiteboards. In our chat, users drag a PDF or paste a screenshot and the agent reads it directly. Practical notes:

• Send documents as images with a task-specific prompt ("review against our NDA standard, list deviations") — generic "describe this" prompts waste the capability
• For multi-page documents, page-by-page extraction followed by a synthesis pass beats one giant request
• Image-based analysis is one of our automatic triggers for the multi-model consensus path — stakes are usually higher

Tool Use: Actions, Not Answers

Tool use is where an assistant becomes an agent. Our pattern:

1. Define tools narrowly — create_task, search_email, log_time_entry — with tight JSON schemas
2. Read-only tools execute immediately; write tools produce an approval-queue proposal instead of executing
3. Return tool errors to the model verbatim — Claude is good at self-correcting a malformed call when it sees the validation message
4. Cap tool-call loops (we use 5) to prevent runaway chains

The schema is your safety net. A model cannot invent a parameter your schema rejects, and it cannot touch a system you never gave it a tool for.

Operational Lessons

• Set max_tokens deliberately. Routing needs 200 tokens, not 4,000. Latency follows output length.
• Log every request/response pair with model, latency, and token counts — you cannot optimize what you don't measure.
• Pin model versions and test before upgrading. Behavior shifts between releases are real, especially in routing.
• Design for refusal. Occasionally the model declines a legitimate request; catch it and reroute rather than surfacing a raw refusal to the user.

The Claude API gives you the pieces: a fast tier for reflexes, a deep tier for judgment, vision for the messy real world, and tool use for touching your systems. The architecture — routing, thresholds, approval queues — is what turns those pieces into an agent people trust.