Graph Routing vs Prompt Chains: Building Deterministic AI Agents

There are two ways to build a multi-step AI agent. The first is the prompt chain: model output feeds the next prompt, which feeds the next, with the LLM deciding at each step what happens. The second is the graph: an explicit decision tree where nodes are steps, edges are conditions, and the LLM fills in content but never controls the flow.

We started with chains. We rebuilt on graphs. Every production system we've shipped since — including one running 30 actions at 500+ executions per week — is graph-routed. Here's why.

The Prompt Chain Trap

Chains are seductive because the first version takes an afternoon. "Classify the request, then extract the details, then draft the output, then format it." Four prompts, glued with string interpolation. It works in the demo.

Then production happens:

• Failures cascade invisibly. Step 2 extracts a wrong client name; steps 3 and 4 confidently build on it. You see a bad final output with no idea which link broke.
• Behavior is unrepeatable. The same input can take different paths because an intermediate model output phrased something differently. You cannot write a test for "usually does the right thing."
• Changes are global. Editing the step-2 prompt shifts the distribution of inputs to step 3. Every prompt edit is a full-system change with no blast-radius boundary.
• There's no map. Six months in, nobody can draw what the agent actually does. Onboarding a new engineer means reading prompts and guessing.

What Graph Routing Looks Like

In a graph-routed agent, the control flow lives in code, not in prompts. Each action is a small decision tree:

1. Nodes are typed steps: fetch data, ask a clarifying question, run an LLM transform, propose an action
2. Edges are explicit conditions: confidence thresholds, presence of required fields, user choices
3. The LLM operates inside nodes — drafting, extracting, summarizing — but never chooses the next node freely; it selects from enumerated edges

When a request needs narrowing — say, "update the agreement" could mean four different documents — the graph doesn't guess. It walks a narrowing tree: which company, which document type, which version. Each question is a node with clickable options, and every answer deterministically selects the next edge. Users experience this as the agent being careful, not slow.

Side by Side

The rule we build by: the LLM writes the content, the graph makes the decisions. Any decision the model makes freely is a decision you can't test.

Where Chains Still Win

We're not dogmatic. Chains are fine when:

• The task is purely generative with no side effects — summarize, rewrite, translate
• You're prototyping to discover what the steps even are, before formalizing them as a graph
• A wrong output costs nothing but a retry

The moment an agent touches real systems — sending, creating, billing, signing — chains are disqualified. You cannot put an approval queue and an audit trail on a process you can't enumerate.

Migrating from Chains to Graphs

1. Log the chain first. Run it in shadow mode and record the actual paths taken. The real graph is in that data.
2. Extract the decisions. Every place the model's output changed what happened next becomes an explicit edge with a condition.
3. Type the node boundaries. Each node gets a defined input and output schema. Structured outputs at every boundary — free-text handoffs are where chains rot.
4. Add fallback edges. Every node needs an explicit "couldn't do it" edge leading to clarification or human handoff. Chains fail silently; graphs must fail loudly.
5. Migrate one action at a time. Run graph and chain in parallel, compare outputs, cut over per action.

The Payoff

Our routing accuracy sits at 95% not because the models got smarter but because the graph made every misroute visible, attributable, and fixable. When a path fails, we fix one node and re-test one node. That maintenance loop — not any single model choice — is what lets a 30-action system keep growing without collapsing under its own complexity.

Determinism isn't a constraint on AI agents. It's what makes them shippable.