GraphRAG vs Vector RAG: Choosing the Right Retrieval for Your Agent

The GraphRAG hype cycle has convinced a lot of teams they need to replace their vector pipeline with a knowledge graph. Most of them don't. We run both in production — 38,000+ embeddings for vector retrieval, Neo4j for graph retrieval — and the honest answer is that they solve different problems and fail in different places.

The Core Difference

Vector RAG retrieves chunks that sound like the question. GraphRAG retrieves facts connected to the entities in the question. That single sentence predicts almost every quality difference you'll measure.

Ask "what did we agree with the client about payment terms?" and vector RAG shines: the answer lives in one or two chunks, semantically close to the query. Ask "how is this vendor connected to our other projects?" and vector RAG returns chunks that mention the vendor — but the answer isn't in any chunk. It's in the relationships between records that never appear in the same paragraph.

Side by Side in Production

The Cost Asymmetry Nobody Mentions

Embedding a document costs a fraction of a cent and never needs a second pass. Extracting entities and relations from that same document requires an LLM call at ingest time — in our pipeline roughly 30–60x the cost of embedding it. Across nine data sources with sync workers running every 10–60 minutes, that difference compounds fast. We only run graph extraction on record types where relationships matter: tasks, contacts, companies, contracts. Chat messages and routine email get embeddings only.

Embed everything. Graph selectively. Extraction cost, not query cost, is what kills naive GraphRAG deployments.

The Hybrid Pattern We Actually Ship

Our router (Claude Haiku, ~300ms) classifies each query before retrieval:

1. Lookup intent → vector search, top-k chunks, done. About 70% of traffic.
2. Relationship intent — the query names 2+ entities or uses connective language ("related to", "who else", "across") → graph traversal from resolved entities, with vector search filling in supporting text. About 20%.
3. Ambiguous → both run in parallel and the synthesis prompt gets both result sets, labeled by source. About 10%.

The parallel branch is cheap insurance. When the two retrievers disagree about what's relevant, the model sees both and cites what it used — and disagreements are a decent proxy for questions worth logging and reviewing.

Quality Numbers From Our Eval Set

On our internal eval of 120 real user queries, vector-only answered single-document questions correctly at a rate the graph never matched, while graph-augmented retrieval roughly doubled correct answers on multi-entity questions. Neither wins overall — the split depends entirely on your query mix. Measure your own before committing to an architecture.

Decision Checklist

• Single knowledge base, document Q&A → vector RAG only. Stop there.
• Multiple systems, users ask cross-system questions → hybrid
• Your product IS the relationship map (due diligence, org intelligence, fraud) → graph-first
• No capacity for entity resolution upkeep → vector only, regardless of the above

GraphRAG isn't an upgrade to vector RAG. It's a second retrieval engine with its own cost curve and its own failure modes. The teams getting value from it are the ones who added it for specific question types they could name in advance — not the ones who swapped architectures because a benchmark said to.