Vector Embeddings for Business Search: 38,000 Vectors Across 9 Systems

Businesses don't have a knowledge base. They have nine: a task tracker, two email inboxes, a document drive, chat, a CRM, spreadsheets, contracts, meeting notes. Keyword search fails across all of them because every team names the same thing differently. Semantic search fixes this — and it's cheaper and simpler than most teams assume.

Our production system holds 38,000+ vectors covering nine connected systems. Here's what mattered and what didn't.

The Model Choice Is the Easy Part

We use OpenAI's text-embedding-3-small (1536 dimensions) for everything. We tested the large variant and saw no measurable retrieval improvement on business documents — emails, tasks, contracts — at 6.5x the price. Business text is not subtle literature; the small model separates it fine. Embedding our entire corpus cost under $3. Re-embedding everything after a chunking change costs the same, which means you can iterate freely.

Chunking: Match the Unit of Meaning

The biggest quality lever wasn't the model — it was chunking per record type instead of one global strategy:

The chat insight took us longest to learn: "yes, sounds good" is a useless vector. Embedding a rolling conversation window made chat retrieval go from noise to one of our most-cited sources.

Metadata Filters Do Half the Work

Every vector carries structured metadata: source system, record type, company ID, author, timestamp, permission scope. Most real queries are filtered semantic searches — "discussions about renewal for this client, in the last quarter." Pre-filtering on metadata before the similarity ranking cut our irrelevant-result rate roughly in half, more than any embedding-side tweak. If your vector store makes metadata filtering awkward, that's a reason to switch stores.

A mediocre embedding with good metadata beats a perfect embedding with none. Business search is always scoped search.

Keeping 9 Sources Fresh

Stale vectors are worse than missing ones — the agent confidently cites a task status from last week. Our sync workers embed on a per-source cadence: tasks every 10 minutes, email every 15, documents hourly, chat every 30. Each worker diffs against a content hash, so unchanged records cost nothing. Updates replace vectors (delete + insert keyed by source ID); we never append versions. Deletions propagate on the same cycle — leaking vectors from deleted records is a real confidentiality bug, not a hygiene detail.

What We'd Tell a Team Starting Today

1. Start with pgvector if you're on Postgres. At 38k vectors we're nowhere near needing a dedicated vector DB. Query time is ~50ms with an HNSW index. Dedicated stores earn their keep in the millions, not thousands.
2. Store the source text and a deep link with each vector. Retrieval that can't show provenance won't be trusted, and untrusted search doesn't get used.
3. Build a 50-query eval set from real user questions before tuning anything. Every chunking argument we had was settled in ten minutes by the eval, not by opinion.
4. Log every query and the top results. The queries that return garbage tell you which source needs better chunking next.

The counterintuitive summary: at business scale, embeddings are a solved, cheap commodity. All the engineering value lives in chunking, metadata, and sync — the unglamorous parts that never make it into vendor demos.