Data Sync Pipelines for AI Agents: Keeping 9 Sources Fresh

Users forgive an AI agent for not knowing something. They don't forgive it for knowing something wrong — citing a closed task as open, or last month's contract draft as current. Which means the unglamorous sync layer, not the model, sets the ceiling on how much your agent gets trusted. Ours keeps nine business systems flowing into Postgres, a vector index of 38,000+ embeddings, and a knowledge graph. Here's the design that survived a year of production.

Cadence Should Match Decay Rate

Not all data goes stale at the same speed. We stopped debating "how often should we sync" and started asking "how fast does this data rot, and how expensive is the API call?"

Workers are staggered so nine jobs never fire in the same minute — on a single Railway-hosted Express service, self-inflicted thundering herds are the dumbest possible outage, and we had one before we staggered.

Incremental Always, Full Sync Nightly

Every worker follows the same contract:

1. Pull records changed since the stored cursor (updated-at timestamp or change token, per API)
2. Compute a content hash per record; skip if unchanged — some APIs report "updates" that are pure noise
3. Upsert to Postgres keyed by (source, external_id)
4. Enqueue downstream work: re-embed, update graph edges
5. Advance the cursor only after downstream writes commit

Step 2 matters more than it looks: content hashing cuts our embedding volume by roughly 80%, because most "changed" records haven't meaningfully changed. And once a night, each source runs a full reconciliation pass — because incremental sync cannot see deletions in most APIs. The nightly diff catches deletes and any drift from missed webhooks. Deleted records must leave the vector store too; orphaned embeddings of deleted documents are a confidentiality incident waiting to happen.

Webhooks Are an Optimization, Not a Foundation

Webhooks reduce latency. Polling guarantees consistency. Build polling first, add webhooks second, and never trust a webhook you can't reconcile against.

Every webhook integration we run has silently dropped events at some point — provider outages, our own deploys, expired subscriptions. We use webhooks to trigger an immediate targeted sync of the affected record, while the scheduled workers remain the source of truth. This "webhook as a hint" pattern gives near-real-time freshness on hot records without betting correctness on delivery guarantees no provider actually honors.

Failure Is Per-Source, Visibility Is Global

The most important architectural rule: one connector's failure must not degrade the other eight. Each worker has independent retry with exponential backoff, its own rate-limit budget, and a health row (last success, last error, records processed, lag). Three consecutive failures alert us. OAuth token expiry is its own alert class — it's the #1 real-world cause of dead connectors, and it fails silently for days if you let it.

We surface freshness to users too: the agent's answers carry "as of N minutes ago" per source. When email sync is degraded, the agent says so instead of pretending completeness.

What We'd Skip Next Time

• Kafka/queues on day one — cron-scheduled workers on one service handled 9 sources fine for our volumes; add infrastructure when a number tells you to
• A generic connector framework — nine bespoke ~200-line workers sharing utilities beat the abstraction we tried to design up front
• Syncing everything — half our early pipeline synced fields no query ever touched; start from the questions users ask and work backwards

None of this is exciting engineering. All of it is why the agent's answers hold up when a user cross-checks them against the source system — and that check is the moment your AI product is actually won or lost.