Every data integration eventually hits the same wall. You need another system, a warehouse, a search index, a cache, a partner's API, to know when a row changes in your primary database. The easy answer is a scheduled job that polls the table every few minutes and pushes anything new. The easy answer also falls apart the moment your table passes a few million rows, because polling means scanning for changes you cannot cheaply identify, and the fallback of "just lock the table while we read it" turns a data sync into a production incident.
Change data capture, usually shortened to CDC, solves this by reading the database's own transaction log instead of querying the table directly. The log already records every insert, update, and delete in order. A CDC pipeline taps that stream, turns it into events, and lets downstream systems subscribe without ever touching the table they're syncing from. It sounds like infrastructure overkill until you've been paged because a nightly batch job locked orders for four minutes during checkout.
What Change Data Capture Actually Solves
The core problem with polling is that "what changed since last time" is expensive to answer honestly. A naive WHERE updated_at > last_run query misses hard deletes entirely, misses updates that don't touch a timestamp column, and gets slower as the table grows because you're scanning rows to find the few that changed.
Change data capture flips the direction. Instead of asking the table "what's new," you subscribe to a stream the database is already producing as a side effect of normal writes. Every commit generates a log entry regardless of whether anyone is watching, so the pipeline never has to guess. Deletes show up as delete events. Updates carry the actual before and after values, not just a changed timestamp. And because the log is append-only, replaying it after an outage is just re-reading a range, not re-scanning a live table.
Why Naive Polling Locks Your Database
Polling gets blamed for performance problems, but the real damage usually comes from what teams do to make polling accurate. To catch every change reliably, someone eventually adds a SELECT ... FOR UPDATE or wraps the read in a transaction that holds a lock long enough to guarantee consistency. That's fine on a table with a few thousand rows. On a table your application writes to constantly, that lock is now competing with checkout, signups, or whatever your product actually does for a living.
Log-based CDC sidesteps this because reading a transaction log doesn't touch the table's rows or indexes at all. Postgres exposes this through logical replication, MySQL through its binary log, and both are designed to be read by multiple consumers without any coordination with the tables being written to. The PostgreSQL project documents logical decoding specifically because replaying the write-ahead log for replication and for CDC use the identical mechanism, so the write path is never blocked by a reader.
Log-Based CDC: Reading the Transaction Log Directly
At a mechanical level, log-based CDC has three moving parts: a connector that taps the database's log, a message bus that holds the resulting event stream, and consumers that apply those events downstream.
The connector
The connector attaches to the database as a logical replication client (Postgres) or a binlog reader (MySQL), and turns raw log entries into structured change events, one per row modification, tagged with the operation type and the table it belongs to. Debezium is the most widely deployed open source option here and supports Postgres, MySQL, MongoDB, and SQL Server connectors that all produce a consistent event shape, which matters if your stack spans more than one database engine.
The event stream
Connectors write events to a durable, ordered log rather than pushing directly to consumers, because consumers fail, restart, and fall behind at different rates. Apache Kafka is the default choice for this layer specifically because it retains events for a configurable window, so a consumer that's down for an hour can resume exactly where it left off instead of missing changes. Managed alternatives like Confluent exist if you'd rather not operate a Kafka cluster yourselves.
The consumers
Each downstream system, a search index, a cache invalidator, a data warehouse loader, reads from the stream at its own pace and applies changes idempotently. This is the part teams underestimate. If a consumer crashes after applying a change but before committing its read offset, it will see that event again on restart, and your apply logic needs to handle that without creating duplicates.
Choosing a Tool: Debezium, Managed, or DIY
Three realistic paths exist here, and the right one depends more on your team's operational appetite than on the data itself.
Running Debezium against Kafka Connect gives you the most control and the largest community, but you're operating both Kafka and the connector infrastructure yourself, which is a real ongoing cost, not a one-time setup task.
Managed CDC services, cloud-native options from AWS, GCP, and others, remove the operational burden at the cost of flexibility and, often, a meaningfully higher bill once you're moving real production volume. They're a reasonable starting point if your team doesn't want to own streaming infrastructure and your data volume is moderate.
Building it yourself by tailing the write-ahead log directly is rarely worth it unless you have extremely specific requirements Debezium and its ecosystem don't cover. The log formats are documented but genuinely fiddly, and you'll end up reimplementing a subset of what an existing connector already handles correctly, including edge cases around schema changes and replica identity that take longer to get right than expected.
Handling Schema Changes Without Breaking Downstream Consumers
A CDC pipeline is only as reliable as its handling of schema drift. Add a column, rename one, or change a type, and every consumer reading that table's event stream needs to know before the next event arrives, not after it fails to parse.
The practical fix is a schema registry that sits between the connector and the consumers, versioning every schema change and enforcing compatibility rules, so a producer can't publish a breaking change without the pipeline flagging it first. This is more setup work than it sounds like it should be, but it's the difference between a schema change being a non-event and it being a 2am page because three downstream consumers started throwing deserialization errors simultaneously.
A second, cheaper habit helps almost as much: treat every new column as additive by default and require a deliberate decision before removing or renaming one. Consumers that only read fields they explicitly expect are far more resilient to additive changes than pipelines built around "read the whole row."
"The pipelines that actually stay reliable are the ones where someone decided up front what happens when a producer changes its schema, instead of finding out during an incident review." - Dennis Traina, founder of 137Foundry
Delivering Changes Exactly Once (or Close Enough)
True exactly-once delivery across a distributed system is a famously hard guarantee to make honestly. Most production CDC pipelines settle for at-least-once delivery from the broker, paired with idempotent consumers, which gets you the same practical outcome without needing a distributed transaction spanning your database and every downstream system.
The pattern that works: each event carries a unique identifier derived from the log position it came from. Consumers record the last identifier they successfully applied and skip anything at or before it. This turns "we might see this event twice" from a bug into a non-issue, because applying the same change twice produces the same end state rather than a duplicate row or a double-counted total.
Monitoring Lag and Catching Silent Failures
The failure mode that actually hurts is not the pipeline crashing loudly, it's the pipeline falling silently behind while everything looks green. A connector that stops advancing but doesn't error looks identical to a healthy, quiet system until someone notices the search index is three days stale.
Track replication lag as a first-class metric, the gap between when a change committed in the source database and when it landed in the consumer, not just whether the connector process is alive. Alert on lag crossing a threshold, not on process death, since a stuck connector often keeps running without making progress. Most teams route this through the same monitoring stack they already use for application metrics, which keeps CDC observability from becoming a separate, forgotten system.
A Practical Rollout Plan
Start with a single table and a single consumer, not your whole schema. Pick something with moderate write volume and a downstream system that already tolerates brief staleness, so mistakes during rollout are cheap. Get the connector, the event stream, and one idempotent consumer working end to end before adding a second table.
Once that path is proven, expanding to more tables is mostly configuration, but expanding to more consumers is where schema discipline and lag monitoring start to matter in practice rather than in theory. Teams that skip straight to "capture everything" usually end up debugging schema drift and duplicate handling across a dozen tables simultaneously instead of learning those lessons once, cheaply, on a single low-risk table.
If you're weighing whether CDC is worth the setup cost against a simpler batch job, the honest answer depends on how much staleness your downstream systems can tolerate and how much a locked table actually costs you during a sync window. For a lot of teams, that calculation shifts hard toward CDC the first time a batch sync collides with peak traffic.
We've walked several client systems through exactly this migration as part of our data integration engagements, usually starting from the same painful trigger: a nightly job that used to run fine started colliding with real usage as the product grew. If you want a sense of how 137Foundry scopes engineering work like this, or want to know who's actually behind this, both are a click away.
Change data capture is not free. It adds a connector, a stream, and a schema registry to your infrastructure, and none of those run themselves. But compared to the alternative, a batch job that gets slower every quarter and occasionally locks a table your product depends on, it's infrastructure that pays for itself the first time it quietly prevents an incident instead of causing one.
