How do you optimize database change data capture without blowing up latency or cost?

What Change Data Capture Actually Does in 2026

Change data capture, or CDC, is the pattern of streaming row-level inserts, updates, and deletes from an operational database into downstream systems as those changes occur. Modern CDC tools read the database's own transaction log rather than polling tables with SELECT statements, which is why the technique has become a default primitive in event-driven stacks. Databricks describes CDC as the mechanism that keeps a destination system synchronized with source-of-truth rows, and Snowflake's engineering team has published detailed write-ups showing how they pushed CDC into Postgres itself to make replication deterministic. The underlying principle is straightforward: instead of asking the source database what changed, you let the database tell you, by reading its write-ahead log (WAL), binlog, or equivalent.

Also worth reading: How to tune PostgreSQL CDC latency for real-time data pipelines in 2026? · postgresql cdc performance 2026: what are the real bottlenecks and how to optimize throughput? · How do you optimize churn prediction model performance in practice?

For a B2B customer-signal inbox that aggregates product feedback, support tickets, and in-app surveys, CDC is the layer that guarantees a support agent's view of a customer is never more than a few seconds stale. Without it, dashboards drift, SLAs slip, and product teams make roadmap calls on data that is already out of date. The risk is that naively configured CDC pipelines can introduce their own latency, double-charge storage costs, or hammer the primary database during peak traffic. Optimization is not optional; it is what separates a CDC pipeline that scales to 50,000 events per second from one that collapses under a campaign launch.

The Core Levers for CDC Optimization

There are four primary levers that determine the throughput, latency, and cost of a CDC pipeline: the log reader, the schema-change handling policy, the batching and backpressure configuration, and the destination write strategy. Each lever has measurable trade-offs, and the wrong combination is responsible for the majority of performance problems teams encounter in production.

The log reader is the component that tails the database's transaction log. In Postgres, that means reading the WAL via logical replication slots; in MySQL, the binlog via the mysqlbinlog API; in MongoDB, the oplog. Performance here is bounded by how much WAL the database is willing to retain, how often the reader commits its position back to the source, and whether the reader holds long-running transactions. AWS's documentation on Debezium with Aurora PostgreSQL explicitly warns that unmanaged replication slots will retain WAL indefinitely and eventually fill the disk, which is one of the most common production incidents in CDC deployments. Optimizing this lever means setting slot retention to a known finite window, using heartbeat tables to advance the slot during idle periods, and monitoring slot lag as a first-class SLO.

Schema-change handling is the second lever, and it is the one that causes the most outages. When a column is added, renamed, or dropped upstream, naive pipelines either crash, silently drop the change, or write corrupt records to the destination. The optimization here is to configure the connector to evolve with the schema using a registry such as Confluent Schema Registry or AWS Glue Schema Registry, and to use Avro or Protobuf rather than JSON for the wire format. Snowflake's own engineering team has documented how they allow upstream Postgres ALTER TABLE operations to flow through the CDC stream without manual intervention, which only works if the schema is versioned end-to-end.

Batching, Compression, and Throughput Numbers

CDC throughput is a function of how many change events you can pack into a single network round trip without violating your latency budget. Realistic numbers from production systems in 2025-2026: a well-tuned Debezium connector on a 16 vCPU Postgres primary can sustain 20,000-40,000 events per second on a single partition, with end-to-end latency of 800-1,500 milliseconds when writing to Kafka with LZ4 compression and batching enabled. Turning batching off drops latency to under 200 milliseconds but caps throughput at around 3,000-5,000 events per second because each event becomes its own network request. The optimization is almost always to right-size the batch window for the workload rather than to choose one extreme.

Compression matters more than most teams expect. LZ4 typically achieves 2-3x compression on CDC payloads, which directly reduces Kafka broker disk costs and inter-AZ data transfer charges. Zstandard (zstd) reaches 3-5x at the cost of roughly 20% more CPU on the connector. Snappy sits between the two. For a pipeline processing one billion events per day, the difference between 2x and 5x compression is the difference between 2 TB and 800 MB of broker storage per day, which compounds into meaningful infrastructure spend over a quarter. Oracle's GoldenGate documentation for OCI explicitly recommends zstd for replication to Google Cloud where network egress dominates the cost model.

The destination write strategy is the final lever, and it is often the most under-optimized. Writing CDC events to a row-oriented warehouse such as Postgres, MySQL, or Aurora is a different problem from writing to a columnar warehouse such as Snowflake, BigQuery, or Redshift. Row stores require small batches with frequent commits to avoid long-running transactions; columnar stores reward large batches with infrequent commits because the per-commit overhead dominates. The 2025-2026 best practice is to use a merge-on-read pattern for row stores and a copy-on-write or append-only pattern for columnar stores, and to size the batch flush interval to roughly 5-10 seconds for sub-second SLAs or 30-60 seconds for minute-level SLAs.

Comparison of CDC Approaches

Log-based CDC is the default for serious workloads. Trigger-based CDC, which writes change rows to a shadow table on every DML, is easier to set up but adds 5-15% write overhead on the source database and is a frequent source of deadlocks. Query-based CDC, which polls a last_modified column, is acceptable for tables under 10 million rows but degrades badly as the index grows. For a customer-signal inbox that needs to sync support tickets, product events, and survey responses in near real time, log-based is the only option that holds up at scale.

Practical Steps to Optimize a CDC Pipeline

The first step is to baseline the current pipeline with three numbers: end-to-end p50 and p99 latency from source commit to destination visibility, sustained events-per-second at peak, and source database CPU utilization attributable to the CDC reader. Without these baselines, optimization is guesswork. Most teams discover that p99 latency is 10-20x higher than p50, which points to batching, backpressure, or destination contention as the actual bottleneck.

The second step is to tune the connector's max.batch.size, max.queue.size, and poll.interval.ms parameters. For Debezium on Postgres, a reasonable starting point is a batch size of 2,048 events, a queue size of 8,192, and a poll interval of 100 milliseconds. These values are workload-dependent; a workload dominated by small updates needs smaller batches, while a workload dominated by bulk imports needs larger batches. The third step is to enable compression on both the Kafka producer and the destination writer, and to verify the actual compression ratio on a production sample rather than trusting the codec's marketing claim.

The fourth step is to partition the topic by a stable key, typically the primary key of the source table, so that events for the same row land on the same partition and preserve order. Without partitioning, out-of-order updates can overwrite newer data with older data, which is the most common silent correctness bug in CDC pipelines. The fifth step is to monitor the replication slot lag in Postgres, the binlog lag in MySQL, and the oplog window in MongoDB as hard alerting thresholds. A replication slot that grows past 24 hours of WAL is a paging incident; the disk will fill within hours, and the source database will refuse writes. AWS documents this failure mode repeatedly in its Debezium-on-Aurora guide.

Common Mistakes That Sabotage CDC Performance

The single most common mistake is leaving the replication slot unmanaged. Postgres will retain WAL forever as long as a logical slot exists, and an idle consumer that fails to commit will eventually exhaust the primary's disk. The fix is mechanical: set a finite wal_keep_size or use a monitoring agent that drops abandoned slots, and never deploy CDC to a production database without a runbook for slot recovery. The second most common mistake is treating the destination as a write-once append-only log when it is actually a transactional store. Writing CDC events to Postgres without an ON CONFLICT clause or upsert pattern produces duplicate rows within minutes and corrupts downstream analytics.

The third mistake is ignoring the cost of cross-AZ and cross-region data transfer. CDC payloads that flow from a primary in us-east-1 to a Kafka cluster in us-west-2 incur per-GB egress charges that can exceed the cost of the compute itself. At one billion events per day with an average payload of 500 bytes, the monthly egress is roughly 15 TB, which translates to $1,200-$1,500 per month on AWS at standard rates. Compressing the stream and keeping the pipeline within a single region where latency requirements allow can cut this cost by 60-80%.

The fourth mistake is failing to test schema changes. A ALTER TABLE ADD COLUMN upstream, if not handled by the schema registry, will cause the connector to either crash or silently drop the change, and the failure is often invisible until a downstream report shows stale data weeks later. The fix is to version every schema, run a CDC-compatible migration tool, and rehearse schema changes in staging with a real production-shaped dataset. The fifth mistake is over-instrumenting the pipeline with synchronous monitoring that adds latency to every event. Asynchronous metrics, sampled tracing, and log-based alerting are nearly always the right answer for hot paths.

When to Act and What It Costs

A team should treat CDC optimization as urgent when the destination lag exceeds the SLA, when the source database shows more than 5% CPU overhead attributable to the CDC reader, or when the monthly infrastructure cost of the pipeline exceeds the cost of the source database itself. None of these are acceptable steady states, and all three are fixable within a sprint by the levers described above. A typical 4-week optimization engagement for a mid-sized B2B SaaS produces a 3-5x throughput improvement, a 50-70% reduction in p99 latency, and a 30-50% reduction in infrastructure cost, based on published case studies from Debezium, Confluent, and Snowflake customers in 2024-2025.

Pricing for managed CDC services in 2026 ranges widely. Debezium itself is open-source and free, but the surrounding Kafka or Pulsar cluster typically costs $0.10-$0.30 per million events processed on a managed cloud. AWS DMS charges roughly $0.40-$0.80 per hour per replication instance plus data transfer. Oracle GoldenGate on OCI is priced by connection and throughput tier, typically $1,000-$5,000 per month for a production-grade setup. Fivetran and Airbyte charge by monthly active rows, with CDC streams at $0.10-$0.25 per million rows synced. For a customer-signal inbox processing 100 million events per month, the all-in CDC infrastructure cost realistically lands between $800 and $3,000 per month depending on the chosen stack and region.

Closing Perspective

CDC is one of the few infrastructure patterns where the difference between a working and an optimized implementation is measurable in dollars and in user-visible latency, but the optimization is not exotic. It comes down to disciplined log reader management, right-sized batching, end-to-end schema versioning, and a destination write pattern matched to the storage engine. Teams that skip the baseline measurement end up tuning the wrong knob, and teams that treat CDC as a set-and-forget utility end up debugging silent data corruption a quarter later. The pipeline that survives a Black Friday traffic spike is the one whose owners knew its three numbers, watched its three dashboards, and rehearsed its three failure modes before they happened. Everything else is paperwork.