The Hidden Infrastructure Tax in AI Data Pipelines: What CDC at Scale Teaches Us About MLOps Resilience

Most MLOps post-mortems arrive at the same convenient conclusion: the model drifted, the features were stale, the training data was noisy. What they rarely surface is the pipeline architecture that allowed stale, noisy, or missing data to reach the model in the first place. Change data capture at scale is one of the most operationally demanding patterns in data engineering, and the failure modes it exposes - WAL retention blowouts, replication slot invalidation, backpressure under concurrent load - are structurally identical to the failure modes that quietly undermine feature stores and training pipelines in production ML systems. Engineering leaders who understand one will recognise the other.

Why CDC Failures Are a Mirror for ML Pipeline Fragility

PostgreSQL logical replication works by holding changes in the write-ahead log until a downstream consumer acknowledges receipt. When that consumer falls behind, the WAL grows. When it grows long enough, the database either runs out of disk or the operator intervenes and drops the replication slot. Either outcome means the consumer restarts from scratch, which in a backfill scenario covering months of history can mean days of re-processing against a live production database.

The ML equivalent is a feature pipeline that reads from a transactional source during a historical backfill to bootstrap a new feature store partition. If the read throughput is not carefully governed, the source database experiences the same pressure: lock contention, query queue buildup, and degraded latency for application traffic that shares the same connection pool.

The mechanism is the same in both cases. An unbounded consumer applies uncontrolled pressure to a stateful source, and the source fails in ways that are opaque until they are catastrophic.

WAL Retention and the Analogous Problem of Unbounded Feature Lag

Replication Slot Invalidation

In PostgreSQL, a replication slot that has not been consumed within max_slot_wal_keep_size is invalidated automatically in newer versions, or holds the WAL indefinitely in older ones. Both outcomes are bad. Invalidation forces a full re-sync. Unbounded WAL retention risks taking down the primary. The operator is choosing between two failure modes rather than preventing them.

Feature pipelines exhibit the same pattern when a streaming consumer falls behind its source topic. Kafka retention windows are finite. If a Flink or Spark Structured Streaming job pauses for longer than the retention period, the offset is lost and the pipeline must backfill from a cold store. That backfill, if unthrottled, then applies the same source pressure the original streaming job was designed to avoid.

Chunk Sizing and Throughput Governance

CDC backfills over large tables require explicit chunk sizing to prevent long-running transactions from bloating the WAL or holding row-level locks. A naive implementation reads the entire table in a single transaction. A production implementation reads in bounded chunks, commits between them, and tracks progress in a resumable cursor.

The analogous discipline in ML pipelines is micro-batch sizing in historical feature computation. Teams that compute features over multi-year datasets without partitioning the workload routinely hit memory limits, produce partial outputs without checkpointing, and have no clean way to resume after failure. The operational requirement is identical: bounded units of work, explicit progress tracking, and a resumption strategy that does not require restarting from zero.

Concurrent Load and the Source Pressure Problem

Running a historical backfill against a live production database is not a data engineering problem. It is a systems design problem. The backfill and the application share I/O bandwidth, connection pool slots, and buffer cache. Under concurrent write load, a poorly timed backfill read can evict hot pages from shared buffers, forcing subsequent application queries to go to disk.

ML training pipelines that read directly from operational databases during business hours impose the same cost. The training job is treated as a background process, but it competes for the same physical resources as the application serving users. The consequence is not a training failure, which would at least be visible. The consequence is degraded application performance, which gets attributed to infrastructure load rather than the ML team's pipeline.

The discipline required to run CDC backfills safely, rate-limited reads, off-peak scheduling, connection pool isolation, and progress-aware resumption, is exactly the discipline that prevents training pipelines from becoming a hidden tax on production systems.

What Operational Maturity Actually Looks Like

Teams that operate CDC reliably at scale have solved a specific class of problems: they monitor WAL lag as a first-class SLO, they test replication slot recovery in staging, and they have runbooks that distinguish between a consumer falling behind and a slot being invalidated. These are not heroic engineering achievements. They are the baseline operational practices that prevent 2am incidents from becoming multi-day recovery efforts.

ML platform teams need the same baseline applied to their own primitives. Feature pipeline lag should be monitored against defined thresholds, not discovered when a model starts producing anomalous predictions. Backfill jobs should be resumable by design, not by convention. Source pressure should be measured and bounded before a pipeline goes to production, not after the first incident.

The gap between these two disciplines is not technical. It is organisational. CDC infrastructure is owned by data engineering teams who have been burned by these failures before. ML pipelines are often owned by teams who have not yet accumulated the operational scar tissue to design for them proactively.

Closing the Gap Between Data Engineering Discipline and MLOps Practice

The practical implication for engineering leaders who own both functions is that the operational standards already applied to CDC infrastructure should be extended explicitly to ML data pipelines. That means treating feature pipelines as production systems with SLOs, not as batch jobs with best-effort delivery. It means requiring that any pipeline reading from a transactional source demonstrates it has been load-tested under concurrent write conditions before it is promoted to production.

It also means that post-mortems for ML system failures should include the pipeline architecture in scope, not just the model or the feature values. A model that receives stale features because a pipeline fell behind its source is not a model quality problem. It is a pipeline reliability problem, and it will recur until the pipeline is treated with the same operational rigour as the systems it depends on.

Engineering leaders who close this gap do not get fewer incidents. They get incidents that are faster to diagnose, easier to recover from, and less likely to be misattributed to the model when the fault lies upstream.

Companion piece to our broader work on ML pipeline operational reliability. See Your ML Pipeline Is a Production System for how to apply production-grade incident response standards to ML orchestration failures and batch scheduling breakdowns.