The Thundering Herd Problem Your MLOps Team Is Probably Ignoring

Most MLOps teams can tell you their p99 inference latency, their GPU utilisation ceiling, and the exact point at which their autoscaler kicks in. What they often cannot tell you is why their serving infrastructure occasionally collapses under load that, on paper, should be well within capacity. The answer, more often than not, has nothing to do with model complexity or hardware limits. It is a classical distributed systems failure mode, sometimes called the thundering herd problem, playing out quietly inside the timing assumptions baked into their inference stack.

Companion piece to our broader work on ML pipeline reliability and operational cost. See ML Pipeline Reliability: Production Incident Response for how to apply production-grade incident response standards to ML pipelines.

What the Thundering Herd Actually Looks Like in ML Inference

The original thundering herd problem describes what happens when a large number of processes, all sleeping and waiting for the same event, are woken simultaneously and compete for a shared resource. In web infrastructure, this typically surfaces when a cache expires and hundreds of requests simultaneously attempt to recompute the same value. In ML inference infrastructure, the dynamics are structurally identical but the triggers are more varied and less visible.

Consider a serving cluster with thirty inference pods. Each pod runs a health check on a fixed ten-second interval. If those pods were all started within the same deployment window, their health check timers are perfectly synchronised. Every ten seconds, all thirty pods simultaneously hit the load balancer, the upstream model registry, or the feature store. The load profile that results is not a smooth curve. It is a series of sharp spikes, each one self-inflicted by the infrastructure's own internal clock.

The same pattern emerges with TTL-based cache expiry for model artefacts, with fixed-interval polling for configuration updates, and with batch inference jobs scheduled to the nearest minute boundary. None of these individually looks dangerous. Together, they create a coordination problem that no amount of horizontal scaling will fully absorb, because scaling adds more synchronised participants, not fewer.

Why Fixed Intervals Are the Default and Why That Is a Problem

Fixed intervals are the default because they are easy to reason about. A ten-second health check interval is simple to configure, simple to monitor, and simple to explain in a runbook. The operational cost of that simplicity is that it transfers coordination burden from the configuration layer to the infrastructure layer, invisibly.

When a shared downstream dependency, a feature store, a model registry, or a Redis cache, receives a burst of simultaneous requests, it must either queue them, reject them, or serve them under contention. In each case, the tail latency for inference requests that depend on that response degrades. The degradation is periodic and correlated with the interval, which makes it easy to misread as a traffic pattern rather than an infrastructure artefact.

The commercial implication is that teams respond to these spikes by provisioning more capacity. That capacity sits idle between spikes, burning budget without improving steady-state performance. The root cause goes unaddressed.

Jitter: The Mechanism and the Implementation

Jitter is the practice of adding a random offset to a fixed interval so that events that would otherwise be synchronised are spread across a time window. The fix is genuinely low-cost. For most interval-based operations in ML infrastructure, adding jitter requires changing a single configuration value or a single line of scheduling code.

Cache TTL Jitter

For TTL-based model artefact caches, the standard approach is to set a base TTL and add a uniformly distributed random value within a defined range. If the base TTL is sixty seconds and the jitter window is twenty seconds, cache expirations across thirty pods will be spread across a forty-second window rather than occurring simultaneously. The downstream dependency sees a steady trickle of refresh requests instead of a synchronised burst.

Health Check and Polling Jitter

For health checks and configuration polling, the jitter is best applied at startup rather than at each interval. Each pod draws a random initial delay from a uniform distribution before beginning its polling cycle. Because the initial offset persists, the pods remain desynchronised for the lifetime of the deployment, not just the first cycle.

Batch Scheduling Jitter

For batch inference jobs, the risk is slightly different. Jobs scheduled to fixed cron boundaries, every hour on the hour, every fifteen minutes at quarter past, create GPU contention and queue saturation at predictable times. Introducing a random start offset within an acceptable tolerance window eliminates the contention without materially affecting job completion times.

What Jitter Does Not Solve

Jitter is a coordination fix, not a capacity fix. It smooths the distribution of load across time but does not reduce the total volume of work. If the aggregate request rate genuinely exceeds infrastructure capacity, jitter will delay the failure but not prevent it.

It is also worth being precise about where jitter applies. Idempotent, stateless operations, cache refreshes, health checks, polling cycles, are safe candidates. Operations that require strict ordering or transactional consistency are not. Applying jitter indiscriminately to operations that have sequencing dependencies introduces a different class of failure.

The practical boundary is straightforward: if the operation would produce the same result regardless of when within a window it executes, jitter is safe. If the result depends on the order of execution relative to another operation, jitter requires more careful analysis before being applied.

Operationalising Jitter Across an Inference Platform

The highest-value place to introduce jitter is wherever a fixed interval touches a shared resource. In a typical inference platform, that means the model registry client, the feature store polling loop, the inference pod health check configuration, and the batch job scheduler.

The implementation cost is low precisely because these are configuration-layer changes. They do not require model redeployment, infrastructure reprovisioning, or changes to serving logic. A platform team can instrument the change, observe the load profile on the shared dependencies, and confirm the spike reduction within a single deployment cycle.

The monitoring signal to watch is not aggregate throughput but the variance in request rate to shared dependencies over time. A successful jitter implementation will show a flatter request distribution and a corresponding reduction in tail latency on those dependencies during what were previously spike windows. That reduction propagates upstream into inference latency percentiles without any change to model serving capacity.