Why Your Reasoning Models Slow Down on Hard Problems and What That Means for Production AI Design

When reasoning models take longer on harder tasks, the instinct in most engineering teams is to treat it as a performance problem. Tune the prompt, swap the model, adjust the temperature, and hope the latency comes down. Recent research suggests that instinct is wrong, and acting on it will cost you time and money without addressing the underlying mechanism.

The Physics Behind the Slowdown

A paper published in September 2026 by Lai, Bao, Quinn, and Gilpin offers a precise explanation for why reasoning models slow down on difficult tasks. Their finding is that reasoning models exhibit transient chaos, a property borrowed from dynamical systems theory, where the model's internal state wanders for extended periods before converging on an answer (Lai et al., arXiv 2026).

The mechanism is specific. During inference on hard problems, the model's reasoning trace becomes trapped near saddle points in its state space. These saddle points correspond to nearly-correct solutions: configurations that look locally plausible but are not quite right. The model keeps revisiting and adjusting, which is what produces the characteristic "overthinking" pattern observed in production.

The consequence is that the relationship between task difficulty and inference cost is not linear and not controllable through conventional optimisation. It is a structural feature of how these models process complexity.

Fractal Basins and What They Mean Operationally

Lai et al. show that reasoning models are dynamical systems with fractal basin boundaries. A fractal basin means that the boundary between regions that lead to correct answers and regions that lead to incorrect ones is infinitely complex at fine scales. Small changes in the initial state of a problem can produce very different reasoning trajectories.

This has a direct operational implication. For problems near the boundary of a model's competence, inference time becomes highly variable and difficult to predict in advance. You cannot reliably estimate how long a given query will take without knowing where it falls relative to those boundaries, and the boundaries themselves are fractal.

The fractality also increases with task difficulty, which the authors demonstrate across a range of tasks including Sudoku, maze solving, visual puzzles, and mathematical logic. The harder the problem class, the more unpredictable the inference cost distribution becomes.

Why Latency Optimisation Misses the Point

Most production latency work focuses on infrastructure: batching, quantisation, caching, hardware selection. These interventions genuinely reduce the baseline cost of inference, and they are worth doing. But they do not change the shape of the cost distribution for hard problems.

If the slowdown is caused by transient chaos near saddle points, then optimising the hardware or the prompt does not remove the saddle points. It may reduce the time spent at each step, but the number of steps remains a function of the problem's computational complexity. You are compressing a distribution whose tail is determined by mathematics, not infrastructure.

The practical implication is that engineering teams chasing p99 latency on reasoning-heavy workloads are often solving the wrong problem. The long tail is not noise to be eliminated. It is signal about where the model is struggling.

Rethinking Latency Budgets by Task Class

The right response is to design systems with task-class-aware cost and time ceilings rather than global latency targets. This requires classifying workloads before they reach the model, not after.

Classifying Problem Hardness Upstream

A lightweight classifier or heuristic layer that estimates problem difficulty before routing to a reasoning model gives you the information needed to apply appropriate ceilings. Easy problems get a tight budget. Problems estimated to be near the model's competence boundary get a higher ceiling or are routed to a different model entirely.

Setting Ceilings, Not Targets

A latency target assumes you know what the right answer looks like and how long it should take. A ceiling accepts that some problems will hit the limit and returns a best-available answer rather than waiting indefinitely. For most commercial applications, a bounded answer delivered on time is more valuable than an optimal answer delivered unpredictably.

Model Selection as a Complexity Decision

Choosing a reasoning model is not just a capability decision. It is a decision about where the fractal basin boundaries sit relative to your task distribution. A model that handles your hardest tasks with less transient chaos may cost more per token but produce a tighter inference cost distribution, which is often the commercially correct trade-off for latency-sensitive applications.

What Engineering Leaders Should Change

The research by Lai et al. does not suggest that reasoning models are impractical. It suggests that the mental model most teams use to evaluate and operate them is incomplete.

Inference cost predictability matters as much as average inference cost, particularly for applications with SLA commitments or real-time requirements. A model that is cheap on average but has a fat tail on hard problems may be more expensive in practice than a model with a higher average cost and a tighter distribution.

The teams that will design better production systems are those that stop treating inference latency as a single number to be minimised and start treating it as a distribution to be understood, segmented by task class, and managed with appropriate ceilings per segment. That is a different engineering discipline, and it starts with accepting that some slowdowns are not fixable because they are not mistakes.