LLM Probabilistic Forecasting Is Broken: What Engineering Leaders Need to Know Before Deploying Decision-Support AI

Enterprise teams are shipping LLM-powered decision-support tools at pace, and most are doing so without asking a foundational question: are the model's probability outputs internally consistent? The answer, based on recent empirical work, is often no. The incoherence is not a minor calibration issue. It is structural, it compounds under realistic operating conditions, and it has direct consequences for any workflow where LLM forecasts inform capital allocation, credit decisions, or operational risk.

Companion piece to our broader work on LLM reliability in production. See LLM Prediction Instability: Beyond Aggregate Metrics for per-example reliability methods, tail risk identification, and governance for enterprise deployment.

What Probabilistic Coherence Actually Means

A set of probability forecasts is coherent if it is consistent with some underlying probability distribution. The classical test for this comes from de Finetti: if a forecaster's stated probabilities allow a counterparty to construct a set of bets that guarantees a profit regardless of outcomes, those probabilities are incoherent. This guaranteed-profit construction is called a Dutch book.

The Dutch book framing matters for engineering leaders because it is outcome-agnostic. You do not need to wait for events to resolve to detect the problem. If a model assigns 60% to event A, 55% to event B, and 30% to the joint event of A and B occurring together, the arithmetic is already broken. An arbitrageur can extract guaranteed profit from those numbers before a single outcome is observed.

Andrews and Sarkar tested this directly against LLM outputs using stock return events and linear programming to compute the maximum guaranteed Dutch book profit (Andrews et al., arXiv 2026). Substantial incoherence was present across models. That result should be read as a baseline finding about the architecture of current LLMs, not as a property of one vendor or one prompt style.

How Richer Context Makes the Problem Worse

The most operationally significant finding is the relationship between context richness and incoherence. When events have richer logical relationships between them, incoherence increases. When irrelevant contextual details are added, incoherence can increase by an order of magnitude (Andrews et al., arXiv 2026).

This is the opposite of what most enterprise teams assume. The intuition is that more context should help a model reason better. In practice, additional context introduces more logical relationships that the model must hold consistent simultaneously, and LLMs do not have a mechanism that enforces that consistency at inference time.

The enterprise implication is direct. Production decision-support systems are typically rich in context. A credit risk workflow might pass in macroeconomic indicators, borrower history, sector data, and market conditions simultaneously. An operations tool might combine supplier lead times, inventory levels, and demand signals. These are exactly the conditions under which incoherence is most severe.

Why This Matters for Finance, Operations, and Risk Functions

In quantitative finance, incoherence in probability estimates is not an abstract concern. A model that assigns inconsistent probabilities to correlated asset events can produce hedging recommendations that are internally contradictory. A risk function relying on those outputs may believe it is covered when it is not.

In credit risk specifically, the problem is compounded by regulatory expectations. Models used in IFRS 9 or Basel frameworks are expected to produce well-calibrated, internally consistent probability of default estimates. An LLM generating PD forecasts that fail a Dutch book test would not satisfy that standard, and the failure would likely be invisible to teams measuring only aggregate accuracy or AUC.

In operations, the risk is subtler but still material. Inventory and procurement decisions are often made on conditional probability reasoning: if demand exceeds X, what is the probability we need to reorder within two weeks? If the model's conditional and marginal probabilities are inconsistent, the decision logic built on top of them is unreliable in ways that standard backtesting will not surface.

What Engineering Teams Can Do About It

Audit Before You Deploy

The Dutch book test is implementable as an offline evaluation step. For a defined set of events relevant to your domain, elicit probability estimates from the model across marginal, conditional, and joint formulations. Then solve the linear program to check whether a guaranteed-profit bet exists. This requires no outcome labels and can be run before a system goes live.

The practical constraint is defining the event set carefully. The test is only as informative as the logical relationships you choose to probe. Teams should focus on the event combinations that actually appear in their decision logic, not on synthetic benchmarks.

Constrain the Context Window Deliberately

Given that irrelevant context is a primary driver of incoherence, the prompt engineering discipline that matters most is not chain-of-thought formatting or persona framing. It is aggressive context pruning. Every piece of information passed to the model that is not directly relevant to the forecast creates additional logical surface area for inconsistency.

This runs against the instinct to give models as much information as possible. The evidence suggests that instinct is wrong for probabilistic tasks. Structured, minimal prompts with explicit event definitions outperform rich narrative prompts on coherence grounds, even if they feel less intuitive to build.

Use Post-Processing Consistency Checks

Training strategies that improve coherence are discussed in the literature, including reinforcement learning approaches that penalise Dutch book violations during training (Andrews et al., arXiv 2026). These are not options available to teams using closed API models, but they are relevant for organisations fine-tuning open-weight models on domain-specific data.

For teams without access to training pipelines, the practical alternative is post-processing. Probabilities returned by the model can be projected onto the nearest coherent distribution using constrained optimisation before they are passed downstream. This does not fix the underlying model, but it prevents incoherent outputs from propagating into decision logic.