LLM Probability Outputs Are Less Reliable Than You Think: What Enterprise Teams Must Know Before Deploying Opinion or Scoring Systems

When an LLM reports a probability, the number looks precise. It carries decimal places. It slots neatly into a scoring pipeline or a multi-agent simulation. The problem is that the number may not obey the basic rules of probability, and that failure is not an edge case confined to poorly prompted models or unusual inputs. Recent research shows it appears systematically, in small widely-deployed models, under controlled conditions, at a scale that should concern any engineering team treating LLM-generated probabilities as production-grade signals.

Companion piece to our broader work on LLM reliability in production. See LLM Prediction Instability: Beyond Aggregate Metrics for why aggregate accuracy metrics conceal the per-example instability that matters most in deployed systems.

The Complement Inconsistency Problem

The specific failure mode is called complement inconsistency. If a model assigns probability p to the claim "X is true," it should assign probability 1-p to the claim "X is false." These two numbers must sum to one, because together they exhaust the possibility space.

Testing this property on Qwen3-4B, Llama-3.2-3B, and Gemma-3-4B across 2,304 agent states drawn from a running network simulation, Qwen's paired reports summed to 1.18 on average rather than 1.0 (Imel and Qi, arXiv 2026). The same failure appeared on 48 real fact-checking claims from the AVeriTeC dataset, and it persisted under repeated sampling and across different peer group sizes. This is not noise. It is a structural property of how these models assign probabilities to complementary statements.

The practical meaning is that the model is not reporting a position on an underlying state of the world. It is reporting a response to a specific sentence. Reframe the sentence and you get a different response, and the two responses are not constrained to be coherent with each other.

Why This Matters for Scoring and Classification Pipelines

Many enterprise scoring systems implicitly assume complement consistency without testing for it. A credit risk model that asks "how likely is this applicant to default" and a fraud detection system that asks "how likely is this transaction to be legitimate" are both eliciting probability-like outputs from LLMs, often as part of a larger feature pipeline.

If the model's answer to the positive framing and the negative framing of the same question do not sum to one, the scores are sensitive to wording choices that should be irrelevant. A prompt engineering decision made during development can shift scores in ways that are invisible to any evaluation that only tests one framing.

The discrepancy found by Imel and Qi (arXiv 2026) was two to ten times larger than the variation produced by a same-meaning paraphrase of the same claim. That means wording direction, not just wording variation, is a significant source of score instability, and it is one that aggregate accuracy metrics will not surface.

The Compounding Problem in Multi-Agent Simulations

The failure becomes harder to contain when LLM probability outputs feed back into subsequent model calls, as they do in multi-agent opinion simulation architectures. When each agent's reported probability becomes part of the input to neighboring agents in the next round, inconsistency in individual reports propagates and compounds across rounds.

Imel and Qi (arXiv 2026) tested this directly by initialising two populations from complementary starting states and feeding model outputs back through five simulation rounds. The populations did not remain mirror images of each other. Their divergence was positive after five rounds for every model tested. A simulation that should have produced symmetric outcomes instead drifted, purely because of complement inconsistency in the local update step.

For teams building multi-agent systems for market research, policy simulation, or opinion aggregation, this means the simulation's collective output cannot be interpreted as a stable representation of any underlying distribution. The trajectory of the simulation is partially an artifact of prompt framing.

Architectural Safeguards Before Production Deployment

The research tested one mitigation: averaging each probability report with the complement of its paired answer, which restores consistency by construction. This approach lowered Brier loss averaged across both wordings, though it only outperformed the better single wording for one of the three models tested (Imel and Qi, arXiv 2026). It is a useful correction but not a complete solution.

At the architecture layer, engineering teams should enforce the following before deploying any LLM-powered scoring or simulation system:

• Dual-framing consistency tests as part of the evaluation suite, not just accuracy on a single prompt direction
• Explicit consistency constraints applied post-hoc, such as averaging paired estimates or normalising outputs to sum to one before downstream use
• Prompt versioning controls that treat wording direction as a variable requiring its own sensitivity analysis
• Monitoring in production that tracks score distributions across semantically equivalent but differently framed queries, not just aggregate accuracy against labels

None of these safeguards are difficult to implement. The risk is that teams do not know they are needed until a downstream decision surfaces an anomaly that is hard to trace back to prompt framing.

What This Means for Engineering Governance

The deeper issue is that LLM probability outputs are not probabilities in the mathematical sense. They are responses to text, and they inherit the model's sensitivity to surface-level linguistic variation. Treating them as calibrated probability estimates without testing for basic consistency properties is an architectural assumption that has not been validated.

This does not mean LLM-generated scores have no production value. It means they require the same consistency and calibration checks applied to any probabilistic model before deployment, and those checks need to be designed with the specific failure modes of language models in mind. Complement inconsistency is one such failure mode. It is testable, and testing for it should be a standard part of any LLM scoring system's evaluation protocol.

Engineering leaders who build these checks into their development and monitoring pipelines are not adding overhead. They are closing a gap between what their system appears to do and what it actually does.