AI Strategy , Data science & AI Jul 15, 2026

Your Model Looks Stable in Aggregate. That Is Exactly the Problem.

VECTOR Labs Team

Last updated on: Jul 15, 2026

Enterprise teams deploying large language models in production have developed a false sense of security around aggregate accuracy. A model clears a benchmark threshold, the numbers look flat across evaluation runs, and the deployment decision gets signed off. The problem is that aggregate accuracy is structurally incapable of revealing what matters most in high-stakes production environments: whether any individual prediction is reliable. Recent research makes this concrete in a way that should change how CTOs think about model governance.

The Two-Sided Instability Problem

When researchers prepended task-irrelevant context to benchmark questions, aggregate accuracy barely moved. For GPT-4.5, the shift was no more than ±0.9%, and for GPT-4.4, no more than ±2.1%. Those numbers look reassuring at the model level. They are not reassuring at the example level (Zhang et al., arXiv 2026).

The underlying mechanism is two-sided. Adding irrelevant context degrades performance on some questions while simultaneously improving it on others. The gains and losses cancel out in aggregate, which is exactly why the headline number stays flat. The model is not stable. It is oscillating in ways that the summary statistic erases.

What makes this particularly difficult to manage is that the affected examples are largely model-specific. A question that destabilises GPT-4.5 may not destabilise Claude or Gemini, and vice versa. Switching models does not solve the problem. It changes which tail you are exposed to.

What Context-Sensitivity Failure Actually Looks Like in Production

The research used pseudo-words formed by randomly combining characters as the irrelevant context. These inputs carry no semantic content whatsoever. They are noise by construction. The fact that even this kind of input can flip individual predictions is significant because it sets a lower bound on the problem (Zhang et al., arXiv 2026).

In real production environments, models operate with far richer and more structured surrounding context. A document review system prepends retrieved clauses. A customer support agent carries conversation history. A financial analysis tool includes retrieved filings alongside the query. Every one of these contexts is more semantically loaded than random character strings, which means the surface area for per-example instability is larger than the controlled research setting implies.

The instability is also modulated by context length, context type, and model development stage. Longer context windows do not automatically reduce the problem. In some configurations they amplify it. This matters for teams who have assumed that extended context support in newer models resolves sensitivity issues.

Why Aggregate Metrics Cannot Catch This

Aggregate accuracy is a population-level statistic. It describes average behaviour across a distribution of inputs. Per-example instability is a tail phenomenon. It describes the behaviour of specific inputs under specific conditions. These are different objects, and no amount of refinement to aggregate evaluation methodology closes the gap between them.

The commercial implication is direct. If a model is used to assess credit risk, triage medical records, flag compliance violations, or generate contractual summaries, the question that matters is not whether the model is right 94% of the time. The question is whether the model is reliable for the specific input in front of it right now. Aggregate accuracy gives no answer to that question.

Teams that benchmark thoroughly but evaluate only at the aggregate level are, in effect, auditing the average while leaving the tail unexamined. That is a governance gap, not just a technical gap.

Building Per-Example Reliability Evaluation

Per-example reliability evaluation means assessing whether a model produces consistent outputs for a given input across variations in surrounding context. This is distinct from adversarial testing. The goal is not to find inputs that break the model through deliberate manipulation. The goal is to characterise how much a model's prediction on any given example depends on context that should be irrelevant to the answer.

A practical implementation has three components. First, a perturbation protocol: for each evaluation example, generate multiple context-augmented variants using content that is semantically orthogonal to the task. Second, a stability metric: measure the prediction consistency across those variants at the example level, not the dataset level. Third, a flagging threshold: identify examples where consistency falls below an acceptable bound and treat those as unreliable for deployment purposes.

This approach produces a reliability profile for the model rather than a single accuracy number. It answers a different and more useful question: for which inputs can we trust this model's output, and for which inputs does the prediction depend on factors we cannot control?

Governance Implications for Enterprise Teams

The deployment gate needs to change. Aggregate accuracy as a pass/fail threshold was a reasonable starting point when models were simpler and deployment contexts were narrower. It is no longer sufficient for production LLMs operating in context-rich environments. Per-example reliability evaluation should be a required component of the model governance stack, not an optional addition.

This has implications for how teams structure evaluation pipelines. Evaluation sets need to be large enough and diverse enough to surface tail behaviour. Stability metrics need to be logged alongside accuracy metrics in model cards and deployment documentation. For high-stakes applications, examples that fall below reliability thresholds should either be routed to human review or excluded from automated decision pathways entirely.

It also has implications for model selection. Because instability is model-specific, the reliability profile of a model on your specific task distribution is a first-order selection criterion. We covered the broader mechanics of evaluating model releases before committing to them in Beyond Benchmarks: How CTOs Should Actually Evaluate New Model Releases Before Committing to Them, which addresses the gap between published benchmark performance and production-relevant evaluation. The per-example reliability question is a natural extension of that framework.

The aggregate accuracy number will not warn you before a tail-risk failure becomes a production incident. The governance infrastructure that catches it has to be built before deployment, not assembled in response to an incident after the fact.

Where Vector Labs Fits

We build production AI systems for enterprise clients where individual-prediction reliability is a regulatory and operational requirement, not an afterthought. Our work on the Probability of Default Prediction Model for a Retail Bank demonstrates how we approach per-example reliability in high-stakes financial decision systems, including out-of-sample validation and individual-level risk scoring integrated into lending and risk management workflows. If you are building model governance infrastructure that needs to go beyond aggregate benchmarks, contact us to discuss your evaluation requirements.

FAQs

What is per-example reliability evaluation and how does it differ from standard benchmarking?

Standard benchmarking measures average model performance across a dataset and reports a single aggregate accuracy figure. Per-example reliability evaluation measures whether a model produces consistent predictions for a specific input when the surrounding context is varied in ways that should not affect the answer. The distinction matters because a model can score well on aggregate benchmarks while being highly unstable on individual examples that are critical to your use case.

How significant is the instability problem in practice? Is it limited to edge cases?

Research shows that even semantically meaningless pseudo-words, with no task-relevant content whatsoever, can shift individual model predictions in ways that aggregate scores completely conceal (Zhang et al., arXiv 2026). The affected examples represent a minority of the evaluation set, but in high-stakes applications, a small fraction of unreliable predictions can have disproportionate consequences. The problem is not theoretical. It is a structural property of how current LLMs process context.

Does switching to a newer or more capable model resolve per-example instability?

Not reliably. The research finding that instability is largely model-specific means that the examples affected by context sensitivity differ across models, but the phenomenon persists across all evaluated architectures. A newer model may be more stable on the examples that destabilised an older model, while introducing new instability on a different subset of inputs. Model selection should therefore include per-example reliability profiling on your specific task distribution, not just aggregate benchmark comparison.

What does a practical per-example reliability evaluation pipeline look like?

The core components are a perturbation protocol, a stability metric, and a flagging threshold. For each evaluation example, you generate multiple variants by prepending or appending context that is orthogonal to the task. You then measure prediction consistency across those variants at the individual example level. Examples where consistency falls below a defined threshold are flagged as unreliable and handled separately, either through human review or exclusion from automated decision pathways.

How should per-example reliability findings be incorporated into model governance documentation?

Reliability profiles should be included in model cards alongside standard accuracy metrics, with explicit documentation of which input types and context configurations produce the highest instability. Deployment approvals for high-stakes applications should require a reliability threshold to be met, not just an accuracy threshold. For regulated industries, the reliability profile also provides an audit trail demonstrating that tail-risk behaviour was assessed before deployment, which is increasingly relevant to emerging AI governance frameworks.

Is this problem specific to question-answering tasks, or does it apply to other LLM use cases?

The controlled research used QA benchmarks to isolate the effect cleanly, but the underlying mechanism applies wherever a model processes task-relevant input embedded within broader context. Document review, contract analysis, compliance flagging, customer support triage, and code generation in agentic workflows all involve exactly this structure. Any production setting where the model receives more context than the core task requires is a setting where per-example instability is a relevant risk to evaluate.

A team that understands you

With 20+ years of experience in the world's leading consultancy companies, implementing AI and ML projects in industry-specific contexts, we are ready to hear your challenges.

Talk with an AI expert