Why Your LLM Cannot Find the Answer That Is Already in the Document

Enterprise teams building document reasoning pipelines have spent the last two years chasing a deceptively simple hypothesis: if the model can see more of the document, it will reason better over it. The result has been a steady migration toward larger and larger context windows, often framed as an infrastructure upgrade. The problem is that context access and context utilisation are not the same capability, and conflating them is producing retrieval pipelines that fail quietly, at exactly the moments that matter most.

The Real Failure Mode Is Not Window Size

When a production RAG system returns a wrong or incomplete answer, the instinct is to ask whether the relevant passage was inside the context window. Usually it was. The failure is not one of coverage. It is one of attention: the model technically has access to the evidence but fails to weight it correctly during generation.

This distinction matters because it changes the engineering response entirely. Buying a 200K-token context window when you had a 128K one does not fix an attention problem. It expands the surface area over which that attention problem can occur.

The mechanism behind this is well-characterised. As context length grows, the model's ability to maintain strong associative links between a query and distant, relevant tokens degrades. The signal from the question does not propagate evenly across the full context. It attenuates.

What Associative Memory Theory Tells Us

A useful frame for understanding this failure comes from treating the context window as an associative memory store. The question acts as a retrieval cue. Attention functions as the cue-to-trace association mechanism. And the model's final answer is a product of which traces get reactivated strongly enough to influence generation.

Under this model, the problem with long contexts is not that relevant evidence is absent. It is that the retrieval cue, the question, does not reliably activate the right memory traces when they are buried deep in a large context. The associative signal is too diffuse to overcome the positional and attentional biases that accumulate at scale.

Research from the University of Illinois Urbana-Champaign supports this view quantitatively. Zhao et al. (arXiv, 2026) found that the top 0.1% of context tokens by relevance score already accounts for approximately 50 to 80 percent of accumulated relevance signal across three different LLM architectures. In a 128K-token context, that corresponds to roughly 128 tokens. The evidence is concentrated. The model's problem is surfacing it reliably.

Why Inference-Time Evidence Organisation Changes the Calculus

If the failure is attentional rather than architectural, then the solution does not require retraining the model or replacing the retrieval system. It requires a different approach to how evidence is organised at inference time, before generation begins.

This is the logic behind recursive evidence replay as an inference pattern. Rather than passing the full context and expecting the model to identify and weight relevant passages independently, the approach uses the model's own internal relevance signals to construct a query-conditioned evidence pool. That pool is then replayed into the prompt before final generation, alongside the original context.

The critical design choice here is that the original context is preserved. This is not context pruning, which discards potentially relevant material and introduces its own failure modes. It is evidence amplification: the model sees the full document and a structured reactivation of the passages most likely to support the answer.

Zhao et al. (arXiv, 2026) demonstrate that this approach consistently improves evidence utilisation across multiple long-context benchmarks at 128K context length, without requiring external memory systems, vector databases, or any modification to model weights.

What This Means for Engineering Teams

The Procurement Trap

The practical implication for teams evaluating context window upgrades is that the evaluation criteria need to change. Measuring whether a model can technically fit a document into its context is a necessary condition but not a sufficient one. The relevant benchmark is whether the model can reliably locate and weight evidence within that context under realistic query distributions.

Teams that skip this distinction will find themselves in a cycle of capability upgrades that do not translate into retrieval quality improvements. The model gets a larger window. The failure rate on long-document queries does not fall proportionally. The diagnosis remains elusive because the symptom looks like a retrieval problem but the cause is an attention problem.

Inference Patterns as a First Response

Before committing to architectural changes, engineering teams should evaluate inference-time evidence organisation as a first intervention. The approach requires no retraining, no additional model deployment, and no changes to the retrieval layer. It operates as a harness around the existing model, restructuring how evidence is presented at generation time.

This matters commercially because it compresses the iteration cycle significantly. A team can test whether recursive evidence replay closes the gap on their specific document types and query patterns before making any infrastructure commitments. If it does, the fix is a prompt engineering and orchestration problem. If it does not, the team has ruled out the attentional failure mode and can pursue deeper architectural changes with better information.

Integration Considerations

Implementing this pattern in a production pipeline requires a few concrete decisions. The model must be queried twice per generation: once to score token relevance against the query, and once to generate the final answer with the evidence pool prepended. This adds latency and token cost, both of which need to be benchmarked against the retrieval quality improvement on the specific use case.

The evidence pool construction also requires decisions about how many tokens to include and how to handle cases where the relevance signal is flat across the context. Teams building document reasoning pipelines for regulated industries, where answer provenance matters as much as answer accuracy, will need to ensure that the evidence pool selection is auditable and that the replay mechanism does not introduce hallucinated context.

The Architecture Decision That Follows

If inference-time evidence organisation closes most of the gap, the remaining design question is where to run the relevance scoring step. Running it inside the same model that generates the answer is the simplest path and the one validated by current research. Running it with a smaller, cheaper model introduces a scoring quality risk but reduces cost at scale.

For teams operating at high document throughput, the cost profile of double-pass inference may eventually push toward hybrid architectures where a lightweight scorer handles evidence pool construction and a larger model handles generation. That is a reasonable direction, but it is an optimisation problem, not a fundamental redesign. The core pattern, separating evidence organisation from answer generation, remains stable across both configurations.

The broader lesson is that long-context reasoning quality is an inference engineering problem as much as it is a model capability problem. Teams that treat it purely as the latter will consistently underperform teams that invest in how evidence is structured and presented at generation time.