Why Multi-Hop Questions Break Enterprise RAG and What Your Architecture Needs to Fix That

Enterprise knowledge retrieval systems tend to perform well on the questions engineers use to evaluate them and quietly fail on the questions knowledge workers actually ask. The gap is not a tuning problem. It is a structural one. Single-pass retrieval cannot recover evidence chains that span multiple documents, and the iterative approaches most teams reach for next introduce a different set of failure modes that are just as damaging at production scale. Understanding why both break down is the prerequisite for designing a retrieval stack that does not.

Why Single-Pass Retrieval Cannot Handle Chained Reasoning

A standard RAG pipeline retrieves documents by measuring semantic similarity between the query and a corpus. That works when the answer lives in a single passage. It fails when the answer requires connecting an entity named in document A to a fact about that entity in document B, where document B is never surfaced because the query never mentions the entity that links them.

This is the core multi-hop problem. The query does not contain the bridge entities, so the retriever never finds the passages that contain them. The model then either hallucinates a connection or returns an incomplete answer, and neither outcome is visible in a standard retrieval accuracy metric.

Graph-based RAG methods partially address this by indexing entity relationships, but they still retrieve from the entities the question explicitly names. If the chain passes through an intermediate entity the question does not mention, the graph traversal stops before it reaches the relevant evidence (Kang and Li, HuggingFace 2026).

The Failure Modes of Iterative RAG

Iterative retrieval approaches attempt to solve multi-hop reasoning by decomposing a question into sub-questions and retrieving evidence sequentially. The architecture is sound in principle. The failure is in how much cognitive work gets loaded onto the LLM inside each iteration.

Planning sub-questions in advance creates dependency errors. If the model schedules a sub-question about a location before it has retrieved the entity whose location it needs, the sub-question is malformed and the retrieval fails. Approaches that plan the full decomposition upfront cannot adapt when earlier retrievals return unexpected results (Kang and Li, HuggingFace 2026).

Approaches that keep all retrieved passages in a growing prompt solve the dependency problem but introduce a different one. As the context window fills, the model loses track of intermediate conclusions it stated earlier. The evidence is present, but the model fails to carry it forward into the final answer. This is a context utilisation failure, not a retrieval failure, and it is not fixed by increasing context length.

What Self-Contained Sub-Question Generation Changes

The structural fix that DeFT-RAG demonstrates is generating one sub-question at a time, where each sub-question is fully self-contained: it names every entity it asks about, constructed from the original question and the QA pairs already accumulated (Kang and Li, HuggingFace 2026). This removes the need for the LLM to maintain a planning state across iterations.

When each sub-question is self-contained, dependency management becomes a rule-based table update rather than an LLM reasoning task. The model handles one fine-grained retrieval and answer task per step. Everything else, tracking what is known, deciding whether to continue, is offloaded to lightweight deterministic components.

The commercial implication is significant. Reducing the cognitive load on the LLM per iteration means fewer tokens consumed per question and fewer opportunities for the model to compound errors across steps. At scale, that translates directly into cost and latency reduction without sacrificing answer quality.

The Decision Model Reranker as a Missing Architectural Layer

Most enterprise RAG stacks treat reranking as an optional quality improvement. The emerging evidence suggests it should be treated as a core component, and the choice of reranker architecture has latency consequences that matter in production.

Generative rerankers produce high-quality rankings by generating ranked sequences of document IDs. The problem is autoregressive decoding latency. Each ranking decision requires a full forward pass through the generation process, which is expensive when applied to a top-100 candidate set on every query.

Decision model rerankers avoid this by predicting a score from the first-token position rather than generating a sequence. DecisionRank demonstrates that a 1.7B decision model trained via on-policy distillation from a 27B generative teacher can outperform that teacher on standard retrieval benchmarks, while running more than 24 times faster than its generative counterpart (Wu et al., HuggingFace 2026). The distillation process bridges the representation gap by constructing the student's training signal from the teacher's conditional feedback on the student's own current rankings, which keeps the training distribution aligned with inference conditions.

Architectural Choices That Determine Production Accuracy

The patterns above point toward a retrieval stack with four distinct layers: a query decomposition component that generates self-contained sub-questions one at a time, a retriever that operates on those sub-questions, a decision model reranker that scores candidates without autoregressive overhead, and a QA cache that stores sub-question answers for reuse across queries.

The cache layer is worth treating seriously. When sub-questions are self-contained and consistently formed, semantically equivalent sub-questions from different users will surface the same cached answer. DeFT-RAG reports that with a shared cache across 1,000 questions, 23.1% of sub-questions were answered from cache, reducing LLM calls by 12.9% without accuracy loss (Kang and Li, HuggingFace 2026). At enterprise query volumes, that reduction compounds.

The decision about where to place the decision model in the pipeline also matters. It should sit between the initial retriever and the LLM, operating on the candidate set before context is assembled. Placing it downstream, after context has already been passed to the model, loses most of the latency benefit and does nothing to improve the quality of evidence the model reasons over.