RAG Is Not the Only Way: When Compacting Conversation Memory Beats Transcript Retrieval in Production

Enterprise teams building conversational AI agents tend to reach for retrieval-augmented generation as their default memory strategy. The logic is intuitive: keep the transcript, index it, retrieve the relevant chunks at query time. What rarely gets evaluated is the alternative: compress each conversational turn into atomic facts and search those instead. Recent empirical work makes a direct comparison possible, and the results are specific enough to inform architectural decisions rather than just frame the debate.

What the Evidence Actually Shows

The comparison that matters here comes from a paired evaluation run against the LoCoMo benchmark, a long-context conversational memory dataset with 1,985 questions across multiple categories. Both a RAG system and a compaction-based memory system answered every question against the same stored content, using the same generator and judge, which makes the comparison unusually clean (Edogun et al., arXiv 2026).

On the four question categories that comparable prior work reports, the compaction approach scored 71.4% against RAG's 64.6%. That 6.8-point lead is statistically significant at p = 5.5e-08. It also came at 34% fewer prompt tokens, which at production volumes translates directly into inference cost.

The mechanism is straightforward. Atomic facts are shorter than the turns that produced them. Retrieval over a compact fact store returns denser signal per token than retrieval over raw transcript chunks. The model receives less noise and more relevant content per context window.

Where Compaction Loses Ground

The aggregate lead disappears when the adversarial question category is included. RAG outperforms compaction on adversarial queries by 14.6 points, and that reversal is large enough to reduce the overall five-category lead to 2.0 points, which is no longer statistically significant (Edogun et al., arXiv 2026).

The structure of that failure is worth understanding precisely. The performance gap correlates with how much of the reference answer depends on source wording. When the correct answer requires only one or two words from the original text, compaction leads by 9.6 points. When the answer requires more than eight words from the original phrasing, compaction trails by 3.1 points. Token overlap between prediction and reference moves in the same direction, which rules out the judge as a confounding factor.

Compaction preserves what a fact asserts and discards how it was said. That is the right trade-off when the answer is a value, a date, or a name. It is the wrong trade-off when the answer is a specific formulation from the source material.

The Deployment Contexts Where This Distinction Is Consequential

For most enterprise conversational use cases, the compaction approach holds up well. A customer support agent recalling a user's stated preferences, a sales assistant tracking account details across sessions, or an internal knowledge agent surfacing policy facts all operate in the regime where what was said matters more than how it was said.

The failure mode becomes material in three specific contexts. Compliance and audit workflows often require verbatim reproduction of commitments, disclosures, or policy language. Legal and contract review agents may need to surface exact clauses rather than paraphrased assertions. Support escalation systems that reconstruct what a customer was told in a prior session face the same constraint.

In these contexts, RAG's transcript-preservation approach is not a legacy default. It is the architecturally correct choice, because the retrieval target is phrasing, not fact.

Token Economics at Scale

The 34% token reduction from compaction is not a marginal efficiency. At the prompt lengths typical of long-session conversational agents, that reduction compounds across every query in every session. For a deployment running tens of thousands of daily conversations, the difference in monthly inference spend can be significant enough to affect the business case for the product.

The caveat is that compaction introduces an extraction step at write time. An LLM or structured extractor must process each turn and produce atomic assertions. That cost is paid once per turn rather than per query, so it amortises well under high query-to-session ratios. If a session generates ten turns but receives fifty queries against its memory, the extraction overhead is a small fraction of the savings.

Choosing the Right Architecture for Your Constraints

The decision between compaction and RAG for conversational memory is not a general preference question. It is a function of what your retrieval targets look like.

If your agents are answering factual questions about prior conversation content, and accuracy on those facts is the primary metric, compaction is likely to outperform RAG while reducing cost. If your agents operate in domains where exact phrasing from prior exchanges has legal, regulatory, or evidentiary weight, RAG preserves what compaction discards.

A hybrid approach is also viable: maintain a compacted fact store for general recall and a parallel transcript index for phrasing-sensitive queries. The added complexity is real, but so is the performance gap in each direction. The architecture should follow the query distribution, not the other way around.