Summary (Overview)

• Central finding: Many dedicated memory systems underperform naive RAG (Retrieval-Augmented Generation) across multiple benchmarks and model backbones, indicating that building memory can have a negative net value over simply retrieving from raw history.
• Key contribution: The paper introduces the concepts of memory benefit (gains from memory processing over direct retrieval) and memory tax (losses introduced by memory processing), dissecting the net value of memory into these two opposing forces.
• Sources of memory tax: The tax arises from three system-side dimensions—faithfulness (information loss during construction), accessibility (fragmentation, dilution, or rephrasing that makes preserved evidence harder to retrieve), and robustness (amplified sensitivity to backbone changes).
• Generalization: The benefit-tax framework extends beyond personal-agent memory to experiential memory (agent task traces) and world-model memory (video consistency), showing the concept is memory-form-agnostic.
• Design implications: Simple interventions—source-augmented retrieval and recursive memory—can consistently reduce memory tax and improve accuracy without introducing new architectures.

---

Introduction and Theoretical Foundation

Long-horizon LLM agents increasingly rely on memory systems to maintain information across interactions. However, the field lacks a clear understanding of whether building structured memory (e.g., summaries, facts, graphs) provides value over a straightforward baseline like naive RAG. The authors identify a critical gap: most memory designs compare against other memories rather than against direct retrieval from raw history, leaving the net value of memory underexamined.

The paper formalizes memory as follows. Let the interaction history be $\mathcal{H} = \{h_1, \dots, h_T\}$. A memory system ingests this into persistent memory $\mathcal{M} = I(\mathcal{H})$, retrieves relevant units $\mathcal{C} = R(q, \mathcal{M})$, and generates an answer $\hat{y} = G(q, \mathcal{C})$. Naive chunk-based RAG, which keeps raw history with minimal processing and retrieves chunks directly, provides the natural reference.

Theoretical framing: Memory is viewed as a double-edged process—an inevitable deviation from raw history. These deviations are intended to benefit tasks (e.g., reorganizing scattered facts, distilling preferences) but can also harm by losing or obscuring evidence. The balance between these outcomes defines the net value of memory.

---

Methodology

Benchmarks

• LongMemEval-s (direct recall + complex queries), PersonaMem-v2 (20% stratified subset), MINT-Eval (10% stratified subset)
• Additional benchmarks for retrieval baseline analysis: AlpacaEval, MT-Bench, etc.

Memory Systems Evaluated (grouped by organization)

Implementation

• Backbones: Qwen3.5-4B, Qwen3.6-27B, GPT-5.6-luna (low); shared embedding model Qwen3-Embedding-0.6B
• Scoring: LLM-as-Judge backed by Qwen3.6-27B
• Fairness: Retrieval Scope (RS) defined as tokens exposed to the LLM normalized by total retrievable store tokens; top-k tuned per system to roughly match RAG(1-round)'s retrieval budget

Tax Analysis Dimensions

1. Faithfulness: Replay QA with gold raw turns vs. gold memory units; annotate atomic fact loss (no loss, partial, major, no unit)
2. Accessibility: Iterative retrieval rounds to recover gold units; query-evidence similarity comparisons
3. Robustness: Cross-backbone construction and evaluation (3×3 matrix)

---

Empirical Validation / Results

Main Results (Table 1 highlights)

RAG (2rd) proves a strong baseline; most memory systems fail to beat it consistently:

Key observations:

• OPENVIKING suffers the largest deficit (up to −40.4 points on LongMemEval-s)
• Memory systems incur substantially higher token costs (see Figure 1 Pareto plot)
• RAG's strong performance persists across all three backbones

Memory Tax Sources

Faithfulness (Table 2, fact-level annotation):

• OPENVIKING loses 13.0% of facts with major loss and 38.0% have no memory unit
• SIMPLEMEM shows 27.6% partial loss
• Even well-preserving systems (MEMORYOS: 98.4% no loss) show accessibility issues

Accessibility (Figure 4–5):

• Memory systems recover fewer queries in the first retrieval round than RAG
• Evidence gets fragmented across units, diluted with unrelated info, or rephrased, weakening query alignment

Robustness (Figure 6):

• With construction fixed, changing QA backbone causes larger performance variation than RAG for many systems

Generalization Results

Experiential memory (Figure 8): Web-based tasks benefit from memory (workflow distillation beats RAG), but tool-use scenarios show memory falling behind.

World-model memory (Table 3): Direct frame retrieval (I3DM: SSIM .560) outperforms latent/KV-based memories (WORLDKV: .372, FRAMEPACK: .217) for long-horizon consistency.

Design Implications (Table 4)

Source-augmented retrieval (adding raw-history retrieval alongside memory retrieval) consistently reduces tax and improves accuracy.

---

Theoretical and Practical Implications

Theoretical: The paper reframes memory from a purely beneficial augmentation to a double-edged deviation from raw history. This shifts the research question from "how to build better memory" to "when and how does memory add value over retrieval?"

Practical:

1. Evaluation practice: Memory papers should report naive RAG baselines to make the net contribution explicit; benchmarks should calibrate difficulty to leave headroom
2. Benchmark design: Many current benchmarks are retrieval-friendly (IDF-keyword achieves >50% absolute score); future benchmarks should include harder settings where abstraction matters
3. Memory architecture: The notion of recoverability—keeping a lightweight path back to source history—is a principled way to reduce tax
4. Cost-awareness: Memory systems incur substantially higher token costs while often underperforming RAG; cost-performance trade-offs must be considered

---

Conclusion

The paper demonstrates that many memory systems deliver negative net value compared to naive RAG, and dissects why: memory tax arises from information loss, reduced accessibility, and amplified backbone sensitivity. The benefit-tax framework generalizes across memory forms (conversational, experiential, world-model). The authors argue for a shift in focus from standalone memory performance to net value and its decomposition, and propose simple, architecture-agnostic interventions (source augmentation, recursive memory) that mitigate tax. Future directions include just-in-time memory, modeling future demand as a recommendation problem, and extending analysis to omni-modal, latent, and parametric memory forms.

Key takeaway: "Building memory has a dual nature and we call the two sides Memory Benefit and Memory Tax, with their balance defining net value of memory."