# Memory Tax: Dissecting the Net Value of Agent Memory

> Most dedicated memory systems deliver negative net value over naive RAG due to memory tax from information loss, reduced accessibility, and backbone sensitivity.

- **Source:** [arXiv](https://arxiv.org/abs/2610.34073)
- **Published:** 2026-10-29
- **Permalink:** https://picx.dev/p/iJSwaR
- **Whiteboard:** https://picx.dev/p/iJSwaR/image

## Summary

## Summary (Overview)

- **Central finding**: Many dedicated memory systems underperform naive RAG (Retrieval-Augmented Generation) across multiple benchmarks and model backbones, indicating that building memory can have a **negative net value** over simply retrieving from raw history.
- **Key contribution**: The paper introduces the concepts of **memory benefit** (gains from memory processing over direct retrieval) and **memory tax** (losses introduced by memory processing), dissecting the net value of memory into these two opposing forces.
- **Sources of memory tax**: The tax arises from three system-side dimensions—**faithfulness** (information loss during construction), **accessibility** (fragmentation, dilution, or rephrasing that makes preserved evidence harder to retrieve), and **robustness** (amplified sensitivity to backbone changes).
- **Generalization**: The benefit-tax framework extends beyond personal-agent memory to **experiential memory** (agent task traces) and **world-model memory** (video consistency), showing the concept is memory-form-agnostic.
- **Design implications**: Simple interventions—source-augmented retrieval and recursive memory—can consistently reduce memory tax and improve accuracy without introducing new architectures.

---

## Introduction and Theoretical Foundation

Long-horizon LLM agents increasingly rely on memory systems to maintain information across interactions. However, the field lacks a clear understanding of whether building structured memory (e.g., summaries, facts, graphs) provides value over a straightforward baseline like naive RAG. The authors identify a **critical gap**: most memory designs compare against other memories rather than against direct retrieval from raw history, leaving the *net value* of memory underexamined.

The paper formalizes memory as follows. Let the interaction history be $\mathcal{H} = \{h_1, \dots, h_T\}$. A memory system ingests this into persistent memory $\mathcal{M} = I(\mathcal{H})$, retrieves relevant units $\mathcal{C} = R(q, \mathcal{M})$, and generates an answer $\hat{y} = G(q, \mathcal{C})$. Naive chunk-based RAG, which keeps raw history with minimal processing and retrieves chunks directly, provides the natural reference.

**Theoretical framing**: Memory is viewed as a **double-edged process**—an inevitable deviation from raw history. These deviations are *intended* to benefit tasks (e.g., reorganizing scattered facts, distilling preferences) but can also *harm* by losing or obscuring evidence. The balance between these outcomes defines the net value of memory.

---

## Methodology

### Benchmarks
- **LongMemEval-s** (direct recall + complex queries), **PersonaMem-v2** (20% stratified subset), **MINT-Eval** (10% stratified subset)
- Additional benchmarks for retrieval baseline analysis: AlpacaEval, MT-Bench, etc.

### Memory Systems Evaluated (grouped by organization)
| Category | Systems |
|---|---|
| Flat store | A-MEM, SIMPLE-MEM, MEMOS |
| Hier/Lifecycle | MEMORYOS, LIGHTMEM, OPENVIKING |
| Graph-based | HIPPORAG2, PLUGMEM, MAGMA |
| Trained policy | MEM-T, MEMORY-R2 |

### Implementation
- **Backbones**: Qwen3.5-4B, Qwen3.6-27B, GPT-5.6-luna (low); shared embedding model Qwen3-Embedding-0.6B
- **Scoring**: LLM-as-Judge backed by Qwen3.6-27B
- **Fairness**: Retrieval Scope (RS) defined as tokens exposed to the LLM normalized by total retrievable store tokens; top-k tuned per system to roughly match RAG(1-round)'s retrieval budget

### Tax Analysis Dimensions
1. **Faithfulness**: Replay QA with gold raw turns vs. gold memory units; annotate atomic fact loss (no loss, partial, major, no unit)
2. **Accessibility**: Iterative retrieval rounds to recover gold units; query-evidence similarity comparisons
3. **Robustness**: Cross-backbone construction and evaluation (3×3 matrix)

---

## Empirical Validation / Results

### Main Results (Table 1 highlights)

RAG (2rd) proves a strong baseline; most memory systems fail to beat it consistently:

| Benchmark | RAG(2rd) best | Best memory system | Gap |
|---|---|---|---|
| LongMemEval-s (Q3.6-27B) | 81.4 | A-Mem 79.4 (−2.0) | −2.0 |
| PersonaMem-v2 (Q3.6-27B) | 48.7 | MemoryOS 50.2 (+1.5) | +1.5 |
| MINT-Eval (Q3.6-27B) | 33.1 | PlugMem 33.3 (+0.1) | +0.1 |

**Key observations**:
- OPENVIKING suffers the largest deficit (up to −40.4 points on LongMemEval-s)
- Memory systems incur substantially higher token costs (see Figure 1 Pareto plot)
- RAG's strong performance persists across all three backbones

### Memory Tax Sources

**Faithfulness** (Table 2, fact-level annotation):
- OPENVIKING loses 13.0% of facts with major loss and 38.0% have no memory unit
- SIMPLEMEM shows 27.6% partial loss
- Even well-preserving systems (MEMORYOS: 98.4% no loss) show accessibility issues

**Accessibility** (Figure 4–5):
- Memory systems recover fewer queries in the first retrieval round than RAG
- Evidence gets fragmented across units, diluted with unrelated info, or rephrased, weakening query alignment

**Robustness** (Figure 6):
- With construction fixed, changing QA backbone causes larger performance variation than RAG for many systems

### Generalization Results

**Experiential memory** (Figure 8): Web-based tasks benefit from memory (workflow distillation beats RAG), but tool-use scenarios show memory falling behind.

**World-model memory** (Table 3): Direct frame retrieval (I3DM: SSIM .560) outperforms latent/KV-based memories (WORLDKV: .372, FRAMEPACK: .217) for long-horizon consistency.

### Design Implications (Table 4)

| Method | LongMemEval Acc ↑ | Tax ↓ | PersonaMem Acc ↑ | Tax ↓ |
|---|---|---|---|---|
| SIMPLEMEM + src-aug | 73.6 → 82.8 | 12.0 → 5.2 | 46.7 → 49.6 | 11.2 → 7.2 |
| MEMORYOS + src-aug | 73.4 → 76.4 | 11.4 → 7.8 | 48.2 → 53.0 | 17.1 → 8.5 |
| MEM-T + src-aug | 75.0 → 80.0 | 10.8 → 5.2 | 42.1 → 49.0 | 15.2 → 8.7 |

Source-augmented retrieval (adding raw-history retrieval alongside memory retrieval) consistently reduces tax and improves accuracy.

---

## Theoretical and Practical Implications

**Theoretical**: The paper reframes memory from a purely beneficial augmentation to a **double-edged deviation** from raw history. This shifts the research question from "how to build better memory" to "when and how does memory add value over retrieval?"

**Practical**:
1. **Evaluation practice**: Memory papers should report naive RAG baselines to make the net contribution explicit; benchmarks should calibrate difficulty to leave headroom
2. **Benchmark design**: Many current benchmarks are retrieval-friendly (IDF-keyword achieves >50% absolute score); future benchmarks should include harder settings where abstraction matters
3. **Memory architecture**: The notion of **recoverability**—keeping a lightweight path back to source history—is a principled way to reduce tax
4. **Cost-awareness**: Memory systems incur substantially higher token costs while often underperforming RAG; cost-performance trade-offs must be considered

---

## Conclusion

The paper demonstrates that **many memory systems deliver negative net value** compared to naive RAG, and dissects why: memory tax arises from information loss, reduced accessibility, and amplified backbone sensitivity. The benefit-tax framework generalizes across memory forms (conversational, experiential, world-model). The authors argue for a shift in focus from standalone memory performance to **net value** and its decomposition, and propose simple, architecture-agnostic interventions (source augmentation, recursive memory) that mitigate tax. Future directions include just-in-time memory, modeling future demand as a recommendation problem, and extending analysis to omni-modal, latent, and parametric memory forms.

> **Key takeaway**: "Building memory has a dual nature and we call the two sides Memory Benefit and Memory Tax, with their balance defining net value of memory."

---

_Markdown view of https://picx.dev/p/iJSwaR, served by PicX — AI-generated visual whiteboard summaries of research papers._