Summary of "Delivered, Not Used: Evaluating Agent Memory at Its Point of Use"

Summary (Overview)

• Novel evaluation framework: The paper introduces MemUse, the first reference-free framework to evaluate individual memory deliveries during LLM agent execution, assessing whether delivered memory was actually used and what consequences followed—without requiring gold answers, reference trajectories, or controlled memory interventions.

• Key finding on outcome concealment: On 8,000 SWE-bench Verified episodes across three memory systems, resolve rates stayed within 0.4 percentage points of a no-memory baseline, yet MemUse revealed that most delivered entries were irrelevant (71.5–98.6%), most relevant entries went unused (51.3–79.7%), and task outcomes concealed both beneficial use in failed episodes (23.2%) and harmful use in successful ones (31.3%).

• Three-part evaluation model: Each memory delivery is assessed along three separate axes—information (relevance, accuracy, novelty), use (influence on actions), and consequence (observed effect)—using an agentic judge with programmatic evidence checks and fixed aggregation rules.

• Practical insights for memory design: Full cards, task recall, and full skills supplied most beneficial entries at small token shares, while previews and indexes mainly supported selection; repeated delivery contributed substantial token overhead (68.1% of Mem0's delivered tokens were restatements of still-visible content).

Introduction and Theoretical Foundation

Background and Motivation

LLM agents increasingly handle long-running tasks (repository-level software engineering, multi-step terminal workflows) that generate more information than a finite context window can hold. Memory systems (MemGPT, Mem0, MemGovern, TencentDB Agent Memory) preserve and organize information for repeated delivery, but delivery does not guarantee use or benefit. Delivered memory also consumes the same finite context whose growth can degrade performance.

Limitations of Existing Evaluations

Three Core Challenges

1. Identifying what the agent saw: The same entry may be delivered many times in different forms (full text, preview, index)
2. Distinguishing memory use from prior knowledge: The agent may already have the same information in visible context
3. Following effects over time: Memory may shape actions many steps later, with corrections or reversals

Methodology

Core Evaluation Unit

Each entry is evaluated as a memory delivery in context:

$$c_{d}(m) = \bigl(q, B_{d}, \epsilon_{d}(m), W_{d}(m), \tau \bigr), \tag{1}$$

where:

• $q$ = task
• $B_{d}$ = prior context (consuming input minus delivery text)
• $\epsilon_{d}(m)$ = exposed text of entry $m$
• $W_{d}(m)$ = credit window (steps whose actions can be credited to entry $m$)
• $\tau$ = full trajectory

Recording Infrastructure

• Memory bridge: Adapters for agent harnesses and memory systems annotate memory creation, updates, searches, and deliveries; placement proofs (digests or request paths) identify memory text locations
• Pass-through trajectory recorder: Captures model calls at the API boundary, normalizes events, verifies placement proofs, and tracks when context rewrites remove memory text

Injection Spans

Repeated identical deliveries form injection spans (judged once) when: (1) entries repeat exactly with same order/placements/exposure, and (2) an earlier copy is still visible in the consuming input. Later deliveries are "restatements" that add no verdicts but count toward token volume.

Credit Windows

The window $W_{S}(m)$ starts at the first consuming step of the span and ends when a later entry of the same item takes over credit or at episode end. A later copy that adds nothing to still-visible content receives an empty window. Effects of credited actions are followed through episode end, even after windows close.

Five Assessment Signals

Triage Labels

Fixed precedence rules derive span verdicts:

$$\ell(S) = \begin{cases} \text{negative}, & \text{if some } m \in M_{S} \text{ is misused}, \\ \text{positive}, & \text{else if some } m \in M_{S} \text{ is used}, \\ \text{unresolved}, & \text{else if some } m \in M_{S} \text{ is unclear}, \\ \text{neutral}, & \text{otherwise}. \end{cases} \tag{2}$$

Entry-Share Profiles

Six axes measure composition of spans:

$$p_{a}(S) = \frac{|A_{a}(S)|}{|M_{S}|}, \quad \bar{p}_{a} = \frac{1}{|\mathcal{P}|} \sum_{S \in \mathcal{P}} p_{a}(S). \tag{3}$$

Axes: relevance, corroboration, information gain, acted on, benefit, harm—each divided by all delivered entries for comparability.

Evidence Checks

• Structural checks: valid signal values, anchors within credit windows, full reads of cited steps
• Mandatory reads: task, prior-context index, first consuming step with preceding step
• Determined novelty/acted-on requires complete reads of all prior context sources
• Determined observed effect requires every step from window start through episode end

Empirical Validation / Results

Experimental Setup

• Tasks: All 500 tasks of SWE-bench Verified (12 Python repositories), ordered by pull-request creation order with interleaved repositories
• Agent: mini-SWE-agent with gpt-6-luna; judge: DeepSeek-V4.1-Flash-1017 (given gold patches but not harness outcomes)
• Configurations: MemGovern, Mem0 (L), Mem0 (L+G), TencentDB Agent Memory, plus no-memory baseline
• Scale: 8,000 memory episodes + 2,000 baseline episodes (4 runs × 500 tasks × 5 configurations)

Task Outcomes vs. Memory Profiles

Table 2: Task success and mean cost per episode

Key Findings

1. Systems separated by MemUse despite similar outcomes: MemGovern's relevance share (15.08%) was >10× Mem0's (1.42–1.43%); TencentDB's was highest (28.48%), but 18.0 points came from its always-relevant persona block

2. Two bottlenecks limit memory effectiveness:

   • 71.5–98.6% of entries were irrelevant on average
   • 51.3–79.7% of relevant entries remained unused (not acted on, selected, or referenced)
   • Agent acted on only 0.40–4.08% of entries; beneficial entries reached only 3.65–7.05% of episodes

3. Benefit dominates harm when use occurs: Of 714 acted-on entries: 421 beneficial, 16 harmful, 64 no effect, 213 undetermined (29.8%)

4. Outcomes conceal effects: 89/383 episodes with beneficial use failed (23.2%); 5/16 with harmful use passed (31.3%)

5. Benefit concentrates in small text shares: Full cards = 96.4% of MemGovern's beneficial entries but 12.5% of tokens; full skills = 86.0% of TencentDB's beneficial entries but 2.0% of tokens

6. Repeated delivery inflates cost: TencentDB restates 88.4% of deliveries, Mem0 78.2%; Mem0's visible copies take 68.1% of delivered tokens

7. Use often comes late: First credited action followed first consumption for 51.8% of acted-on entries

8. Memory histories reveal origins: 22.1% of beneficial entries traced to failed source episodes; two MemGovern cards produced 7 of 10 harmful entries

Theoretical and Practical Implications

Theoretical Contributions

• Attribution without intervention: MemUse demonstrates that observational trajectory analysis can attribute agent actions to specific memory deliveries, distinguishing information quality from use from consequence—a separation existing frameworks conflate

• Credit window concept: The notion that memory effects must be followed through corrections, reversals, and delayed actions provides a principled framework for temporal credit assignment in agent memory

• Null result diagnosis: The framework explains why memory systems often fail to beat baselines: not because memory is unhelpful, but because most delivered content is irrelevant or redundant, and most relevant content goes unused

Practical Implications

• Filtering and deduplication: The findings motivate aggressive filtering of recalled content before delivery, especially given that restatements of still-visible content consume significant tokens

• Delivery form matters: Full content (cards, skills, recall) drives benefit; previews and indexes mainly support selection—suggesting memory systems should prioritize full-content delivery of high-value items

• Task-specific organization: The gap between retrieval and use motivates organizing recalled content into task-specific guidance rather than generic information

• Memory maintenance: Tracing harm to specific recurring items (e.g., two MemGovern cards producing 7/10 harmful entries) identifies direct targets for revision or removal

• Cost-aware design: Token efficiency metrics (50.50 beneficial entries per million tokens for MemGovern vs. 3.08–4.92 for others) provide concrete optimization targets

Conclusion

MemUse provides the first reference-free framework for evaluating memory deliveries through their context, use, and observed consequences. Applied to 8,000 SWE-bench Verified episodes, it reveals that similar resolve rates hide dramatically different memory behaviors: most delivered entries are irrelevant, most relevant entries go unused, and task outcomes conceal both beneficial use in failed tasks and harmful detours in successful ones.

The findings make the transition from retrieval to use a central design concern for memory systems, motivating:

• Filtering and deduplicating recalled content
• Organizing it into task-specific guidance
• Delivering it when the agent can act on it
• Tracking memory histories to identify and remove recurring harmful items

Future directions include extending MemUse to other agent architectures and task domains, incorporating counterfactual reasoning where feasible, and developing automated methods to optimize memory delivery based on the fine-grained feedback MemUse provides.