# MemDev-Bench: Does Memory Help Code Agents?Benchmarking Memory Systems over Real Repository Histories

> Memory improves code agents mainly through finer-grained localization and editing, but noisy history causes regressions by fostering unrevised, misapplied reuse.

- **Source:** [arXiv](https://arxiv.org/abs/2610.27524)
- **Published:** 2026-10-24
- **Permalink:** https://picx.dev/p/lIPZw6
- **Whiteboard:** https://picx.dev/p/lIPZw6/image

## Summary

## Summary (Overview)

- **MemDev-Bench** is a new benchmark to evaluate how memory systems affect code agents' performance across diverse real repository histories, addressing a critical gap: existing benchmarks lack either temporally valid histories or control over prior-task relevance.
- The benchmark pairs 555 base tasks (from SWE-bench Verified and SWE-bench Pro) with 4,869 relevance-annotated historical tasks mined from real GitHub repositories, spanning Python, Go, JavaScript, and TypeScript across 22 repositories.
- Historical tasks are labeled as **Strong Related**, **Weak Related**, or **Unrelated** to each base task, enabling controlled experiments on how memory helps or hurts performance.
- A new automated pipeline constructs executable, verifiable historical tasks using a rule–experience–agent hierarchy, addressing issues in prior pipelines with patch splitting, environment changes, and cost.
- **Key findings:** (1) Memory mainly improves fine-grained localization and editing (Line/Edit Recall), increasing functional correctness (F2P); (2) Memory also causes regressions on solvable tasks, especially with unrelated (noisy) history, due to agents failing to reassess memory applicability; (3) Raw full trajectories (FullTraj) substantially reduce agent steps compared to memory systems, suggesting compressed memories omit valuable process information.

---

## Introduction and Theoretical Foundation

### Background and Motivation

Code agents powered by large language models (LLMs) have made significant progress on real-world software engineering tasks such as bug fixing and feature development. However, existing agents typically solve each task **in isolation**, starting from scratch each time. Human developers, by contrast, accumulate experience from past issues and reuse it to solve new problems efficiently.

**Memory systems** (e.g., A-Mem, Mem0, MemOS, Supermemory, Claude-Mem) aim to close this gap by allowing code agents to retain repository context and problem-solving experience across tasks. However, real repository histories contain a **mix of valuable experience and irrelevant information**, raising a critical question: *"Do memory systems consistently benefit code agents across long-term repository development?"*

### Requirements for a Proper Benchmark

The authors identify two necessary properties that prior benchmarks lack:

1. **Temporally valid histories**: Each prior task must precede the target task without revealing its solution; otherwise memory gains may come from *answer leakage* rather than experience reuse.
2. **Controlled relevance**: Whether a prior task can help the target task must be specified, so the benchmark can separate *ineffective memory use* from *missing useful experience* and test whether irrelevant history misleads agents.

### Deficiencies in Existing Benchmarks

| Benchmark Type | Deficiency |
|---|---|
| Session-based benchmarks | Histories built from synthetic multi-session scenarios, not real repository evolution |
| Repository-based benchmarks (no temporal filtering) | Histories may include tasks that postdate the target task, leaking solutions |
| Repository-based benchmarks (with temporal order) | Overlook actual task relevance OR use chains of almost entirely interdependent tasks |

---

## Methodology

### 3.1 Benchmark Construction Pipeline

The construction process has three main stages:

#### Stage 1: Candidate Historical PR Mining

Two complementary routes identify candidate PRs:

- **Reference-based discovery**: Extracts references from the base PR's description and linked issues, expanding up to two hops. Candidate set $R_{ref}$ consists of same-repository PRs.
- **Code-history-based discovery**: Traces commit histories of files modified by the base PR, mapping earlier commits to PRs. Two filters reduce accidental overlap:
  - Filter 1: PRs must share at least $\min(3, n)$ changed files with the base PR (where $n$ = files changed by base PR).
  - Filter 2: PRs embedded with MiniMax embo-01, keeping those with cosine similarity > 0.65, forming $R_{code}$.

The preliminary pool is $R_0 = R_{ref} \cup R_{code}$. Human reviewers screen for solution leakage (e.g., backports, near-duplicate fixes), and the remaining candidates form pool $R$.

#### Stage 2: Candidate Historical PR Annotation

Each candidate receives one of three relevance levels:

| Level | Definition |
|---|---|
| **Strong Related** | Provides direct, actionable help (reusable implementation, same root cause) |
| **Weak Related** | Provides relevant context, but the solution cannot transfer directly |
| **Unrelated** | Provides no substantive help (includes unmerged PRs reflecting failed attempts) |

Annotation is done via:
1. DeepSeek-V4-Pro LLM labeling (20 fine-grained relation types)
2. Three senior software engineers verify and cross-check all annotations (87.82% of LLM annotations retained)
3. Independent blind annotation of 2,270 samples achieves **Fleiss' κ = 0.88** for relevance level and **0.80** for relation type

#### Stage 3: Executable Historical Task Construction

The pipeline addresses three key issues in prior methods:

| Issue | Solution |
|---|---|
| Test/solution patch misclassification | Patch splitter combining repository test configurations with AST-based static analysis |
| Missed patch-induced environment changes | Pre- and post-fix environments prepared separately |
| High cost of agent-dependent pipelines | Rule–experience–agent hierarchy: agent invoked only when deterministic rules and validated experience fail |

### 3.2 Benchmark Statistics

**Historical task composition:**

| Relevance Level | Verified (%) | Pro (%) |
|---|---|---|
| Strong Related | 19.6 | 13.5 |
| Weak Related | 20.5 | 16.9 |
| Unrelated, merged | 47.0 | 65.2 |
| Unrelated, unmerged | 12.9 | 4.4 |

Time gaps span from under a day to several years; patch sizes and changed-file counts span wide ranges at every relevance level.

### 3.3 Experimental Setup

- **Agent**: Claude Code; **Base model**: DeepSeek-V4-Flash
- **Information isolation**: Blocks search engines and public code-hosting platforms to prevent answer leakage. Without this, the resolve rate on SWE-bench Verified tasks is 93.31% ± 0.87%; with isolation, it drops to 73.16% ± 0.61%.
- **Task subsets**:
  - **Improvement Tasks** (133): fail in ≥2 of 3 no-memory runs
  - **Preservation Tasks** (50 sampled from 211): succeed in ≥2 of 3 no-memory runs
- **History scenarios**:
  - **Relevant History**: Strong + Weak Related (3–10 tasks)
  - **Noisy History**: Adds Unrelated tasks (7–21 tasks)
  - **Repo History**: Related + most recent tasks from other base tasks (13–35 tasks, capped)
- **Baselines**: NoMem (no history), FullTraj (raw historical trajectories in context)
- **Memory systems evaluated**: A-Mem, Mem0, MemOS, Supermemory, Claude-Mem (29 systems in additional experiments)

---

## Empirical Validation / Results

### 4.1 End-to-End Performance

**Improvement Tasks** (Table 2): Resolved rises from **35.3%** (NoMem) to **39.6%** (FullTraj) and up to **42.6%** (Claude-Mem, best under Relevant and Repo history). Key metrics:

- **Rescue**: 9.9%–13.2%
- **Harm**: 4.8%–9.9% (Claude-Mem lowest at 4.8% under Repo history)

**Preservation Tasks** (Table 3): NoMem resolves 94.0%. With memory systems, **Harm exceeds Rescue** under Noisy (3.7% vs. 3.2%) and Repo (3.2% vs. 2.6%) history, lowering Resolved to 93.5% and 93.3% respectively.

**Key pattern**: History noise tends to increase regressions (Harm) rather than suppress Rescue. Mean Rescue stays nearly constant across scenarios (11.7%, 11.5%, 11.7%) while mean Harm rises (7.1% → 8.0%) from Relevant to Noisy.

### 4.2 Attribution of Memory Gains

On 68 tasks where all NoMem runs fail:

| Metric | NoMem | Memory Systems (range) |
|---|---|---|
| File Recall | 74.22% | 70.42%–76.76% (rarely exceeds baseline) |
| Line Recall | 18.41% | 17.65%–22.54% (13/18 settings exceed) |
| Edit Recall | 11.02% | 9.69%–14.77% (16/18 settings exceed) |
| F2P (functional correctness) | 18.10% | 23.15%–28.57% |
| P2P (behavior preservation) | 97.39% | 97.05%–98.44% |

**Interpretation**: Memory rarely expands gold-file coverage but more consistently improves **within-file localization and editing**, leading to greater functional correctness improvements.

### 4.3 Attribution of Memory-Induced Regressions

On 76 tasks solved in all three NoMem runs, manual inspection identifies **115 failures caused by memory**. Four recurring failure modes:

1. **Compatibility mismatch** — memory assumptions don't fit current task
2. **Misapplied transfer** — historical solution applied to wrong component/stage (most common, especially under Noisy history)
3. **Scope expansion** — extending history-derived solutions beyond task needs
4. **Failure to revise** — following memory despite conflicting evidence (most common under Repo history)

### 4.4 How FullTraj Changes the Solving Process

Compared to NoMem and memory systems (averaged):

| Metric | Reduction |
|---|---|
| Steps | ~34% fewer than NoMem; 29–35% fewer than memory systems |
| Output tokens | ~28% fewer than NoMem; 28–34% fewer than memory systems |
| Search calls | 5.9–8.1 fewer per trajectory |
| Read calls | 5.3–8.9 fewer per trajectory |

The largest step reduction occurs during the **edit span** (24.8–31.3 steps). FullTraj retains process information (code inspection sequences, failed attempts, revisions, test outcomes) that compressed memories omit, requiring fewer reconstruction steps.

---

## Theoretical and Practical Implications

### Design Directions for Future Memory Systems

The results suggest two key requirements:

1. **Preserve process information**: Compressed memories are useful for high-level guidance but unreliable for fine-grained decisions. FullTraj's superior step efficiency suggests retaining the solving process in execution order helps agents avoid costly reconstruction of omitted details.

2. **Reassess memory applicability**: Agents often follow memory without checking whether its assumptions hold for the current task. Memory systems should expose *when* and *for what* each memory applies.

**The central challenge**: Coupling memory exposure (what granularity/amount of evidence to present) with memory reliance (how strongly that evidence should influence action) at decision time. Recent work (MCMA, MemHarness, CoEvo-Mem, BASM, MSCE, Recuris) addresses one side or the other but not both in an integrated, adaptive manner. Effective memory should serve as **adaptive decision support**, not fixed historical guidance.

### Practical Implications

- Noisy, realistic repository histories tend to increase Harm more than they reduce Rescue, meaning deployment of memory systems in production must carefully manage noise.
- Existing memory systems may not justify their compression overhead versus simply providing raw trajectories, since FullTraj achieves comparable or better results with fewer steps.
- The information-isolation protocol is crucial for evaluating memory: without it, answer leakage inflates resolve rates and obscures memory effects.

---

## Conclusion

### Main Takeaways

1. Memory systems can improve code agent performance on tasks that would otherwise fail, mainly through **finer-grained localization and editing** within files.
2. Memory also causes failures on otherwise solvable tasks, even with relevant history; unrelated history amplifies these regressions. Agents fail by **reusing memory without reassessing applicability**.
3. Raw historical trajectories (FullTraj) substantially reduce solving steps compared to compressed memory systems, suggesting that future memory systems should **preserve process information** and **reassess memory applicability** against current task context.

### Limitations

- Evaluation limited to SWE-bench Verified subset under a single agent–model configuration
- No systematic comparison of construction pipeline with existing methods
- Broader evaluation and direct pipeline comparisons remain future work

### Future Directions

- Coupling memory exposure and memory reliance at decision time
- Determining the granularity and amount of historical evidence needed per decision
- Adaptively adjusting historical evidence selection and influence as current state and feedback evolve

---

_Markdown view of https://picx.dev/p/lIPZw6, served by PicX — AI-generated visual whiteboard summaries of research papers._