Summary (Overview)

• MemDev-Bench is a new benchmark to evaluate how memory systems affect code agents' performance across diverse real repository histories, addressing a critical gap: existing benchmarks lack either temporally valid histories or control over prior-task relevance.
• The benchmark pairs 555 base tasks (from SWE-bench Verified and SWE-bench Pro) with 4,869 relevance-annotated historical tasks mined from real GitHub repositories, spanning Python, Go, JavaScript, and TypeScript across 22 repositories.
• Historical tasks are labeled as Strong Related, Weak Related, or Unrelated to each base task, enabling controlled experiments on how memory helps or hurts performance.
• A new automated pipeline constructs executable, verifiable historical tasks using a rule–experience–agent hierarchy, addressing issues in prior pipelines with patch splitting, environment changes, and cost.
• Key findings: (1) Memory mainly improves fine-grained localization and editing (Line/Edit Recall), increasing functional correctness (F2P); (2) Memory also causes regressions on solvable tasks, especially with unrelated (noisy) history, due to agents failing to reassess memory applicability; (3) Raw full trajectories (FullTraj) substantially reduce agent steps compared to memory systems, suggesting compressed memories omit valuable process information.

---

Introduction and Theoretical Foundation

Background and Motivation

Code agents powered by large language models (LLMs) have made significant progress on real-world software engineering tasks such as bug fixing and feature development. However, existing agents typically solve each task in isolation, starting from scratch each time. Human developers, by contrast, accumulate experience from past issues and reuse it to solve new problems efficiently.

Memory systems (e.g., A-Mem, Mem0, MemOS, Supermemory, Claude-Mem) aim to close this gap by allowing code agents to retain repository context and problem-solving experience across tasks. However, real repository histories contain a mix of valuable experience and irrelevant information, raising a critical question: "Do memory systems consistently benefit code agents across long-term repository development?"

Requirements for a Proper Benchmark

The authors identify two necessary properties that prior benchmarks lack:

1. Temporally valid histories: Each prior task must precede the target task without revealing its solution; otherwise memory gains may come from answer leakage rather than experience reuse.
2. Controlled relevance: Whether a prior task can help the target task must be specified, so the benchmark can separate ineffective memory use from missing useful experience and test whether irrelevant history misleads agents.

Deficiencies in Existing Benchmarks

---

Methodology

3.1 Benchmark Construction Pipeline

The construction process has three main stages:

Stage 1: Candidate Historical PR Mining

Two complementary routes identify candidate PRs:

• Reference-based discovery: Extracts references from the base PR's description and linked issues, expanding up to two hops. Candidate set $R_{ref}$ consists of same-repository PRs.
• Code-history-based discovery: Traces commit histories of files modified by the base PR, mapping earlier commits to PRs. Two filters reduce accidental overlap:
  • Filter 1: PRs must share at least $\min(3, n)$ changed files with the base PR (where $n$ = files changed by base PR).
  • Filter 2: PRs embedded with MiniMax embo-01, keeping those with cosine similarity > 0.65, forming $R_{code}$.

The preliminary pool is $R_0 = R_{ref} \cup R_{code}$. Human reviewers screen for solution leakage (e.g., backports, near-duplicate fixes), and the remaining candidates form pool $R$.

Stage 2: Candidate Historical PR Annotation

Each candidate receives one of three relevance levels:

Annotation is done via:

1. DeepSeek-V4-Pro LLM labeling (20 fine-grained relation types)
2. Three senior software engineers verify and cross-check all annotations (87.82% of LLM annotations retained)
3. Independent blind annotation of 2,270 samples achieves Fleiss' κ = 0.88 for relevance level and 0.80 for relation type

Stage 3: Executable Historical Task Construction

The pipeline addresses three key issues in prior methods:

3.2 Benchmark Statistics

Historical task composition:

Time gaps span from under a day to several years; patch sizes and changed-file counts span wide ranges at every relevance level.

3.3 Experimental Setup

• Agent: Claude Code; Base model: DeepSeek-V4-Flash
• Information isolation: Blocks search engines and public code-hosting platforms to prevent answer leakage. Without this, the resolve rate on SWE-bench Verified tasks is 93.31% ± 0.87%; with isolation, it drops to 73.16% ± 0.61%.
• Task subsets:
  • Improvement Tasks (133): fail in ≥2 of 3 no-memory runs
  • Preservation Tasks (50 sampled from 211): succeed in ≥2 of 3 no-memory runs
• History scenarios:
  • Relevant History: Strong + Weak Related (3–10 tasks)
  • Noisy History: Adds Unrelated tasks (7–21 tasks)
  • Repo History: Related + most recent tasks from other base tasks (13–35 tasks, capped)
• Baselines: NoMem (no history), FullTraj (raw historical trajectories in context)
• Memory systems evaluated: A-Mem, Mem0, MemOS, Supermemory, Claude-Mem (29 systems in additional experiments)

---

Empirical Validation / Results

4.1 End-to-End Performance

Improvement Tasks (Table 2): Resolved rises from 35.3% (NoMem) to 39.6% (FullTraj) and up to 42.6% (Claude-Mem, best under Relevant and Repo history). Key metrics:

• Rescue: 9.9%–13.2%
• Harm: 4.8%–9.9% (Claude-Mem lowest at 4.8% under Repo history)

Preservation Tasks (Table 3): NoMem resolves 94.0%. With memory systems, Harm exceeds Rescue under Noisy (3.7% vs. 3.2%) and Repo (3.2% vs. 2.6%) history, lowering Resolved to 93.5% and 93.3% respectively.

Key pattern: History noise tends to increase regressions (Harm) rather than suppress Rescue. Mean Rescue stays nearly constant across scenarios (11.7%, 11.5%, 11.7%) while mean Harm rises (7.1% → 8.0%) from Relevant to Noisy.

4.2 Attribution of Memory Gains

On 68 tasks where all NoMem runs fail:

Interpretation: Memory rarely expands gold-file coverage but more consistently improves within-file localization and editing, leading to greater functional correctness improvements.

4.3 Attribution of Memory-Induced Regressions

On 76 tasks solved in all three NoMem runs, manual inspection identifies 115 failures caused by memory. Four recurring failure modes:

1. Compatibility mismatch — memory assumptions don't fit current task
2. Misapplied transfer — historical solution applied to wrong component/stage (most common, especially under Noisy history)
3. Scope expansion — extending history-derived solutions beyond task needs
4. Failure to revise — following memory despite conflicting evidence (most common under Repo history)

4.4 How FullTraj Changes the Solving Process

Compared to NoMem and memory systems (averaged):

The largest step reduction occurs during the edit span (24.8–31.3 steps). FullTraj retains process information (code inspection sequences, failed attempts, revisions, test outcomes) that compressed memories omit, requiring fewer reconstruction steps.

---

Theoretical and Practical Implications

Design Directions for Future Memory Systems

The results suggest two key requirements:

1. Preserve process information: Compressed memories are useful for high-level guidance but unreliable for fine-grained decisions. FullTraj's superior step efficiency suggests retaining the solving process in execution order helps agents avoid costly reconstruction of omitted details.

2. Reassess memory applicability: Agents often follow memory without checking whether its assumptions hold for the current task. Memory systems should expose when and for what each memory applies.

The central challenge: Coupling memory exposure (what granularity/amount of evidence to present) with memory reliance (how strongly that evidence should influence action) at decision time. Recent work (MCMA, MemHarness, CoEvo-Mem, BASM, MSCE, Recuris) addresses one side or the other but not both in an integrated, adaptive manner. Effective memory should serve as adaptive decision support, not fixed historical guidance.

Practical Implications

• Noisy, realistic repository histories tend to increase Harm more than they reduce Rescue, meaning deployment of memory systems in production must carefully manage noise.
• Existing memory systems may not justify their compression overhead versus simply providing raw trajectories, since FullTraj achieves comparable or better results with fewer steps.
• The information-isolation protocol is crucial for evaluating memory: without it, answer leakage inflates resolve rates and obscures memory effects.

---

Conclusion

Main Takeaways

1. Memory systems can improve code agent performance on tasks that would otherwise fail, mainly through finer-grained localization and editing within files.
2. Memory also causes failures on otherwise solvable tasks, even with relevant history; unrelated history amplifies these regressions. Agents fail by reusing memory without reassessing applicability.
3. Raw historical trajectories (FullTraj) substantially reduce solving steps compared to compressed memory systems, suggesting that future memory systems should preserve process information and reassess memory applicability against current task context.

Limitations

• Evaluation limited to SWE-bench Verified subset under a single agent–model configuration
• No systematic comparison of construction pipeline with existing methods
• Broader evaluation and direct pipeline comparisons remain future work

Future Directions

• Coupling memory exposure and memory reliance at decision time
• Determining the granularity and amount of historical evidence needed per decision
• Adaptively adjusting historical evidence selection and influence as current state and feedback evolve