# VibeMemBench: Evaluating Memory Systems for Coding Agents on Real Repository Coding Tasks

> Existing memory systems fail to improve coding agent performance because raw transcript volume, not instruction semantics, degrades retrieved fix patterns in 69.3% of failures.

- **Source:** [arXiv](https://arxiv.org/abs/2609.23570)
- **Published:** 2026-09-26
- **Permalink:** https://picx.dev/p/1Ps2mo
- **Whiteboard:** https://picx.dev/p/1Ps2mo/image

## Summary

## Summary (Overview)

- **VibeMemBench** is a new benchmark for evaluating persistent memory systems in coding agents, built on 111 coding targets from 90 SWE-rebench V2 repositories with 3,634 history trajectories.
- The benchmark uses a **matched intervention protocol** where only the memory condition changes between paired runs, with executable tests determining task resolution.
- A four-stage **SIEVE construction pipeline** filters history by patch patterns and retains only targets where injected experience shows verified uplift in a reference setting.
- **Key finding**: Frozen verified experience raises task resolution on 4 of 5 transfer solvers (by 1.1–4.5 percentage points), but 11 of 12 existing memory system–solver pairings fail to beat the memory-off baseline.
- **Failure attribution** identifies "form degradation" (69.3% of failures) as the dominant break, with transcript volume rather than instruction semantics as the primary harm mechanism.

---

## Introduction and Theoretical Foundation

### Background and Motivation

Persistent memory systems promise to reuse experience across coding tasks, yet existing evaluations fail to measure whether these systems improve executable repository work:

- **Memory benchmarks** (LoCoMo, LongMemEval, MemBench) probe fact recall from interaction histories but don't establish whether recalled experience helps repair a repository.
- **Repository benchmarks** (SWE-bench, RepoBench) measure executable task outcomes but don't isolate a persistent memory intervention.

The core measurement problem: executable tests give coding tasks a grounded outcome, while *useful memory remains latent*—no instance-level label says which earlier observation a system ought to write, retain, or retrieve.

### Theoretical Foundation

The benchmark reframes memory evaluation as a **matched comparison of repository work**:

> Both runs share the task, agent, tools, sandbox, and budget. Only the declared memory condition changes. Executable tests measure whether the supplied experience improves task resolution rather than whether it matches a gold retrieval label.

---

## Methodology

### SIEVE Construction Pipeline

Four stages control what enters the benchmark:

1. **Source validation**: Runs declared tests with the gold patch, discarding instances where the gold patch fails.
2. **Patch-pattern screening**: Oracle-assisted offline screening admitting only earlier instances from the same repository, excluding target identifiers. A deterministic score over shared files/directories shortlists candidates; an LLM judges repair patterns with structured evidence.
3. **History execution with experience distillation**: Executes selected history tasks, distilling one experience per pair from the target problem statement, history problem statement, compressed trajectory, and history final patch. Each experience records a bug class, root cause, fix pattern, and reusable lesson.
4. **Retention by uplift**: Retains targets only when injected experience shows positive uplift in a reference setting.

### Uplift Definition

For target $t$ with candidate-memory combinations $k \in \mathcal{K}_t$ (with $0 \notin \mathcal{K}_t$ denoting memory-off):

$$
g_{t,k} = \hat{p}_{t,k}^{\mathrm{ref}} - \hat{p}_{t,0}^{\mathrm{ref}} \tag{1}
$$

where $\hat{p}_{t,k}^{\mathrm{ref}}$ is the resolved share for target $t$ after injecting combination $k$, and $\hat{p}_{t,0}^{\mathrm{ref}}$ is the memory-off share. Retention requires $\hat{p}_{t,0}^{\mathrm{ref}} < 1$ and $g_{t,k_t^\star} > 0$.

### Matched Intervention Protocol

For each $(t,s) \in \mathcal{P} = \mathcal{T} \times \mathcal{S}$ (targets × seeds), VibeMemBench compares $A(t,s,\mathrm{off})$ and $A(t,s,\mathrm{on})$ where:
- $A$ = fixed agent configuration (model, prompt, tools, sandbox, context budget, resource limits)
- $m \in \{\mathrm{off}, \mathrm{on}\}$ = memory condition, changing **only** the declared persistence and retrieval mechanism

### Evaluation Layers

1. **Frozen verified experience transfer**: Tests whether the reference-identified experience transfers across five solver models (deepseek-v4-pro, glm-5, glm-5.2, kimi-k2.7-code, qwen3.8-max) with 4 seeds per condition.
2. **Existing memory system retrieval**: Mem0, SimpleMem, MemoryOS, and A-MEM ingest full eligible history; a declared target query selects one top-ranked experience for injection.

---

## Empirical Validation / Results

### Metrics

The primary metric is **Resolved**—the share of target-seed runs passing the declared test predicate:

$$\widehat{\Delta} = \widehat{p}_{\mathrm{on}} - \widehat{p}_{\mathrm{off}}$$

Uncertainty is reported via percentile bootstrap (10,000 resamples of the 111 targets with seeds intact).

### Frozen Verified Experience Transfer (Table 1)

| Condition | deepseek-v4-pro | glm-5 | glm-5.2 | kimi-k2.7-code | qwen3.8-max |
|---|---|---|---|---|---|
| Memory off | 67.1% | 55.0% | 76.8% | 69.8% | 79.1% |
| Frozen verified | 67.1% (+0.0) | 57.4% (+2.4) | 80.4% (+3.6) | 74.3% (+4.5) | 80.2% (+1.1) |
| Irrelevant control | 63.7% (−3.4) | 54.3% (−0.7) | — | 69.1% (−0.7) | — |

**Key observations**:
- All five bootstrap intervals cross zero (directional evidence, not separable from zero at 111 targets)
- Irrelevant-memory control stays below baseline—injected text alone reproduces none of the gains
- Injection lowers agent steps for **every** solver and input tokens for 4 of 5

### Existing Memory Systems (Table 2)

**11 of 12 solver–system pairings fail to exceed the matched memory-off baseline**. The single exception: MemoryOS on glm-5 (+2.0 points), which still trails the frozen verified experience.

### Failure Attribution (Table 4)

| First broken stage | Definition | Pairings | Share |
|---|---|---|---|
| Coverage miss | Source history never entered store | 37 | 16.0% |
| Ranking miss | Stored, but no anchor shared | 3 | 1.3% |
| **Form degradation** | **Fix pattern dropped/buried in raw transcript** | **160** | **69.3%** |
| Use failure | Record matches reference, yet run fails | 31 | 13.4% |

### Form Degradation Mechanisms (Table 5)

Instruction pollution (40.0%) and overgeneralized fix patterns (18.1%) dominate. Transcript systems (MemoryOS, A-MEM) produce all instruction pollution and shell noise; atomic systems (Mem0, SimpleMem) account for overgeneralization, schema omission, and symptom substitution.

### Headroom Conditioning (Table 3)

Among 118 pairs where both solvers move, 64 move in opposite directions. In 61 of those, the solver with lower memory-off count gains. Opposite-sign share rises with memory-off gap (52.3% at 1 seed → 85.7% at 3+ seeds).

### Strip Ablation

On 62 of 64 instruction pollution pairings, deleting flagged lines recovers 0.60 seeds/pairing; deleting equally many random lines recovers 0.48. Because random deletion recovers comparably:

> The harm operates through transcript volume rather than instruction semantics.

---

## Theoretical and Practical Implications

### Design Implications

1. **Gated injection on expected solver headroom**: When a solver fails without memory and the record carries at least a directory/identifier anchor, the record raises Resolved on 55.2% of pairings with no observed loss. At a 4-of-4 ceiling, even the frozen verified experience lowers Resolved on 25.7% of pairings.

2. **Compress transcripts to fix-stating spans**: The dominant break is raw transcript volume, not instruction semantics. Systems should compress retrieved trajectories to the span stating the fix.

3. **Preserve fix fields with anchoring constraints**: An extraction schema must retain constraints anchoring the fix to the target—preserving the fix pattern alone is necessary but not sufficient.

### Availability-to-Use Gap

The benchmark separates two questions:
- Whether frozen verified experience helps a solver once it reaches context (yes, modestly)
- Whether existing memory systems turn history into useful context (mostly no)

The gap is a **floor**—the frozen record witnesses what history makes available without bounding it from above.

---

## Conclusion

VibeMemBench exposes the gap between the useful experience that repository history holds and the experience existing memory systems deliver for repository coding tasks. The benchmark's contributions:

1. **Construction**: 3,634 trajectories with SIEVE leakage control, verifying useful experience exists for every retained target.
2. **Protocol**: Matched intervention for agent components lacking an oracle, with frozen verified experience transfer separated from system retrieval under an auditable reporting contract.
3. **Findings**: The dominant break is record form (raw transcript volume), and record value is conditioned on solver headroom rather than intrinsic.

### Limitations

- Solver pool omits proprietary frontier models (Claude, GPT families) due to inference budget
- The 111 targets have positive reference uplift **by design**—suitable for measuring downstream value of known transferable experience, not estimating how often deployed agents find such pairs
- Selection creates a boundary: the retrieval-cohort ceiling reflects solver strength rather than lost opportunity

### Future Directions

The release supplies task histories, executable predicates, and the intervention protocol with full provenance, enabling further study of memory system design for repository coding tasks—particularly form-aware record compression and headroom-gated injection policies.

---

_Markdown view of https://picx.dev/p/1Ps2mo, served by PicX — AI-generated visual whiteboard summaries of research papers._