# Coding Agent Memory Post-training: Unlocking the Memory Potential of Pre-trained File Operations for Long-Horizon Tasks via Reinforcement Learning

> CAMG-RL trains a single policy to use ordinary filesystem operations as memory via asynchronous PPO, matching models 4-13x larger on external benchmarks.

- **Source:** [arXiv](https://arxiv.org/abs/2609.34422)
- **Published:** 2026-10-17
- **Permalink:** https://picx.dev/p/PyRijr
- **Whiteboard:** https://picx.dev/p/PyRijr/image

## Summary

## Summary (Overview)

- **Novel Environment Suite**: Introduces **Coding Agent Memory Gym (CAMG)** , a suite of four long-horizon agentic-RL environments (Shop, Coding, DeepResearch, AutoResearch) that pair native task interfaces with executable shell access and an episode-persistent workspace, enabling agents to use ordinary filesystem operations as memory.
- **New RL Algorithm**: Proposes **CAMG-RL**, which trains a single policy jointly across all four environments using fully asynchronous PPO, learning file-based memory behavior directly from downstream task reward without introducing any special memory tools.
- **Strong Empirical Results**: CAMG-RL-4B achieves 54.4% average success rate across the CAMG test set, outperforming training-free memory systems, learned compaction (48.0%), and tool-based memory baselines.
- **External Transfer**: CAMG-RL-4B matches Qwen3.5-35B-A3B and CAMG-RL-9B is competitive with Qwen3.5-122B-A10B on SWE-bench Verified and MLE-bench Lite, despite being trained on much smaller backbones.
- **Key Insight**: The frozen base model already prefers filesystem memory actions over dedicated memory tools (lower surprisal in nats/token), and RL training converts this pre-trained preference into complete memory chains with minimal weight movement.

## Introduction and Theoretical Foundation

### Background and Motivation

Language-model agents increasingly tackle long-horizon tasks involving repeated tool use and environment interaction. As interactions accumulate, information needed for later decisions can fall out of the model's active context. For example, a coding agent may need to recall why an earlier fix was rejected after that exchange has left the prompt.

### Limitations of Prior Work

Recent work uses reinforcement learning to make memory control part of the policy, but relies on:
- **Predefined memory tools** within domain-specific training environments
- **Relatively short horizons**

This setup ties learned memory behavior to environment-specific interfaces that lie outside the base model's pre-training and must be learned from scratch. Even after post-training, agents struggle to use memory in long-horizon tasks.

### Theoretical Foundation

The paper models each environment as a finite-horizon partially observable Markov decision process:

$$M^d = (U^d, S^d, A^d, O^d, T^d, r^d)$$

where $U^d$ is the instruction space, $S^d$ the latent-state space, $A^d$ the action space, $O^d$ the observation space, $T^d$ the state-transition function, and $r^d$ the reward function.

The policy maximizes expected task return:

$$J_d(\theta) = \mathbb{E}_{\tau \sim p_{\theta,d}}[G_d(\tau)]$$

with the vanilla policy gradient:

$$\nabla_\theta J_d(\theta) = \mathbb{E}_{\tau \sim p_{\theta,d}}\left[G_d(\tau) \sum_{t=1}^{T(\tau)} \nabla_\theta \log \pi_\theta(a_t | h_t)\right]$$

## Methodology

### CAMG Environment Suite

The four environments are constructed as follows:

| Environment | Task Description | Data Source | Task Count |
|---|---|---|---|
| **Shop** | Retain customer requirements across 6 dependent purchase sessions | 720 verified WebShop products | Generated |
| **Coding** | Complete software repair spanning multiple active contexts | SWE-smith (93 repositories) | 11,218 |
| **DeepResearch** | Accumulate and reuse retrieved evidence over extended search | LiteResearcher (fixed retrieval corpus) | 26,597 |
| **AutoResearch** | Preserve hypotheses and validation evidence across iterative model development | OpenMLE task packages | 931 |

### Common Coding-Agent Interface

The action space combines native task actions with shared filesystem actions:

$$A_d = A^{\text{task}}_d \cup A^{\text{fs}}, \quad a_t \in A_d$$

The shared filesystem action is a single `shell_command` that can create, inspect, search, modify, and execute files. This allows the policy to preserve and revise task-relevant evidence in the workspace across context replacements.

### CAMG-RL: Dual-Granularity PPO

The joint objective across all four environments is:

$$J_{\text{CAMG-RL}}(\theta) = \frac{1}{|\mathcal{D}|} \sum_{d \in \mathcal{D}} J_d(\theta), \quad |\mathcal{D}| = 4$$

**Dual-granularity PPO** combines:
- **Turn-level advantage estimation**: One reward arrives per response, providing stability
- **Token-level ratio clipping**: Each sampled token gets its own clipped ratio, providing fine-grained updates

This differs from existing approaches:
- **Turn-PPO**: Turn-level for both estimation and update (stable but coarse)
- **SAO**: Token-level for both (fine-grained but high-variance)

### Fully Asynchronous PPO

Episode durations vary across CAMG due to different turn counts and tool execution costs. The algorithm runs rollout and learning fully asynchronously:
1. One rollout process collects trajectories from all four environments
2. Workers append completed episodes to a persistent queue
3. Learner assembles them into optimization batches
4. Updated actor weights are published for later episodes

This prevents workers from waiting for the slowest episode.

## Empirical Validation / Results

### CAMG Test Set Results

**Table 2: CAMG-RL achieves the highest average success rate across the CAMG test set**

| Method | Shop | Coding | Deep Research | Auto Research | Average |
|---|---|---|---|---|---|
| Qwen3.5-4B | 11.6 | 10.9 | 36.7 | 10.2 | 17.4 |
| Mem0 | 15.6 | 14.8 | 37.5 | 10.9 | 19.7 |
| Letta Code | 12.6 | 14.8 | 32.0 | 11.7 | 17.8 |
| CompactionRL | 97.5 | 19.5 | 50.8 | 24.2 | 48.0 |
| AgeMem | 16.7 | 26.6 | 28.1 | 29.7 | 25.3 |
| **CAMG-RL** | **97.1** | **26.6** | **55.5** | **38.3** | **54.4** |

### Key Findings on Memory Behavior

1. **Frozen-base action surprisal**: Filesystem memory actions cost 0.076–1.193 nats/token across paired store/retrieve/revise/remove intents, while AgeMem's six dedicated tools are more surprising in every intent. Renaming the filesystem tool to an unseen name keeps surprisal within 0.013 nats/token of native actions.

2. **Memory chain acquisition**: CAMG-RL's complete store–retrieve–use chains rise from 8.0% of episodes (first quarter) to 20.9% (last quarter), while AgeMem attempts its memory tools in only 53 of 261,941 responses and never completes a chain.

3. **Minimal policy shift**: Both distance measures (KL and weight change) are small for CAMG-RL, yet it converts this movement into complete memory chains—training on filesystem memory makes effective use of pre-training.

### External Benchmark Transfer

**Table 3: CAMG-RL matches much larger frozen models on external benchmarks**

| Method | SWE-bench Verified | MLE-bench Lite | Average |
|---|---|---|---|
| Qwen3.5-4B | 7.6 | 0.0 | 3.8 |
| Qwen3.5-9B | 14.8 | 4.5 | 9.7 |
| Qwen3.5-27B | 17.4 | 4.5 | 11.0 |
| Qwen3.5-35B-A3B | 15.6 | 4.5 | 10.1 |
| Qwen3.5-122B-A10B | 22.0 | 9.1 | 15.6 |
| Qwen3.5-397B-A17B | 34.4 | 13.6 | 24.0 |
| **CAMG-RL-4B** | **15.8** | **4.5** | **10.2** |
| **CAMG-RL-9B** | **27.6** | **9.1** | **18.4** |

### Ablation Study

Component attribution (Figure 4):
- **Full CAMG-RL**: 54.4% average
- **Without memory-action gradients**: 21.1% (near frozen base)
- **Retrained without general memory files**: 41.8%
- **Evaluated without general memory files**: 28.1%

### Intervention Study

Blanking saved memory files at the first context replacement cuts average success by **14.1 points**; transplanting task-mismatched content cuts it by **11.5 points**, with the largest drop in Shop—confirming saved memory content causally drives later success.

## Theoretical and Practical Implications

### Theoretical Contributions

1. **Pre-training as memory priors**: The paper demonstrates that code pre-training implicitly provides a strong prior for filesystem-based memory operations. The frozen base's surprisal analysis (0.076–1.193 nats/token for filesystem actions vs. higher for dedicated tools) provides quantitative evidence that pre-trained capabilities can be elicited through RL rather than learned from scratch.

2. **Interface alignment matters**: The gap between CAMG-RL and AgeMem is attributable to the action interface itself (filesystem vs. dedicated tools), not tool names, call formats, or documentation—ruling out superficial explanations.

3. **Dual-granularity credit assignment**: The combination of turn-level advantage estimation with token-level ratio clipping offers a principled middle ground between Turn-PPO and SAO, balancing stability and update granularity.

### Practical Implications

1. **Scaling efficiency**: CAMG-RL-9B (9B parameters) matches Qwen3.5-122B-A10B (122B total, 10B active) on external benchmarks—a dramatic efficiency gain suggesting that memory post-training can substitute for raw scale.

2. **No new tools needed**: Practitioners can improve long-horizon agent memory without designing special memory interfaces—ordinary shell access suffices when trained with RL.

3. **Cross-domain generalization**: A single policy trained jointly across four diverse domains transfers to unseen tasks in Coding and AutoResearch domains, suggesting the learned memory behavior is domain-agnostic.

## Conclusion

### Main Takeaways

- **CAMG** provides a diverse, verifiable environment suite requiring memory across context boundaries, with a common coding-agent interface (shell access + episode-persistent workspace) across all four environments.
- **CAMG-RL** learns file-based memory behavior directly from downstream task reward via fully asynchronous PPO with dual-granularity updates, without introducing special memory tools.
- The trained policy achieves state-of-the-art results on the CAMG test set and competitive external benchmark performance against models 4–13× larger.
- The success stems from eliciting and channeling pre-trained preferences for file operations rather than learning new memory interfaces from scratch.

### Limitations and Future Directions

- Memory is limited to **one episode and one user**—extending across episodes and users is natural future work.
- Training backbones only up to **9B parameters**—training larger backbones may yield further gains.
- The paper suggests that reinforcement learning on top of code pre-training can elicit a coding agent's ability to retain and reuse information beyond its active context, opening new directions for scalable agent memory research.

---

_Markdown view of https://picx.dev/p/PyRijr, served by PicX — AI-generated visual whiteboard summaries of research papers._