Summary (Overview)

• Novel Environment Suite: Introduces Coding Agent Memory Gym (CAMG) , a suite of four long-horizon agentic-RL environments (Shop, Coding, DeepResearch, AutoResearch) that pair native task interfaces with executable shell access and an episode-persistent workspace, enabling agents to use ordinary filesystem operations as memory.
• New RL Algorithm: Proposes CAMG-RL, which trains a single policy jointly across all four environments using fully asynchronous PPO, learning file-based memory behavior directly from downstream task reward without introducing any special memory tools.
• Strong Empirical Results: CAMG-RL-4B achieves 54.4% average success rate across the CAMG test set, outperforming training-free memory systems, learned compaction (48.0%), and tool-based memory baselines.
• External Transfer: CAMG-RL-4B matches Qwen3.5-35B-A3B and CAMG-RL-9B is competitive with Qwen3.5-122B-A10B on SWE-bench Verified and MLE-bench Lite, despite being trained on much smaller backbones.
• Key Insight: The frozen base model already prefers filesystem memory actions over dedicated memory tools (lower surprisal in nats/token), and RL training converts this pre-trained preference into complete memory chains with minimal weight movement.

Introduction and Theoretical Foundation

Background and Motivation

Language-model agents increasingly tackle long-horizon tasks involving repeated tool use and environment interaction. As interactions accumulate, information needed for later decisions can fall out of the model's active context. For example, a coding agent may need to recall why an earlier fix was rejected after that exchange has left the prompt.

Limitations of Prior Work

Recent work uses reinforcement learning to make memory control part of the policy, but relies on:

• Predefined memory tools within domain-specific training environments
• Relatively short horizons

This setup ties learned memory behavior to environment-specific interfaces that lie outside the base model's pre-training and must be learned from scratch. Even after post-training, agents struggle to use memory in long-horizon tasks.

Theoretical Foundation

The paper models each environment as a finite-horizon partially observable Markov decision process:

$$M^d = (U^d, S^d, A^d, O^d, T^d, r^d)$$

where $U^d$ is the instruction space, $S^d$ the latent-state space, $A^d$ the action space, $O^d$ the observation space, $T^d$ the state-transition function, and $r^d$ the reward function.

The policy maximizes expected task return:

$$J_d(\theta) = \mathbb{E}_{\tau \sim p_{\theta,d}}[G_d(\tau)]$$

with the vanilla policy gradient:

$$\nabla_\theta J_d(\theta) = \mathbb{E}_{\tau \sim p_{\theta,d}}\left[G_d(\tau) \sum_{t=1}^{T(\tau)} \nabla_\theta \log \pi_\theta(a_t | h_t)\right]$$

Methodology

CAMG Environment Suite

The four environments are constructed as follows:

Common Coding-Agent Interface

The action space combines native task actions with shared filesystem actions:

$$A_d = A^{\text{task}}_d \cup A^{\text{fs}}, \quad a_t \in A_d$$

The shared filesystem action is a single shell_command that can create, inspect, search, modify, and execute files. This allows the policy to preserve and revise task-relevant evidence in the workspace across context replacements.

CAMG-RL: Dual-Granularity PPO

The joint objective across all four environments is:

$$J_{\text{CAMG-RL}}(\theta) = \frac{1}{|\mathcal{D}|} \sum_{d \in \mathcal{D}} J_d(\theta), \quad |\mathcal{D}| = 4$$

Dual-granularity PPO combines:

• Turn-level advantage estimation: One reward arrives per response, providing stability
• Token-level ratio clipping: Each sampled token gets its own clipped ratio, providing fine-grained updates

This differs from existing approaches:

• Turn-PPO: Turn-level for both estimation and update (stable but coarse)
• SAO: Token-level for both (fine-grained but high-variance)

Fully Asynchronous PPO

Episode durations vary across CAMG due to different turn counts and tool execution costs. The algorithm runs rollout and learning fully asynchronously:

1. One rollout process collects trajectories from all four environments
2. Workers append completed episodes to a persistent queue
3. Learner assembles them into optimization batches
4. Updated actor weights are published for later episodes

This prevents workers from waiting for the slowest episode.

Empirical Validation / Results

CAMG Test Set Results

Table 2: CAMG-RL achieves the highest average success rate across the CAMG test set

Key Findings on Memory Behavior

1. Frozen-base action surprisal: Filesystem memory actions cost 0.076–1.193 nats/token across paired store/retrieve/revise/remove intents, while AgeMem's six dedicated tools are more surprising in every intent. Renaming the filesystem tool to an unseen name keeps surprisal within 0.013 nats/token of native actions.

2. Memory chain acquisition: CAMG-RL's complete store–retrieve–use chains rise from 8.0% of episodes (first quarter) to 20.9% (last quarter), while AgeMem attempts its memory tools in only 53 of 261,941 responses and never completes a chain.

3. Minimal policy shift: Both distance measures (KL and weight change) are small for CAMG-RL, yet it converts this movement into complete memory chains—training on filesystem memory makes effective use of pre-training.

External Benchmark Transfer

Table 3: CAMG-RL matches much larger frozen models on external benchmarks

Ablation Study

Component attribution (Figure 4):

• Full CAMG-RL: 54.4% average
• Without memory-action gradients: 21.1% (near frozen base)
• Retrained without general memory files: 41.8%
• Evaluated without general memory files: 28.1%

Intervention Study

Blanking saved memory files at the first context replacement cuts average success by 14.1 points; transplanting task-mismatched content cuts it by 11.5 points, with the largest drop in Shop—confirming saved memory content causally drives later success.

Theoretical and Practical Implications

Theoretical Contributions

1. Pre-training as memory priors: The paper demonstrates that code pre-training implicitly provides a strong prior for filesystem-based memory operations. The frozen base's surprisal analysis (0.076–1.193 nats/token for filesystem actions vs. higher for dedicated tools) provides quantitative evidence that pre-trained capabilities can be elicited through RL rather than learned from scratch.

2. Interface alignment matters: The gap between CAMG-RL and AgeMem is attributable to the action interface itself (filesystem vs. dedicated tools), not tool names, call formats, or documentation—ruling out superficial explanations.

3. Dual-granularity credit assignment: The combination of turn-level advantage estimation with token-level ratio clipping offers a principled middle ground between Turn-PPO and SAO, balancing stability and update granularity.

Practical Implications

1. Scaling efficiency: CAMG-RL-9B (9B parameters) matches Qwen3.5-122B-A10B (122B total, 10B active) on external benchmarks—a dramatic efficiency gain suggesting that memory post-training can substitute for raw scale.

2. No new tools needed: Practitioners can improve long-horizon agent memory without designing special memory interfaces—ordinary shell access suffices when trained with RL.

3. Cross-domain generalization: A single policy trained jointly across four diverse domains transfers to unseen tasks in Coding and AutoResearch domains, suggesting the learned memory behavior is domain-agnostic.

Conclusion

Main Takeaways

• CAMG provides a diverse, verifiable environment suite requiring memory across context boundaries, with a common coding-agent interface (shell access + episode-persistent workspace) across all four environments.
• CAMG-RL learns file-based memory behavior directly from downstream task reward via fully asynchronous PPO with dual-granularity updates, without introducing special memory tools.
• The trained policy achieves state-of-the-art results on the CAMG test set and competitive external benchmark performance against models 4–13× larger.
• The success stems from eliciting and channeling pre-trained preferences for file operations rather than learning new memory interfaces from scratch.

Limitations and Future Directions

• Memory is limited to one episode and one user—extending across episodes and users is natural future work.
• Training backbones only up to 9B parameters—training larger backbones may yield further gains.
• The paper suggests that reinforcement learning on top of code pre-training can elicit a coding agent's ability to retain and reuse information beyond its active context, opening new directions for scalable agent memory research.