# CompactionRL: Reinforcement Learning with Context Compaction for Long-Horizon Agents

> CompactionRL trains long-horizon agentic LLMs by jointly optimizing context summarization and task execution under a shared reward, achieving 66.8% Pass@1 on SWE-bench Verified.

- **Source:** [arXiv](https://arxiv.org/abs/2607.05378)
- **Published:** 2026-08-29
- **Permalink:** https://picx.dev/p/ro6Vx2
- **Whiteboard:** https://picx.dev/p/ro6Vx2/image

## Summary

## Summary (Overview)

- **CompactionRL** is a PPO-based reinforcement learning framework that trains long-horizon agentic LLMs with *trainable context compaction*, jointly optimizing task execution and summary generation under a shared task-level reward.
- The method introduces **token-level loss normalization** and **cross-trajectory Generalized Advantage Estimation (GAE)** to handle variable-length compacted rollout segments, addressing issues of loss weighting bias and temporal credit assignment across compaction boundaries.
- CompactionRL achieves **Pass@1 of 66.8% on SWE-bench Verified** and **24.5% on Terminal-Bench 2.0** with GLM-4.5-Air (106B-A30B), representing absolute gains of **7.0 and 3.1 points** over the base model with inference-time compaction.
- For GLM-4.7-Flash (30B-A3B), CompactionRL improves Pass@1 by **5.5 and 6.8 points**, reaching **56.0% on SWE-bench Verified** and **20.2% on Terminal-Bench 2.0**.
- The method is deployed in the production RL pipeline for training the open **GLM-5.2 model (750B-A40B)**, and ablations confirm that summary training, token-level loss, and cross-trajectory GAE are each critical to the gains.

---

## Introduction and Theoretical Foundation

### Background and Motivation

Long-horizon agentic LLMs (e.g., for software engineering, terminal interaction, web tasks) accumulate interaction history—tool outputs, reasoning traces, error messages—that can exceed the finite context window before task completion. While longer-context models help, scaling context length is costly and suffers from degraded utilization over long sequences. **Context compaction**—summarizing earlier history into a compressed state and resuming from it—offers a natural solution, but prior uses treat it as an inference-time heuristic or external memory operation rather than a trainable component.

### Key Insight

In RL training, compaction plays a fundamental role: once a summary replaces the original history, it determines what information is available for all subsequent actions. Therefore, **task success depends on both the execution policy and the quality of the compaction policy**. The authors empirically demonstrate (Table 1) that varying only the summary agent (while fixing the execution agent) changes SWE-bench Verified accuracy from 49.0 to 55.5—a 6.5-point swing—confirming that compaction quality is performance-critical.

### Why Standard RL Fails

- **GRPO/group-wise methods** assume fixed-size groups of complete rollouts; compaction creates variable segment counts, breaking group normalization assumptions.
- **Standard PPO** assumes each trajectory is processed without changing its context representation, and training compacted segments as independent samples distorts loss weighting and temporal credit assignment.

---

## Methodology

### Trainable Context Compaction

**Rollout structure.** The interaction history is represented as:

$$h_t = (s, u, z_1, \dots, z_t), \qquad z_i = (a_i, o_i)$$

where $s$ is the system prompt, $u$ is the user instruction, $a_i$ is the assistant response, and $o_i$ is the environment observation.

**Compaction trigger.** Compaction is triggered when the remaining context budget falls below threshold $T_{\text{comp}}$:

$$C - |h_t| < T_{\text{comp}}$$

**Summary generation.** The policy samples a summary conditioned on the current history plus a fixed summarization instruction:

$$S_t \sim \pi_\theta(\cdot | h_t \oplus q_{\text{sum}})$$

**Context reconstruction.** After summary generation, the rollout continues from:

$$\bar{h}_t = (s) \oplus u_{\text{resume}}(S_t) \oplus (z_{t-k+1}, \ldots, z_t)$$

where $k = 2$ recent steps are retained by default. The rollout is partitioned into segments $\tau = (\sigma_1, \ldots, \sigma_K)$, each being either an execution or summarization segment.

### Optimization

**Token-level loss.** The policy objective uses a clipped PPO surrogate over token positions:

$$\mathcal{L}_\pi = -\frac{1}{|\mathcal{M}|} \sum_{(s,i) \in \mathcal{M}} \min\left(\rho_{s,i}(\theta) \widehat{A}_{s,i}, \operatorname{clip}\left(\rho_{s,i}(\theta), 1-\epsilon, 1+\epsilon\right) \widehat{A}_{s,i}\right)$$

where $\rho_{s,i}(\theta)$ is the importance sampling ratio and $\mathcal{M}$ is the set of optimized assistant-token positions. Token-level (rather than segment-level) averaging removes segment-count bias.

**Cross-trajectory GAE.** Local GAE within segment $\sigma_s$ is:

$$A_{s,i}^{\text{loc}} = \sum_{\ell=0}^{n_s - i} (\gamma\lambda)^\ell \delta_{s,i+\ell}, \qquad \delta_{s,i} = r_{s,i} + \gamma V_\phi(x_{s,i+1}) - V_\phi(x_{s,i})$$

With $N_{>s} = \sum_{j>s} n_j$ (tokens generated after segment $\sigma_s$), the corrected advantage is:

$$\widehat{A}_{s,i} = (\gamma\lambda)^{N_{>s}} A_{s,i}^{\text{loc}}$$

This discounts earlier segments by the number of subsequent trainable tokens, matching the true temporal distance to the final outcome in the concatenated rollout.

---

## Empirical Validation / Results

### Main Results (Table 2)

| Model | Peak Len. | SWE-bench Verified (Single ×1) | SWE-bench Verified (Compacted ×4) | Terminal-Bench 2.0 (Single ×1) | Terminal-Bench 2.0 (Compacted ×4) |
|---|---|---|---|---|---|
| GLM-4.7-Flash (30B-A3B) | 64k | 47.5 | 50.5 | 14.6 | 13.4 |
| + RL (w/o compaction) | 64k | 50.0 | 48.0 | 16.9 | 12.4 |
| **+ CompactionRL (ours)** | 64k | 43.7 | **56.0** | 16.9 | **20.2** |
| GLM-4.5-Air (106B-A30B) | 80k | 57.8 | 59.8 | 17.9 | 21.4 |
| + RL (w/o compaction) | 80k | 58.3 | 62.5 | 20.2 | 23.6 |
| **+ CompactionRL (ours)** | 80k | 57.3 | **66.8** | 21.4 | **24.5** |

### Ablation: Summary Training (Table 3)

Including summary responses in the RL objective consistently improves compacted-inference performance. For GLM-4.5-Air, CompactionRL with summary training achieves 66.8 vs. 64.5 without summary training on SWE-bench Verified (×4), and 24.5 vs. 21.5 on Terminal-Bench 2.0.

### Ablation: Optimization Components (Table 4)

| System | SWE-bench Verified | Terminal-Bench 2.0 |
|---|---|---|
| GLM-4.5-Air (base) | 59.8 | 21.4 |
| + CompactionRL | 66.8 | 24.5 |
| – w/o token-level loss | 60.0 | 21.3 |
| – w/o cross-trajectory GAE | 63.0 | 22.5 |

Removing token-level loss causes the larger degradation, indicating that correcting segment-count/length bias is critical.

### Training Dynamics (Figure 4)

- **Summary length increases** when summary responses are included in the RL objective (vs. decreasing without), indicating more detailed, actionable compacted states.
- **Reasoning tokens per turn increase** under CompactionRL, suggesting learned compaction expands the effective reasoning budget.
- **Policy entropy increases more slowly**, indicating more controlled optimization.

---

## Theoretical and Practical Implications

- **Compaction as a learned skill**: The results demonstrate that context compaction should be treated as a trainable decision process rather than a passive preprocessing step. The shared task-level reward provides a principled training signal for summary quality without hand-crafted summary metrics.
- **Fixed-budget training**: CompactionRL enables training under a fixed peak context budget while effectively extending the usable horizon—comparable to or better than training with a 2× larger non-compacted context window (Table 3).
- **Credit assignment across boundaries**: The cross-trajectory GAE correction provides a principled way to propagate rewards across compaction boundaries, addressing a fundamental issue in segment-based RL.
- **Production deployment**: The method's deployment in the GLM-5.2 (750B-A40B) training pipeline indicates practical industrial applicability.

---

## Conclusion

CompactionRL demonstrates that incorporating context compaction into RL training—rather than treating it as an inference-time heuristic—substantially improves long-horizon agentic performance under fixed context budgets. Key contributions include token-level loss normalization to remove segment-count bias and cross-trajectory GAE for correct temporal credit assignment. The method achieves consistent gains across two model scales on SWE-bench Verified and Terminal-Bench 2.0.

**Future directions and limitations:**
- CompactionRL's gains do not consistently transfer to single-window evaluation (train–test mismatch when compaction is disabled).
- Cross-trajectory GAE remains an approximation to full-trajectory credit assignment; long-term effects of early summaries across multiple boundaries may not be fully captured.
- Experiments focus on code-oriented benchmarks; extending to broader agent domains (web interaction, robotics, multi-modal observations) with different reward structures remains open.

---

_Markdown view of https://picx.dev/p/ro6Vx2, served by PicX — AI-generated visual whiteboard summaries of research papers._