Summary (Overview)

• CompactionRL is a PPO-based reinforcement learning framework that trains long-horizon agentic LLMs with trainable context compaction, jointly optimizing task execution and summary generation under a shared task-level reward.
• The method introduces token-level loss normalization and cross-trajectory Generalized Advantage Estimation (GAE) to handle variable-length compacted rollout segments, addressing issues of loss weighting bias and temporal credit assignment across compaction boundaries.
• CompactionRL achieves Pass@1 of 66.8% on SWE-bench Verified and 24.5% on Terminal-Bench 2.0 with GLM-4.5-Air (106B-A30B), representing absolute gains of 7.0 and 3.1 points over the base model with inference-time compaction.
• For GLM-4.7-Flash (30B-A3B), CompactionRL improves Pass@1 by 5.5 and 6.8 points, reaching 56.0% on SWE-bench Verified and 20.2% on Terminal-Bench 2.0.
• The method is deployed in the production RL pipeline for training the open GLM-5.2 model (750B-A40B), and ablations confirm that summary training, token-level loss, and cross-trajectory GAE are each critical to the gains.

---

Introduction and Theoretical Foundation

Background and Motivation

Long-horizon agentic LLMs (e.g., for software engineering, terminal interaction, web tasks) accumulate interaction history—tool outputs, reasoning traces, error messages—that can exceed the finite context window before task completion. While longer-context models help, scaling context length is costly and suffers from degraded utilization over long sequences. Context compaction—summarizing earlier history into a compressed state and resuming from it—offers a natural solution, but prior uses treat it as an inference-time heuristic or external memory operation rather than a trainable component.

Key Insight

In RL training, compaction plays a fundamental role: once a summary replaces the original history, it determines what information is available for all subsequent actions. Therefore, task success depends on both the execution policy and the quality of the compaction policy. The authors empirically demonstrate (Table 1) that varying only the summary agent (while fixing the execution agent) changes SWE-bench Verified accuracy from 49.0 to 55.5—a 6.5-point swing—confirming that compaction quality is performance-critical.

Why Standard RL Fails

• GRPO/group-wise methods assume fixed-size groups of complete rollouts; compaction creates variable segment counts, breaking group normalization assumptions.
• Standard PPO assumes each trajectory is processed without changing its context representation, and training compacted segments as independent samples distorts loss weighting and temporal credit assignment.

---

Methodology

Trainable Context Compaction

Rollout structure. The interaction history is represented as:

$$h_t = (s, u, z_1, \dots, z_t), \qquad z_i = (a_i, o_i)$$

where $s$ is the system prompt, $u$ is the user instruction, $a_i$ is the assistant response, and $o_i$ is the environment observation.

Compaction trigger. Compaction is triggered when the remaining context budget falls below threshold $T_{\text{comp}}$:

$$C - |h_t| < T_{\text{comp}}$$

Summary generation. The policy samples a summary conditioned on the current history plus a fixed summarization instruction:

$$S_t \sim \pi_\theta(\cdot | h_t \oplus q_{\text{sum}})$$

Context reconstruction. After summary generation, the rollout continues from:

$$\bar{h}_t = (s) \oplus u_{\text{resume}}(S_t) \oplus (z_{t-k+1}, \ldots, z_t)$$

where $k = 2$ recent steps are retained by default. The rollout is partitioned into segments $\tau = (\sigma_1, \ldots, \sigma_K)$, each being either an execution or summarization segment.

Optimization

Token-level loss. The policy objective uses a clipped PPO surrogate over token positions:

$$\mathcal{L}_\pi = -\frac{1}{|\mathcal{M}|} \sum_{(s,i) \in \mathcal{M}} \min\left(\rho_{s,i}(\theta) \widehat{A}_{s,i}, \operatorname{clip}\left(\rho_{s,i}(\theta), 1-\epsilon, 1+\epsilon\right) \widehat{A}_{s,i}\right)$$

where $\rho_{s,i}(\theta)$ is the importance sampling ratio and $\mathcal{M}$ is the set of optimized assistant-token positions. Token-level (rather than segment-level) averaging removes segment-count bias.

Cross-trajectory GAE. Local GAE within segment $\sigma_s$ is:

$$A_{s,i}^{\text{loc}} = \sum_{\ell=0}^{n_s - i} (\gamma\lambda)^\ell \delta_{s,i+\ell}, \qquad \delta_{s,i} = r_{s,i} + \gamma V_\phi(x_{s,i+1}) - V_\phi(x_{s,i})$$

With $N_{>s} = \sum_{j>s} n_j$ (tokens generated after segment $\sigma_s$), the corrected advantage is:

$$\widehat{A}_{s,i} = (\gamma\lambda)^{N_{>s}} A_{s,i}^{\text{loc}}$$

This discounts earlier segments by the number of subsequent trainable tokens, matching the true temporal distance to the final outcome in the concatenated rollout.

---

Empirical Validation / Results

Main Results (Table 2)

Ablation: Summary Training (Table 3)

Including summary responses in the RL objective consistently improves compacted-inference performance. For GLM-4.5-Air, CompactionRL with summary training achieves 66.8 vs. 64.5 without summary training on SWE-bench Verified (×4), and 24.5 vs. 21.5 on Terminal-Bench 2.0.

Ablation: Optimization Components (Table 4)

Removing token-level loss causes the larger degradation, indicating that correcting segment-count/length bias is critical.

Training Dynamics (Figure 4)

• Summary length increases when summary responses are included in the RL objective (vs. decreasing without), indicating more detailed, actionable compacted states.
• Reasoning tokens per turn increase under CompactionRL, suggesting learned compaction expands the effective reasoning budget.
• Policy entropy increases more slowly, indicating more controlled optimization.

---

Theoretical and Practical Implications

• Compaction as a learned skill: The results demonstrate that context compaction should be treated as a trainable decision process rather than a passive preprocessing step. The shared task-level reward provides a principled training signal for summary quality without hand-crafted summary metrics.
• Fixed-budget training: CompactionRL enables training under a fixed peak context budget while effectively extending the usable horizon—comparable to or better than training with a 2× larger non-compacted context window (Table 3).
• Credit assignment across boundaries: The cross-trajectory GAE correction provides a principled way to propagate rewards across compaction boundaries, addressing a fundamental issue in segment-based RL.
• Production deployment: The method's deployment in the GLM-5.2 (750B-A40B) training pipeline indicates practical industrial applicability.

---

Conclusion

CompactionRL demonstrates that incorporating context compaction into RL training—rather than treating it as an inference-time heuristic—substantially improves long-horizon agentic performance under fixed context budgets. Key contributions include token-level loss normalization to remove segment-count bias and cross-trajectory GAE for correct temporal credit assignment. The method achieves consistent gains across two model scales on SWE-bench Verified and Terminal-Bench 2.0.

Future directions and limitations:

• CompactionRL's gains do not consistently transfer to single-window evaluation (train–test mismatch when compaction is disabled).
• Cross-trajectory GAE remains an approximation to full-trajectory credit assignment; long-term effects of early summaries across multiple boundaries may not be fully captured.
• Experiments focus on code-oriented benchmarks; extending to broader agent domains (web interaction, robotics, multi-modal observations) with different reward structures remains open.