# ReTerm: Recovery Training for Self-Improving Terminal Agents

> ReTerm trains terminal agents to recover from their own execution failures via self-generated recovery data and online RL, boosting Terminal-Bench 2.0 Pass@1 by 14.6 points.

- **Source:** [arXiv](https://arxiv.org/abs/2610.18484)
- **Published:** 2026-10-17
- **Permalink:** https://picx.dev/p/8lO9M8
- **Whiteboard:** https://picx.dev/p/8lO9M8/image

## Summary

## Summary (Overview)

- **ReTerm** is a self-improving terminal agent trained to recover from its own execution errors, addressing a critical weakness in long-horizon terminal tasks where agents often fail to recover from command errors, faulty implementations, or misdiagnoses.
- The paper introduces **ReTerm-10K**, a high-quality dataset of recovery trajectories constructed via a three-stage pipeline: collecting failed attempts from the base model, using a stronger teacher model to generate recovery continuations from failure states, and applying four-step quality filtering.
- The authors propose **ReTerm-GRPO**, an online reinforcement learning algorithm that forks from the agent's own failure states to generate multiple independent self-recovery attempts, enabling self-recovery capability to co-evolve with task-solving capability.
- Built on Qwen3.5-9B, **ReTerm-9B achieves 34.8% Pass@1 on Terminal-Bench 2.0** (improving over base by 14.6 points), 60.6% on OpenThoughts-TBLite, and 58.4% on SWE-bench Verified, surpassing several larger 32B specialized agents.
- ReTerm demonstrates strong generalization across different agent harnesses and out-of-domain benchmarks (e.g., +20.0 points on AIME 2025, +11.0 points on SpreadsheetBench Verified), with consistently higher recovery rates across five error categories compared to standard GRPO training.

## Introduction and Theoretical Foundation

### Background and Motivation

Terminal agents interact with command-line environments to tackle complex coding tasks through long-horizon interactions. While recent advances have produced large collections of executable tasks supporting supervised fine-tuning (SFT) and reinforcement learning (RL), a critical weakness persists: **terminal agents frequently encounter errors during extended execution and struggle to recover from them**. For instance, an agent may meet a command error, misdiagnose its root cause, and continue acting in the wrong direction, ultimately leading to task failure.

### Limitations of Prior Work

The paper identifies two key limitations in existing approaches:

1. **TermiGen** (Zhu et al., 2026) injects hand-crafted errors into trajectories to generate recovery samples, but these artificial errors may exhibit a **distributional mismatch** with agents' own naturally generated errors.
2. TermiGen relies solely on SFT, leaving the potential of RL to further enhance terminal agents' recovery capability unexplored.

### Theoretical Foundation

The core insight is that recovery training should be based on **self-generated failures** rather than artificially constructed ones. Since the policy evolves during RL training, its failure distribution shifts over time—therefore, recovery capability must adapt alongside task-solving capability. This motivates the design of ReTerm-GRPO, which learns directly from failure states produced by the evolving policy.

## Methodology

### 3.1 Recovery Dataset Formulation (ReTerm-10K)

The data pipeline consists of three stages (illustrated in Figure 2 of the paper):

**Stage 1: Failure State Collection**
- The base model (Qwen3.5-9B) samples multiple attempts for each task from TMax-15K
- Each attempt executes in a task container inside a VM; at the end, the VM is paused and snapshotted
- Failed attempts' snapshots and interaction histories are retained; successful attempts are excluded

**Stage 2: Recovery Trajectory Generation**
- For each failed attempt, the snapshot is restored and DeepSeek-V4-Pro is employed as a teacher
- The teacher receives only the original task instruction and the student's interaction history (notified of failure but no hidden test feedback)
- A recovery workflow guides the teacher to: audit inherited state, reconstruct the original task contract, diagnose the failure, handle inherited state, and execute repairs

**Stage 3: Recovery Trajectory Filtering**
Four filtering dimensions ensure quality:
1. **Verifier filtering**: retains only recoveries that pass the task verifier
2. **Format filtering**: removes malformed messages or tool calls
3. **Compliance filtering**: excludes actions violating task constraints
4. **Efficiency filtering**: discards meaningless action loops

The final ReTerm-10K dataset contains ~10K high-quality recovery trajectories with relatively even domain distribution (8.4–14.2% across nine domains) and 69.6% of recoveries containing 2–8k output tokens.

**Recovery-Targeted SFT Loss:**

$$\mathcal{L}_{SFT} = -\mathbb{E}_{(x, h_f, r, y) \sim \mathcal{D}_{ReTerm}} \left[ \sum_{t \in \mathcal{A}(y)} \log \pi_\theta(y_t | x, h_f, r, y_{<t}) \right] \tag{1}$$

where $x$ is the task instruction, $h_f$ is the student's failed interaction history, $r$ is the recovery prompt, $y$ is the teacher's recovery trajectory, and $\mathcal{A}(y)$ denotes positions of teacher-generated tokens.

### 3.2 ReTerm-GRPO

**Original Task Rollouts:** For each original task, the policy generates $G$ independent rollouts from initial state $s_0$:

$$\tau^O_i \sim \pi_{\theta_{old}}(\cdot | x, s_0), \quad R^O_i = V_x(s^O_i), \quad i = 1, \ldots, G \tag{2}$$

**Recovery Task Construction:** For each original task group with at least one failed rollout, one failed rollout is sampled, preserving its history $h_f$ and final state $s_f$, forming recovery task $(x, h_f, s_f)$.

**Recovery Rollouts:**

$$\tau^R_i \sim \pi_{\theta_{old}}(\cdot | x, h_f, r, s_f), \quad R^R_i = V_x(s^R_i), \quad i = 1, \ldots, G \tag{3}$$

**Recovery Task Queue:** A queue with capacity $K$ retains recovery tasks across training steps. New tasks are inserted at the front (sampled first), and oldest tasks are removed when capacity is exceeded—reducing rollout collection overhead while keeping training focused on recent policy failures.

**Policy Optimization Objective:**

$$\mathcal{J}^c_g(\theta) = \mathbb{E} \left[ \frac{1}{G} \sum_{i=1}^{G} \frac{1}{|\mathcal{A}^c_{g,i}|} \sum_{t \in \mathcal{A}^c_{g,i}} \min \left\{ \rho^c_{g,i,t}(\theta) \hat{A}^c_{g,i}, \text{clip}\left(\rho^c_{g,i,t}(\theta), 1-\epsilon_{low}, 1+\epsilon_{high}\right) \hat{A}^c_{g,i} \right\} \right] \tag{4}$$

$$\rho^c_{g,i,t}(\theta) = \frac{\pi_\theta(y^c_{g,i,t} | h^c_{g,i,t})}{\pi_{\theta_{old}}(y^c_{g,i,t} | h^c_{g,i,t})}, \quad \hat{A}^c_{g,i} = \frac{R^c_{g,i} - \text{mean}(R^c_{g,j})}{\text{std}(R^c_{g,j}) + \delta} \tag{5}$$

**Overall Objective:**

$$\mathcal{J}_{ReTerm}(\theta) = \frac{1}{M_O + M_R} \left( \sum_{g=1}^{M_O} \mathcal{J}^O_g(\theta) + \sum_{g=1}^{M_R} \mathcal{J}^R_g(\theta) \right) \tag{6}$$

### Training Details

- 16 NVIDIA A100 (80GB) GPUs; base model Qwen3.5-9B
- ReTerm-10K split: 6K trajectories for SFT, 4K task instances for RL
- Full-parameter SFT: AdamW, global batch size 16, learning rate $5 \times 10^{-6}$
- ReTerm-GRPO: 8 original task groups + 2 recovery task groups per update; 16 rollouts per group (global batch size 160); learning rate $5 \times 10^{-7}$; max generation 8,192 tokens; 65k context window; 32 max interaction turns

## Empirical Validation / Results

### Main Results (Table 1, selected comparisons)

| Model | Model Size | OpenThoughts-TBLite | Terminal-Bench 2.0 | SWE-bench Verified |
|-------|-----------|---------------------|---------------------|---------------------|
| Qwen3.5-9B (base) | 9B | 44.2 | 20.2 | 45.2 |
| TMax-9B | 9B | 57.2 | 27.2 | 53.5 |
| OpenThinkerAgent-32B | 32B | 41.3 | 26.2 | 54.0 |
| Terminal-World-32B | 32B | – | 31.5 | – |
| **ReTerm-9B (Ours)** | **9B** | **60.6** | **34.8** | **58.4** |

ReTerm-9B achieves gains of **+16.4, +14.6, and +13.2 points** over the base model on the three benchmarks respectively, and outperforms TMax-9B at the same scale.

### Generalization Across Harnesses (Table 2)

| Model | Ours (ReAct) | Terminus-2 | mini-SWE-agent | OpenHands |
|-------|--------------|------------|----------------|-----------|
| Qwen3.5-9B | 20.2 | 22.5 | 19.1 | 15.7 |
| ReTerm-9B | 34.8 | 32.6 | 25.8 | 28.1 |
| Δ | +14.6 | +10.1 | +6.7 | +12.4 |

### Generalization Across Task Domains (Table 3)

| Model | AIME 2024 | AIME 2025 | Aider Polyglot | SpreadsheetBench-V |
|-------|-----------|-----------|----------------|---------------------|
| Qwen3.5-9B | 76.7 | 73.3 | 17.3 | 37.8 |
| ReTerm-9B | 86.7 | 93.3 | 20.4 | 48.8 |
| Δ | +10.0 | +20.0 | +3.1 | +11.0 |

### Ablation Study (Table 4)

| Method | Terminal-Bench 2.0 (Pass@1 %) |
|--------|-------------------------------|
| Base | 20.2 |
| SFT | 25.8 |
| GRPO | 28.0 |
| **ReTerm-GRPO** | **34.8** |

Each stage contributes positively; ReTerm-GRPO substantially outperforms standard GRPO, confirming the value of explicit recovery optimization.

### Error Analysis

Five error categories were identified: **Planning Error**, **Tool Use Error**, **Implementation Error**, **Verification Error**, and **Resource Management Error**. ReTerm-GRPO achieves higher recovery rates across all categories than both the base model and GRPO baseline. However, recovery remains challenging—except for tool use errors, recovery rates remain at or below 32% for other categories, indicating substantial room for improvement.

### Case Study Insights

Detailed case studies (Appendix B) reveal qualitative differences: ReTerm-GRPO-trained models demonstrate superior diagnosis (e.g., recognizing that the power law cannot independently control both KL divergences in the distribution-search task), better state auditing (e.g., detecting an existing QEMU process occupying port 2222), and more effective resource management (e.g., preserving useful setup rather than repeatedly deleting and rebuilding in the git-multibranch task).

## Theoretical and Practical Implications

### Theoretical Implications

1. **Recovery training from self-generated failures is more effective than from injected errors**: The distributional match between training failures and natural policy failures is crucial for recovery capability.
2. **RL with recovery tasks enables co-evolution**: By training on failures from the evolving policy, self-recovery capability adapts alongside task-solving capability, unlike fixed SFT datasets.
3. **Group-relative advantages work for recovery**: The GRPO framework extends naturally to recovery tasks, with verifier outcomes on multiple independent recovery attempts providing learning signals.

### Practical Implications

1. **Data pipeline design**: The three-stage pipeline (failure collection → teacher recovery → quality filtering) provides a reusable template for constructing recovery datasets.
2. **Recovery task queue**: The queue mechanism reduces rollout collection overhead and maintains relevance to the current policy's failure distribution—a practical solution for RL training when failures are sparse.
3. **Transferable capabilities**: Improvements generalize across agent harnesses and task domains, suggesting recovery training induces broadly transferable interaction skills rather than harness-specific behaviors.
4. **Remaining challenges**: Recovery rates remain low (≤32% for most error categories), highlighting the significant headroom for further research in terminal agent recovery.

## Conclusion

**ReTerm** presents a comprehensive framework for training self-improving terminal agents through recovery-oriented post-training. The key contributions are:

1. **ReTerm-10K dataset**: A high-quality recovery trajectory dataset built from the base model's own failures, with teacher-generated recoveries and four-step quality filtering.
2. **ReTerm-GRPO algorithm**: An online RL method that learns from failure states produced by the evolving policy, enabling self-recovery to co-evolve with task-solving capability.
3. **Strong empirical results**: 34.8% Pass@1 on Terminal-Bench 2.0 (+14.6 over base), 58.4% on SWE-bench Verified (+13.2), with consistent gains across harnesses and out-of-domain benchmarks.

The work demonstrates that explicitly training agents to recover from their own failures provides benefits beyond standard RL training. Future directions include improving recovery rates for harder error categories, scaling to larger models, and exploring richer recovery strategies beyond the current framework. All code, data, and models are released at https://github.com/twothsnd/ReTerm.

---

_Markdown view of https://picx.dev/p/8lO9M8, served by PicX — AI-generated visual whiteboard summaries of research papers._