# AGENTOPSD: RECURSIVE SELF-DISTILLATION FOR AGENTIC REINFORCEMENT LEARNING

> AgentOPSD achieves state-of-the-art agentic reinforcement learning by recursively updating success beliefs via turn-level self-distillation, outperforming GRPO without critics or extra rollouts.

- **Source:** [arXiv](https://arxiv.org/abs/2608.05987)
- **Published:** 2026-08-08
- **Permalink:** https://picx.dev/p/7WLkGI

## Summary

# AgentOPSD: Recursive Self-Distillation for Agentic Reinforcement Learning

## Summary (Overview)

- **Core Contribution**: AgentOPSD introduces a critic-free, recursive turn-level credit assignment method for long-horizon agentic reinforcement learning, transforming sparse outcome supervision into dense turn-level credit signals via Bayesian belief updates.

- **Key Innovation**: The method formalizes turn-level credit as the *revision* of a trajectory-success belief state induced by each turn, rather than treating local self-distillation gaps in isolation. This is achieved by aggregating token-level teacher-student log-probability gaps at environment-aligned turn boundaries and recursively updating a belief in log-odds space.

- **Empirical Results**: AgentOPSD achieves state-of-the-art performance across three interactive environments (ALFWorld, WebShop, Search-QA) and two model scales (Qwen2.5-3B/7B), reaching **89.1% success on ALFWorld** with Qwen2.5-7B, outperforming GRPO and strong self-distillation baselines.

- **Efficiency**: The method requires no additional rollouts, no learned critic, and only one extra teacher forward pass per trajectory, making it highly practical for real-world agentic training.

- **Robustness**: AgentOPSD demonstrates superior horizon-robustness, losing only **−0.54 success points per additional turn** on ALFWorld compared to −2.91 (GRPO) and −3.59 (RLSD), confirming its effectiveness in long-horizon credit assignment.

## Introduction and Theoretical Foundation

### Background and Motivation

Agentic post-training has become crucial for improving LLMs' ability to solve complex interactive tasks. However, a fundamental challenge exists: **many interactive environments provide verifiable rewards only after an entire trajectory terminates**, forcing training algorithms to infer the contribution of each intermediate decision from a single sparse outcome.

### The Credit Assignment Problem

Standard group-relative policy optimization methods (e.g., GRPO) construct a trajectory-level advantage and **broadcast it uniformly across the trajectory**:

$$A_{\mathrm{seq}}^{(i)} = \frac{R^{(i)} - \bar{R}}{\widehat{\sigma}_R + \epsilon_0}, \qquad \bar{R} = \frac{1}{G}\sum_{j=1}^{G} R^{(j)}$$

This uniform credit cannot distinguish pivotal decisions from routine operations. The problem worsens as the interaction horizon grows.

### Key Theoretical Insight

The authors' central insight is that **the credit of a turn should be determined not by its local signal in isolation, but by how much that signal changes the estimated probability of eventual success**. They formalize this via Bayesian belief updating:

Let $C$ denote the event that the trajectory eventually succeeds. Bayes' rule expresses the belief change as an action-side likelihood ratio:

$$\operatorname{logit}\, p(C \mid s_k, a_k) - \operatorname{logit}\, p(C \mid s_k) = \log \frac{p(a_k \mid s_k, C)}{p(a_k \mid s_k, \neg C)}$$

where $\operatorname{logit}(u) = \log \frac{u}{1-u}$.

### From Bayes Factor to Self-Teacher Contrast

Since outcome-conditional behavioral distributions are unavailable, AgentOPSD estimates this evidence using a **per-turn self-distillation contrast** between a privileged, success-associated self-teacher and the standard student policy. For token $y_{k,t}$:

$$\delta_{k,t} = \log \pi_\theta(y_{k,t} \mid h_{k,t}^{+}) - \log \pi_\theta(y_{k,t} \mid h_{k,t})$$

where $h_{k,t}^{+} = (s_k, c^{+}, y_{k,<t})$ includes a training-only retrieved skill $c^{+}$.

Summing over tokens gives turn-level evidence:

$$e_k = \sum_{t=1}^{L_k} \delta_{k,t} = \log \frac{\pi_\theta(a_k \mid s_k, c^{+})}{\pi_\theta(a_k \mid s_k)}$$

## Methodology

### 1. Turn-Level Evidence Aggregation

AgentOPSD aggregates token-level teacher-student log-probability gaps at turn boundaries, aligning with environment transitions where the environment responds to complete actions rather than individual tokens.

### 2. Recursive Belief Update

The method maintains a decaying evidence accumulator and measures each turn by its marginal support revision:

$$\begin{aligned}
B_0 &= \operatorname{clip}(\bar{R}, \epsilon_0, 1-\epsilon_0), \quad c_0 = 0, \\
c_k &= \gamma c_{k-1} + e_k, \\
\ell_k &= \operatorname{logit}(B_0) + c_k = \operatorname{logit}(B_0) + \sum_{j=1}^{k} \gamma^{k-j} e_j,
\end{aligned}$$

with $B_k = \sigma(\ell_k)$ and $\sigma(u) = (1+e^{-u})^{-1}$. Here $\bar{R} = S/G$ is the fraction of successful trajectories in the group, and $\gamma \in (0,1]$ is a decay factor.

The importance of turn $k$ is its **marginal support revision**:

$$\Delta B_k = B_k - B_{k-1} = \sigma(\ell_k) - \sigma(\ell_{k-1})$$

$$\Delta B_k \approx B_{k-1}(1-B_{k-1})\big(e_k - (1-\gamma)c_{k-1}\big)$$

This formulation reveals that evidence has greatest effect under uncertainty (when $B_{k-1}(1-B_{k-1})$ is maximal at $B = 1/2$) and is suppressed once support saturates.

### 3. Outcome-Aligned Recursive Credit

The revision is aligned with the terminal outcome:

$$q_k = \operatorname{sign}(A_{\mathrm{seq}})\Delta B_k$$

The magnitude $|\Delta B_k|$ measures how much support the turn revises, while the sign records whether that revision agrees with the verifier's outcome signal.

### 4. Bounded Advantage Reshaping

For trajectory $i$, the raw credits are normalized and converted to a bounded multiplier:

$$\begin{aligned}
\mu_q^{(i)} &= K_i^{-1}\sum_{j=1}^{K_i} q_j^{(i)}, \quad \sigma_q^{(i)} = \sqrt{K_i^{-1}\sum_{j=1}^{K_i}\left(q_j^{(i)} - \mu_q^{(i)}\right)^2}, \\
z_k^{(i)} &= \frac{q_k^{(i)} - \mu_q^{(i)}}{\sigma_q^{(i)} + \epsilon_0}, \\
w_k^{(i)} &= \operatorname{clip}\left(1 + b z_k^{(i)}, 1-b, 1+b\right), \\
\widetilde{A}_k^{(i)} &= A_{\mathrm{seq}}^{(i)}\left[(1-\lambda) + \lambda w_k^{(i)}\right], \quad b \in (0,1), \quad \lambda \in [0,1].
\end{aligned}$$

The final loss is a clipped GRPO objective with reshaped advantages:

$$\mathcal{L}_{\text{AgentOPSD}}(\theta) = -\frac{1}{G}\sum_{i=1}^{G}\frac{1}{\sum_t M_{i,t}}\sum_t M_{i,t}\min\left(r_{i,t}\widetilde{A}_{\kappa_i(t)}^{(i)}, \operatorname{clip}(r_{i,t}, 1-\varepsilon, 1+\varepsilon)\widetilde{A}_{\kappa_i(t)}^{(i)}\right) + \beta\mathcal{L}_{\mathrm{KL}}$$

### Theoretical Properties

The reshaping satisfies several desirable properties:

- **Proposition 1 (Boundedness)**: $|\widetilde{A}_k - A^{(i)}| \leq \lambda b |A^{(i)}|$
- **Proposition 2 (Sign Preservation)**: $\operatorname{sign}(\widetilde{A}_k) = \operatorname{sign}(A^{(i)})$ for every turn
- **Proposition 3 (Recovery of GRPO)**: At $\lambda = 0$, AgentOPSD reduces exactly to GRPO
- **Proposition 4 (First-Order Decomposition)**: $\Delta B_k = B_{k-1}(1-B_{k-1})\Delta\ell_k + O((\Delta\ell_k)^2)$
- **Proposition 5 (Exact Budget)**: $\sum_k \Delta B_k = B_K - B_0$ (telescoping property)
- **Proposition 6 (Non-Identifiability)**: Per-turn credit is not identifiable from trajectory return alone
- **Proposition 7**: $B_0$ as the group success-rate estimate (MLE of Binomial success probability)

## Empirical Validation / Results

### Experimental Setup

- **Environments**: ALFWorld (6 household task categories), WebShop (128 fixed validation tasks), Search-QA (7 QA datasets with NQ and HotpotQA in-domain)
- **Models**: Qwen2.5-3B-Instruct and Qwen2.5-7B-Instruct trained on 8×H800 GPUs
- **Baselines**: Vanilla, Skill-Prompt, GRPO, Skill-GRPO, OPSD, GRPO+OPSD, Skill-SD, RLSD, SDAR, StepOPSD

### Main Results

**Table 1: Performance on ALFWorld, Search-QA and WebShop** (key results)

| Method | ALFWorld Avg (3B) | ALFWorld Avg (7B) | Search-QA Avg (3B) | Search-QA Avg (7B) | WebShop Score (3B) | WebShop Score (7B) |
|--------|-------------------|-------------------|--------------------|--------------------|--------------------|--------------------|
| GRPO   | 75.0 | 81.2 | 36.4 | 42.0 | 79.8 | 80.9 |
| SDAR   | 84.4 | 85.9 | 43.4 | 49.0 | 85.0 | 89.4 |
| StepOPSD | 73.4 | 88.4 | 43.7 | 48.2 | 82.4 | 87.2 |
| **AgentOPSD** | **84.4** | **89.1** | **46.7** | **49.2** | **90.4** | **90.2** |

Key findings:

1. **Controlled-information comparison**: AgentOPSD outperforms GRPO+OPSD, Skill-SD, and RLSD on all eight aggregate comparisons across both model scales, and exceeds SDAR on six of eight.

2. **Horizon robustness** (Figure 1b): Uniform-credit methods degrade fastest (−3.59 for RLSD, −2.91 for GRPO points per turn), while AgentOPSD is flattest at −0.54.

### Mechanism Ablation (Table 2)

| Component | Ablation | ALFWorld Success (%) |
|-----------|----------|---------------------|
| **AgentOPSD (full)** | turn-level, bounded, λ=0.5 | **89.1** |
| Turn-level granularity | per-token accumulation | 85.9 |
| Recursive state revision | raw local gap $e_k$ instead of $\Delta B_k$ | 82.8 |
| Signed direction | magnitude $\|\Delta B_k\|$ only | 80.5 |
| State prior anchor | drop empirical-rate initialization | 78.9 |

The ablations isolate three roles:
- **Belief revision** localizes credit (89.1 → 82.8 without recursion)
- **Signed direction** aligns credit with final outcome (→ 80.5 without sign)
- **Prior anchoring** stabilizes the reference point (→ 78.9 without prior)

### Hyperparameter Sensitivity

- **Reshaping weight λ**: λ=0.5 is optimal; smaller values reduce performance (89.1 → 83.6–85.9)
- **Evidence decay γ**: Sweeping γ∈{1.0, 0.95, 0.9, 0.8} moves results within a few points (82.0–87.5), showing robustness
- **Policy clipping ε_high**: Largely unaffected (88.3 at both 0.2 and 0.28)

## Theoretical and Practical Implications

### Theoretical Contributions

1. **Formalization of turn-level credit as belief revision**: The paper provides a principled Bayesian framework connecting per-turn evidence to sequential credit, revealing that an isolated self-distillation gap is not, by itself, sequential credit.

2. **Unified view of credit assignment**: The belief state plays the role of GAE's value baseline, and its per-turn revision plays the role of the TD signal—but without a learned value network cost.

3. **Non-identifiability result** (Proposition 6): Demonstrates that per-turn credit cannot be inferred from trajectory return alone, justifying the need for additional per-turn signals.

### Practical Implications

1. **Efficiency**: AgentOPSD requires only one extra teacher forward pass per trajectory, making it practical for large-scale training (no critic, no additional rollouts).

2. **Versatility**: Works across diverse environments (embodied text worlds, web shopping, RAG-based QA) and model scales (3B and 7B) with a single shared hyperparameter configuration.

3. **Robustness**: The bounded reshaping (Propositions 1–3) guarantees stable training by preserving GRPO's sign and trust-region properties.

4. **Complementarity with existing methods**: The approach is orthogonal to reward-side credit assignment (e.g., GiGPO), suggesting potential for combination.

## Conclusion

### Main Takeaways

AgentOPSD addresses the fundamental challenge of credit assignment in long-horizon agentic reinforcement learning by:
1. Aggregating token-level self-distillation gaps at environment-aligned turn boundaries
2. Recursively updating a trajectory-success belief in log-odds space
3. Using belief revisions to redistribute trajectory-level advantages across turns

The method consistently outperforms GRPO and strong self-distillation baselines across three environments and two model scales, with the most significant gains in long-horizon tasks.

### Future Directions

The authors' results suggest that recursive belief updating provides a simple and effective approach to restoring fine-grained temporal credit in critic-free agentic RL. Future work could explore:
- Combining AgentOPSD with reward-side credit assignment methods (e.g., GiGPO)
- Extending to other privileged information sources beyond retrieved skills
- Investigating adaptive decay factors $\gamma$ rather than fixed settings
- Scaling to even longer horizons and more complex interactive environments

---

_Markdown view of https://picx.dev/p/7WLkGI, served by PicX — AI-generated visual whiteboard summaries of research papers._