AgentOPSD: Recursive Self-Distillation for Agentic Reinforcement Learning

Summary (Overview)

  • Core Contribution: AgentOPSD introduces a critic-free, recursive turn-level credit assignment method for long-horizon agentic reinforcement learning, transforming sparse outcome supervision into dense turn-level credit signals via Bayesian belief updates.

  • Key Innovation: The method formalizes turn-level credit as the revision of a trajectory-success belief state induced by each turn, rather than treating local self-distillation gaps in isolation. This is achieved by aggregating token-level teacher-student log-probability gaps at environment-aligned turn boundaries and recursively updating a belief in log-odds space.

  • Empirical Results: AgentOPSD achieves state-of-the-art performance across three interactive environments (ALFWorld, WebShop, Search-QA) and two model scales (Qwen2.5-3B/7B), reaching 89.1% success on ALFWorld with Qwen2.5-7B, outperforming GRPO and strong self-distillation baselines.

  • Efficiency: The method requires no additional rollouts, no learned critic, and only one extra teacher forward pass per trajectory, making it highly practical for real-world agentic training.

  • Robustness: AgentOPSD demonstrates superior horizon-robustness, losing only −0.54 success points per additional turn on ALFWorld compared to −2.91 (GRPO) and −3.59 (RLSD), confirming its effectiveness in long-horizon credit assignment.

Introduction and Theoretical Foundation

Background and Motivation

Agentic post-training has become crucial for improving LLMs' ability to solve complex interactive tasks. However, a fundamental challenge exists: many interactive environments provide verifiable rewards only after an entire trajectory terminates, forcing training algorithms to infer the contribution of each intermediate decision from a single sparse outcome.

The Credit Assignment Problem

Standard group-relative policy optimization methods (e.g., GRPO) construct a trajectory-level advantage and broadcast it uniformly across the trajectory:

Aseq(i)=R(i)−Rˉσ^R+ϵ0,Rˉ=1G∑j=1GR(j)A_{\mathrm{seq}}^{(i)} = \frac{R^{(i)} - \bar{R}}{\widehat{\sigma}_R + \epsilon_0}, \qquad \bar{R} = \frac{1}{G}\sum_{j=1}^{G} R^{(j)}

This uniform credit cannot distinguish pivotal decisions from routine operations. The problem worsens as the interaction horizon grows.

Key Theoretical Insight

The authors' central insight is that the credit of a turn should be determined not by its local signal in isolation, but by how much that signal changes the estimated probability of eventual success. They formalize this via Bayesian belief updating:

Let CC denote the event that the trajectory eventually succeeds. Bayes' rule expresses the belief change as an action-side likelihood ratio:

logit⁡ p(C∣sk,ak)−logit⁡ p(C∣sk)=log⁡p(ak∣sk,C)p(ak∣sk,¬C)\operatorname{logit}\, p(C \mid s_k, a_k) - \operatorname{logit}\, p(C \mid s_k) = \log \frac{p(a_k \mid s_k, C)}{p(a_k \mid s_k, \neg C)}

where logit⁡(u)=log⁡u1−u\operatorname{logit}(u) = \log \frac{u}{1-u}.

From Bayes Factor to Self-Teacher Contrast

Since outcome-conditional behavioral distributions are unavailable, AgentOPSD estimates this evidence using a per-turn self-distillation contrast between a privileged, success-associated self-teacher and the standard student policy. For token yk,ty_{k,t}:

δk,t=log⁡πθ(yk,t∣hk,t+)−log⁡πθ(yk,t∣hk,t)\delta_{k,t} = \log \pi_\theta(y_{k,t} \mid h_{k,t}^{+}) - \log \pi_\theta(y_{k,t} \mid h_{k,t})

where hk,t+=(sk,c+,yk,<t)h_{k,t}^{+} = (s_k, c^{+}, y_{k,<t}) includes a training-only retrieved skill c+c^{+}.

Summing over tokens gives turn-level evidence:

ek=∑t=1Lkδk,t=log⁡πθ(ak∣sk,c+)πθ(ak∣sk)e_k = \sum_{t=1}^{L_k} \delta_{k,t} = \log \frac{\pi_\theta(a_k \mid s_k, c^{+})}{\pi_\theta(a_k \mid s_k)}

Methodology

1. Turn-Level Evidence Aggregation

AgentOPSD aggregates token-level teacher-student log-probability gaps at turn boundaries, aligning with environment transitions where the environment responds to complete actions rather than individual tokens.

2. Recursive Belief Update

The method maintains a decaying evidence accumulator and measures each turn by its marginal support revision:

B_0 &= \operatorname{clip}(\bar{R}, \epsilon_0, 1-\epsilon_0), \quad c_0 = 0, \\ c_k &= \gamma c_{k-1} + e_k, \\ \ell_k &= \operatorname{logit}(B_0) + c_k = \operatorname{logit}(B_0) + \sum_{j=1}^{k} \gamma^{k-j} e_j, \end{aligned}$$ with $B_k = \sigma(\ell_k)$ and $\sigma(u) = (1+e^{-u})^{-1}$. Here $\bar{R} = S/G$ is the fraction of successful trajectories in the group, and $\gamma \in (0,1]$ is a decay factor. The importance of turn $k$ is its **marginal support revision**:

\Delta B_k = B_k - B_{k-1} = \sigma(\ell_k) - \sigma(\ell_{k-1})

\Delta B_k \approx B_{k-1}(1-B_{k-1})\big(e_k - (1-\gamma)c_{k-1}\big)

This formulation reveals that evidence has greatest effect under uncertainty (when $B_{k-1}(1-B_{k-1})$ is maximal at $B = 1/2$) and is suppressed once support saturates. ### 3. Outcome-Aligned Recursive Credit The revision is aligned with the terminal outcome:

q_k = \operatorname{sign}(A_{\mathrm{seq}})\Delta B_k

The magnitude $|\Delta B_k|$ measures how much support the turn revises, while the sign records whether that revision agrees with the verifier's outcome signal. ### 4. Bounded Advantage Reshaping For trajectory $i$, the raw credits are normalized and converted to a bounded multiplier: $$\begin{aligned} \mu_q^{(i)} &= K_i^{-1}\sum_{j=1}^{K_i} q_j^{(i)}, \quad \sigma_q^{(i)} = \sqrt{K_i^{-1}\sum_{j=1}^{K_i}\left(q_j^{(i)} - \mu_q^{(i)}\right)^2}, \\ z_k^{(i)} &= \frac{q_k^{(i)} - \mu_q^{(i)}}{\sigma_q^{(i)} + \epsilon_0}, \\ w_k^{(i)} &= \operatorname{clip}\left(1 + b z_k^{(i)}, 1-b, 1+b\right), \\ \widetilde{A}_k^{(i)} &= A_{\mathrm{seq}}^{(i)}\left[(1-\lambda) + \lambda w_k^{(i)}\right], \quad b \in (0,1), \quad \lambda \in [0,1]. \end{aligned}$$ The final loss is a clipped GRPO objective with reshaped advantages:

\mathcal{L}{\text{AgentOPSD}}(\theta) = -\frac{1}{G}\sum{i=1}^{G}\frac{1}{\sum_t M_{i,t}}\sum_t M_{i,t}\min\left(r_{i,t}\widetilde{A}{\kappa_i(t)}^{(i)}, \operatorname{clip}(r{i,t}, 1-\varepsilon, 1+\varepsilon)\widetilde{A}{\kappa_i(t)}^{(i)}\right) + \beta\mathcal{L}{\mathrm{KL}}

### Theoretical Properties The reshaping satisfies several desirable properties: - **Proposition 1 (Boundedness)**: $|\widetilde{A}_k - A^{(i)}| \leq \lambda b |A^{(i)}|$ - **Proposition 2 (Sign Preservation)**: $\operatorname{sign}(\widetilde{A}_k) = \operatorname{sign}(A^{(i)})$ for every turn - **Proposition 3 (Recovery of GRPO)**: At $\lambda = 0$, AgentOPSD reduces exactly to GRPO - **Proposition 4 (First-Order Decomposition)**: $\Delta B_k = B_{k-1}(1-B_{k-1})\Delta\ell_k + O((\Delta\ell_k)^2)$ - **Proposition 5 (Exact Budget)**: $\sum_k \Delta B_k = B_K - B_0$ (telescoping property) - **Proposition 6 (Non-Identifiability)**: Per-turn credit is not identifiable from trajectory return alone - **Proposition 7**: $B_0$ as the group success-rate estimate (MLE of Binomial success probability) ## Empirical Validation / Results ### Experimental Setup - **Environments**: ALFWorld (6 household task categories), WebShop (128 fixed validation tasks), Search-QA (7 QA datasets with NQ and HotpotQA in-domain) - **Models**: Qwen2.5-3B-Instruct and Qwen2.5-7B-Instruct trained on 8×H800 GPUs - **Baselines**: Vanilla, Skill-Prompt, GRPO, Skill-GRPO, OPSD, GRPO+OPSD, Skill-SD, RLSD, SDAR, StepOPSD ### Main Results **Table 1: Performance on ALFWorld, Search-QA and WebShop** (key results) | Method | ALFWorld Avg (3B) | ALFWorld Avg (7B) | Search-QA Avg (3B) | Search-QA Avg (7B) | WebShop Score (3B) | WebShop Score (7B) | |--------|-------------------|-------------------|--------------------|--------------------|--------------------|--------------------| | GRPO | 75.0 | 81.2 | 36.4 | 42.0 | 79.8 | 80.9 | | SDAR | 84.4 | 85.9 | 43.4 | 49.0 | 85.0 | 89.4 | | StepOPSD | 73.4 | 88.4 | 43.7 | 48.2 | 82.4 | 87.2 | | **AgentOPSD** | **84.4** | **89.1** | **46.7** | **49.2** | **90.4** | **90.2** | Key findings: 1. **Controlled-information comparison**: AgentOPSD outperforms GRPO+OPSD, Skill-SD, and RLSD on all eight aggregate comparisons across both model scales, and exceeds SDAR on six of eight. 2. **Horizon robustness** (Figure 1b): Uniform-credit methods degrade fastest (−3.59 for RLSD, −2.91 for GRPO points per turn), while AgentOPSD is flattest at −0.54. ### Mechanism Ablation (Table 2) | Component | Ablation | ALFWorld Success (%) | |-----------|----------|---------------------| | **AgentOPSD (full)** | turn-level, bounded, λ=0.5 | **89.1** | | Turn-level granularity | per-token accumulation | 85.9 | | Recursive state revision | raw local gap $e_k$ instead of $\Delta B_k$ | 82.8 | | Signed direction | magnitude $\|\Delta B_k\|$ only | 80.5 | | State prior anchor | drop empirical-rate initialization | 78.9 | The ablations isolate three roles: - **Belief revision** localizes credit (89.1 → 82.8 without recursion) - **Signed direction** aligns credit with final outcome (→ 80.5 without sign) - **Prior anchoring** stabilizes the reference point (→ 78.9 without prior) ### Hyperparameter Sensitivity - **Reshaping weight λ**: λ=0.5 is optimal; smaller values reduce performance (89.1 → 83.6–85.9) - **Evidence decay γ**: Sweeping γ∈{1.0, 0.95, 0.9, 0.8} moves results within a few points (82.0–87.5), showing robustness - **Policy clipping ε_high**: Largely unaffected (88.3 at both 0.2 and 0.28) ## Theoretical and Practical Implications ### Theoretical Contributions 1. **Formalization of turn-level credit as belief revision**: The paper provides a principled Bayesian framework connecting per-turn evidence to sequential credit, revealing that an isolated self-distillation gap is not, by itself, sequential credit. 2. **Unified view of credit assignment**: The belief state plays the role of GAE's value baseline, and its per-turn revision plays the role of the TD signal—but without a learned value network cost. 3. **Non-identifiability result** (Proposition 6): Demonstrates that per-turn credit cannot be inferred from trajectory return alone, justifying the need for additional per-turn signals. ### Practical Implications 1. **Efficiency**: AgentOPSD requires only one extra teacher forward pass per trajectory, making it practical for large-scale training (no critic, no additional rollouts). 2. **Versatility**: Works across diverse environments (embodied text worlds, web shopping, RAG-based QA) and model scales (3B and 7B) with a single shared hyperparameter configuration. 3. **Robustness**: The bounded reshaping (Propositions 1–3) guarantees stable training by preserving GRPO's sign and trust-region properties. 4. **Complementarity with existing methods**: The approach is orthogonal to reward-side credit assignment (e.g., GiGPO), suggesting potential for combination. ## Conclusion ### Main Takeaways AgentOPSD addresses the fundamental challenge of credit assignment in long-horizon agentic reinforcement learning by: 1. Aggregating token-level self-distillation gaps at environment-aligned turn boundaries 2. Recursively updating a trajectory-success belief in log-odds space 3. Using belief revisions to redistribute trajectory-level advantages across turns The method consistently outperforms GRPO and strong self-distillation baselines across three environments and two model scales, with the most significant gains in long-horizon tasks. ### Future Directions The authors' results suggest that recursive belief updating provides a simple and effective approach to restoring fine-grained temporal credit in critic-free agentic RL. Future work could explore: - Combining AgentOPSD with reward-side credit assignment methods (e.g., GiGPO) - Extending to other privileged information sources beyond retrieved skills - Investigating adaptive decay factors $\gamma$ rather than fixed settings - Scaling to even longer horizons and more complex interactive environments

Related papers