# SEED: Self-Evolving On-Policy Distillation for Agentic Reinforcement Learning

> SEED extracts hindsight skills from on-policy trajectories for dense token-level supervision, outperforming outcome-only RL and static distillation across agentic tasks.

- **Source:** [arXiv](https://arxiv.org/abs/2607.14777)
- **Published:** 2026-07-18
- **Permalink:** https://picx.dev/p/uhzsl4
- **Whiteboard:** https://picx.dev/p/uhzsl4/image

## Summary

## Summary (Overview)

- **Problem:** Outcome-based reinforcement learning for LLM agents provides only sparse, trajectory-level rewards, offering limited guidance for intermediate decisions and token-level credit assignment.
- **Proposed Method:** SEED (Self-Evolving On-Policy Distillation) extracts natural-language **hindsight skills** from completed on-policy trajectories and distills their behavioral effect back into the policy via dense token-level supervision.
- **Key Mechanism:** The policy simultaneously acts as a rollout actor and a trajectory analyzer. Skills are generated on-policy and used to re-score the same sampled actions under skill-augmented versus ordinary contexts, producing a token-level distillation signal.
- **Core Advantages:** The self-evolving loop keeps hindsight supervision aligned with the evolving policy; dense token-level signals fill the granularity gap of sparse rewards; no skill prompts or external memory are needed at inference time.
- **Empirical Results:** SEED consistently outperforms outcome-only RL (GRPO) and static skill-distillation baselines across embodied interaction (ALFWorld), web navigation (WebShop), search-based QA, and vision-based tasks, with improved sample efficiency and cross-domain generalization.

## Introduction and Theoretical Foundation

LLM-based agents must make sequential decisions over long horizons, often receiving feedback only at the end of an episode. Standard reinforcement learning (RL) optimizes against such trajectory-level outcomes (e.g., success/failure), but this coarse supervision cannot pinpoint which intermediate actions, observations, or tool calls should be reinforced or corrected—a **supervision gap** between episode-level outcomes and token-level policy learning.

A key insight is that completed trajectories reveal **hindsight information** unavailable during online decision making: reusable strategies, decisive observations, failure causes, and workflows. While prior work has used verbal reflection, episodic memory, or experience summaries, such hindsight is often static or used only as inference-time context. For practical agentic RL, hindsight supervision should be:

1. **On-policy** – derived from the current policy’s own trajectories to match the state-action distribution.
2. **Dense** – providing per-token credit assignment rather than a single trajectory-level signal.
3. **Self-evolving** – adapting as the policy improves and encounters new behaviors and failure modes.

SEED addresses these requirements by converting completed on-policy trajectories into natural-language **hindsight skills** (e.g., “When placing an object, first locate the target, take it to the destination, and confirm placement”) and then using those skills to construct a dense, token-level on-policy distillation (OPD) signal that is jointly optimized with outcome-based RL (specifically GRPO, Shao et al., 2024).

## Methodology

SEED operates in two stages, as illustrated in Figure 2 of the paper.

### Stage 1: Hindsight Skill Supervised Fine-Tuning (SFT)

1. **Offline trajectory collection:** A base policy $\pi_{\theta_{\text{base}}}$ collects $K_0$ rollouts per task from a set of training tasks $Q_\text{sft}$, producing a pool $\mathcal{B}$ of trajectories without any skill augmentation.
2. **Hindsight skill annotation:** An external analyzer $A_\text{ext}$ (GLM-5.2) examines each completed trajectory $\tau$ and generates a natural-language hindsight skill $s_\tau$:
   - For successful trajectories: reusable strategies or workflows.
   - For failed trajectories: corrective or avoidance rules.
3. **Supervised fine-tuning:** The policy model is fine-tuned on the trajectory–skill pairs using a standard negative log-likelihood objective:
   $$ \mathcal{L}_\text{sft}(\theta) = -\mathbb{E}_{(x_\tau, s_\tau) \sim \mathcal{D}_\text{sft}} \left[ \sum_{\ell=1}^{|s_\tau|} \log \pi_\theta(s_{\tau,\ell} \mid x_\tau, s_{\tau,<\ell}) \right] $$
   This equips the same model to later serve both as actor and analyzer.

### Stage 2: Self-Evolving On-Policy Distillation

At each outer iteration $k$:

1. **On-policy hindsight skill generation:** The current policy snapshot $\pi_{\theta_{\text{old}}}$ (frozen) samples $N$ trajectories per task $q$:
   $$ \mathcal{G}_q = \{ \tau_q^{(1)}, \dots, \tau_q^{(N)} \}, \quad \tau_q^{(n)} \sim \pi_{\theta_{\text{old}}}(\cdot \mid q) $$
   The same snapshot analyzes each completed trajectory to produce a hindsight skill:
   $$ s_q^{(n)} = A_{\theta_{\text{old}}}\big( x_{\tau_q^{(n)}} \big) $$
   Because the actor and analyzer share the same parameters, improving the policy also improves its analysis capability—creating a **self-evolving loop**.

2. **Skill-augmented context:** For each timestep $t$, the ordinary interaction history $h_{q,n,t}$ is augmented with the generated skill:
   $$ \tilde{h}_{q,n,t} = \mathcal{H}\big( h_{q,n,t}, s_q^{(n)} \big) $$
   where $\mathcal{H}$ is a deterministic insertion function.

3. **Paired contextual re-scoring and OPD loss:** The same policy $\pi_\theta$ (trainable, initialized from $\pi_{\theta_{\text{old}}}$) re-scores the **same sampled action tokens** under two contexts:
   - **Teacher branch** (skill-augmented): $\ell^\text{skill}_{q,n,t,\ell} = \log\pi_\theta(a_{q,n,t,\ell} \mid \tilde{h}_{q,n,t}, a_{q,n,t,<\ell})$
   - **Student branch** (ordinary): $\ell^\theta_{q,n,t,\ell} = \log\pi_\theta(a_{q,n,t,\ell} \mid h_{q,n,t}, a_{q,n,t,<\ell})$

   The detached log-probability shift is:
   $$ \Delta_{q,n,t,\ell} = \text{sg}\big[ \ell^\text{skill}_{q,n,t,\ell} - \ell^\theta_{q,n,t,\ell} \big] $$
   This shift is mapped to a confidence gate:
   $$ g_{q,n,t,\ell} = \sigma(\beta_\text{opd} \Delta_{q,n,t,\ell}) $$
   where $\sigma$ is the logistic sigmoid and $\beta_\text{opd}$ controls sharpness. The OPD loss is:
   $$ \mathcal{L}_\text{opd}(\theta) = \mathbb{E}_{q,n,t,\ell} \Big[ m_{q,n,t,\ell} \cdot g_{q,n,t,\ell} \cdot \big( \text{sg}[ \ell^\text{skill}_{q,n,t,\ell} ] - \ell^\theta_{q,n,t,\ell} \big) \Big] \tag{1} $$

4. **Joint training objective:** SEED combines the OPD loss with a group-relative policy optimization (GRPO) loss:
   $$ \mathcal{L}_\text{SEED}(\theta) = \mathcal{L}_\text{rl}(\theta) + \lambda_\text{opd} \mathcal{L}_\text{opd}(\theta) $$
   The RL loss uses group-normalized advantages:
   $$ A^\text{rl}_{q,n} = \frac{R(\tau_q^{(n)}) - \mu_q}{\sigma_q + \epsilon} $$
   with a clipped importance-ratio objective and KL regularization. After the update, the optimized policy becomes the new snapshot $\pi_{\theta_{\text{old}}}$, closing the loop.

5. **Inference:** Skills are removed; the agent acts only from the ordinary interaction history $a_t \sim \pi_\theta(\cdot \mid h_t)$.

## Empirical Validation / Results

### Benchmarks and Baselines

- **ALFWorld** (embodied household tasks, 6 categories), **WebShop** (web navigation), **Search-based QA** (7 datasets, following Search-R1 protocol).
- Backbones: Qwen2.5-3B-Instruct, Qwen2.5-7B-Instruct, Qwen3-1.7B-Instruct.
- Baselines: Vanilla (no training), Skill-Prompt (skills at test time), GRPO (outcome-only RL), Skill-GRPO (skills during RL training), OPSD, GRPO+OPSD, Skill-SD, RLSD, SDAR (static/offline skill distillation).

### Main Results (Table 1)

| Method (Qwen2.5-3B) | ALFWorld Avg | Search-QA Avg | WebShop Score | WebShop Succ |
|----------------------|--------------|---------------|---------------|--------------|
| Vanilla              | 21.9         | 31.7          | 6.7           | 0.8          |
| GRPO                 | 75.0         | 36.4          | 79.8          | 63.3         |
| Skill-GRPO*          | 80.5         | 36.1          | 76.3          | 66.4         |
| GRPO+OPSD            | 81.2         | 44.6          | 77.8          | 66.4         |
| SDAR                 | 84.4         | 43.4          | 85.0          | 68.0         |
| **SEED (Ours)**      | **91.8**     | **45.7**      | **88.5**      | **78.9**     |

- **Across all backbones:** SEED consistently outperforms GRPO (e.g., ALFWorld: +14.9 to +45.9 points), Skill-GRPO, and all static distillation baselines.
- **Skill internalization beats skill prompting:** SEED (no skills at test) beats Skill-GRPO* (skills at test) in 11/12 aggregate comparisons.
- **Self-evolving beats static distillation:** On ALFWorld, SEED outperforms the best static baseline (SDAR) by 7.4 (3B), 10.2 (7B), and 38.1 (1.7B) points.

### Training Dynamics and Sample Efficiency

- **Figure 3:** On ALFWorld, SEED achieves higher success rates and shorter episode lengths than GRPO from early training steps, indicating more efficient exploration.
- **Sample efficiency (Figure 4, Table 6):** With only 60% of training data, SEED reaches 80.7% success on ALFWorld, exceeding GRPO with 100% data (75.0%). On WebShop, with 80% data SEED scores 75.0 vs. GRPO full-data 63.3.

### Cross-Domain Generalization

- **Figure 5, Table 7:** On ALFWorld Unseen tasks, SEED improves average success rate from 70.9% (GRPO) to 86.2%, with largest gains on Heat (+35.0) and Look (+18.3).

### Ablation Study (Table 2)

| Variant | ALFWorld Avg | Drop |
|---------|--------------|------|
| Full SEED | 91.8 | – |
| w/o Hindsight Skill SFT | 86.0 | -5.8 |
| w/o Self-Evolving OPD | 87.0 | -4.8 |
| w/o On-Policy Skills (static) | 84.4 | -7.4 |

All three components are critical; the largest degradation comes from replacing on-policy skills with a static offline library.

### Vision-Based Extension (Table 8)

- On Sokoban (grid pushing) and EZPoints (visual arithmetic) with Qwen2.5-VL-3B, SEED achieves 82.0% and 100.0%, improving over GRPO by 14.9 and 13.1 points respectively.

### Key Equations from Theoretical Analysis (Appendix A)

- **Proposition 1 (Occupancy-matched hindsight target):** The expected OPD gradient is equivalent to KL divergence minimization toward a skill-reweighted target on the current policy’s own token-context occupancy:
  $$ \nabla_\theta \mathcal{L}_{\text{opd},k}(\theta)\big|_{\theta=\theta_k} = \mathbb{E}_{c \sim d_k} \Big[ Z_k(c) \nabla_\theta \mathrm{D}_{\mathrm{KL}}( r_k(\cdot|c) \| \pi_\theta(\cdot|c) ) \big|_{\theta=\theta_k} \Big] $$
  where $r_k(v|c) \propto \pi_k(v|c) w_k(c,v)$.

- **Proposition 2 (Dense signal under reward ties):** When all group outcomes are identical, the OPD gradient is nonzero if the expected hindsight gate is non-constant across candidate tokens. The gradient magnitude satisfies
  $$ \| \nabla_{z(c)} \mathcal{L}_{\text{opd},k,c} \|^2_{\pi_k^{-1}} = \mathrm{Var}_{v \sim \pi_k(\cdot|c)}[ w_k(c,v) ] $$

- **Proposition 3 (Analyzer-staleness bound):** The discrepancy between the OPD gradient induced by the synchronized analyzer vs. an older analyzer is bounded by
  $$ \| U_k(A_k; \theta) - U_k(A_j; \theta) \|_2 \leq \frac{\beta_\text{opd} G}{4} \mathbb{E}_{x \sim \mu_k}[ | \Delta_{k,k}(x) - \Delta_{k,j}(x) | ] $$

## Theoretical and Practical Implications

- **Bridging the supervision gap:** SEED provides a principled way to convert sparse outcome feedback into dense, decision-level guidance without requiring expensive process reward models or human annotations.
- **Self-evolving supervision:** The shared-parameter actor–analyzer design ensures that hindsight skills continually adapt to the policy’s changing behavior, avoiding the staleness problem of static skill libraries or fixed teachers.
- **Practical deployability:** Skills are used only during training; at inference time, the agent operates from ordinary interaction histories without extra memory, retrieval, or prompting overhead.
- **Sample efficiency:** By extracting more information from each trajectory, SEED achieves strong performance with substantially less training data than outcome-only RL.
- **Generalization:** The internalized skills transfer to unseen tasks and domains, as demonstrated on ALFWorld Unseen and vision-based benchmarks.

## Conclusion

SEED introduces a self-evolving on-policy distillation framework for long-horizon agentic reinforcement learning. By extracting hindsight skills from completed on-policy trajectories and distilling their behavioral effects back into the policy via a dense token-level signal, SEED bridges the granularity gap between sparse trajectory rewards and token-level credit assignment. Joint optimization with outcome-based RL (GRPO) enables the policy to internalize reusable behavioral guidance without relying on skills at inference time.

Experiments across embodied interaction (ALFWorld), web navigation (WebShop), search-based QA, and vision-based tasks demonstrate consistent improvements in task performance, sample efficiency, and cross-domain generalization over outcome-only RL and static skill-distillation baselines. The three key components—hindsight skill SFT, self-evolving OPD, and on-policy skills—are all essential, with on-policy skill generation providing the largest individual contribution.

**Future directions** include scaling to longer and more complex benchmarks, mitigating self-generated supervision biases (e.g., through verifier anchoring or uncertainty-aware gating), and improving training efficiency with speculative decoding and selective trajectory analysis. The code is publicly available at `jinyangwu/SEED`.

---

_Markdown view of https://picx.dev/p/uhzsl4, served by PicX — AI-generated visual whiteboard summaries of research papers._