Summary (Overview)

• Problem: Outcome-based reinforcement learning for LLM agents provides only sparse, trajectory-level rewards, offering limited guidance for intermediate decisions and token-level credit assignment.
• Proposed Method: SEED (Self-Evolving On-Policy Distillation) extracts natural-language hindsight skills from completed on-policy trajectories and distills their behavioral effect back into the policy via dense token-level supervision.
• Key Mechanism: The policy simultaneously acts as a rollout actor and a trajectory analyzer. Skills are generated on-policy and used to re-score the same sampled actions under skill-augmented versus ordinary contexts, producing a token-level distillation signal.
• Core Advantages: The self-evolving loop keeps hindsight supervision aligned with the evolving policy; dense token-level signals fill the granularity gap of sparse rewards; no skill prompts or external memory are needed at inference time.
• Empirical Results: SEED consistently outperforms outcome-only RL (GRPO) and static skill-distillation baselines across embodied interaction (ALFWorld), web navigation (WebShop), search-based QA, and vision-based tasks, with improved sample efficiency and cross-domain generalization.

Introduction and Theoretical Foundation

LLM-based agents must make sequential decisions over long horizons, often receiving feedback only at the end of an episode. Standard reinforcement learning (RL) optimizes against such trajectory-level outcomes (e.g., success/failure), but this coarse supervision cannot pinpoint which intermediate actions, observations, or tool calls should be reinforced or corrected—a supervision gap between episode-level outcomes and token-level policy learning.

A key insight is that completed trajectories reveal hindsight information unavailable during online decision making: reusable strategies, decisive observations, failure causes, and workflows. While prior work has used verbal reflection, episodic memory, or experience summaries, such hindsight is often static or used only as inference-time context. For practical agentic RL, hindsight supervision should be:

1. On-policy – derived from the current policy’s own trajectories to match the state-action distribution.
2. Dense – providing per-token credit assignment rather than a single trajectory-level signal.
3. Self-evolving – adapting as the policy improves and encounters new behaviors and failure modes.

SEED addresses these requirements by converting completed on-policy trajectories into natural-language hindsight skills (e.g., “When placing an object, first locate the target, take it to the destination, and confirm placement”) and then using those skills to construct a dense, token-level on-policy distillation (OPD) signal that is jointly optimized with outcome-based RL (specifically GRPO, Shao et al., 2024).

Methodology

SEED operates in two stages, as illustrated in Figure 2 of the paper.

Stage 1: Hindsight Skill Supervised Fine-Tuning (SFT)

1. Offline trajectory collection: A base policy $\pi_{\theta_{\text{base}}}$ collects $K_0$ rollouts per task from a set of training tasks $Q_\text{sft}$, producing a pool $\mathcal{B}$ of trajectories without any skill augmentation.
2. Hindsight skill annotation: An external analyzer $A_\text{ext}$ (GLM-5.2) examines each completed trajectory $\tau$ and generates a natural-language hindsight skill $s_\tau$:
   • For successful trajectories: reusable strategies or workflows.
   • For failed trajectories: corrective or avoidance rules.
3. Supervised fine-tuning: The policy model is fine-tuned on the trajectory–skill pairs using a standard negative log-likelihood objective: $$\mathcal{L}_\text{sft}(\theta) = -\mathbb{E}_{(x_\tau, s_\tau) \sim \mathcal{D}_\text{sft}} \left[ \sum_{\ell=1}^{|s_\tau|} \log \pi_\theta(s_{\tau,\ell} \mid x_\tau, s_{\tau,<\ell}) \right]$$ This equips the same model to later serve both as actor and analyzer.

Stage 2: Self-Evolving On-Policy Distillation

At each outer iteration $k$:

1. On-policy hindsight skill generation: The current policy snapshot $\pi_{\theta_{\text{old}}}$ (frozen) samples $N$ trajectories per task $q$:

   $$\mathcal{G}_q = \{ \tau_q^{(1)}, \dots, \tau_q^{(N)} \}, \quad \tau_q^{(n)} \sim \pi_{\theta_{\text{old}}}(\cdot \mid q)$$

   The same snapshot analyzes each completed trajectory to produce a hindsight skill:

   $$s_q^{(n)} = A_{\theta_{\text{old}}}\big( x_{\tau_q^{(n)}} \big)$$

   Because the actor and analyzer share the same parameters, improving the policy also improves its analysis capability—creating a self-evolving loop.

2. Skill-augmented context: For each timestep $t$, the ordinary interaction history $h_{q,n,t}$ is augmented with the generated skill:

   $$\tilde{h}_{q,n,t} = \mathcal{H}\big( h_{q,n,t}, s_q^{(n)} \big)$$

   where $\mathcal{H}$ is a deterministic insertion function.

3. Paired contextual re-scoring and OPD loss: The same policy $\pi_\theta$ (trainable, initialized from $\pi_{\theta_{\text{old}}}$) re-scores the same sampled action tokens under two contexts:

   • Teacher branch (skill-augmented): $\ell^\text{skill}_{q,n,t,\ell} = \log\pi_\theta(a_{q,n,t,\ell} \mid \tilde{h}_{q,n,t}, a_{q,n,t,<\ell})$
   • Student branch (ordinary): $\ell^\theta_{q,n,t,\ell} = \log\pi_\theta(a_{q,n,t,\ell} \mid h_{q,n,t}, a_{q,n,t,<\ell})$

   The detached log-probability shift is:

   $$\Delta_{q,n,t,\ell} = \text{sg}\big[ \ell^\text{skill}_{q,n,t,\ell} - \ell^\theta_{q,n,t,\ell} \big]$$

   This shift is mapped to a confidence gate:

   $$g_{q,n,t,\ell} = \sigma(\beta_\text{opd} \Delta_{q,n,t,\ell})$$

   where $\sigma$ is the logistic sigmoid and $\beta_\text{opd}$ controls sharpness. The OPD loss is:

   $$\mathcal{L}_\text{opd}(\theta) = \mathbb{E}_{q,n,t,\ell} \Big[ m_{q,n,t,\ell} \cdot g_{q,n,t,\ell} \cdot \big( \text{sg}[ \ell^\text{skill}_{q,n,t,\ell} ] - \ell^\theta_{q,n,t,\ell} \big) \Big] \tag{1}$$

4. Joint training objective: SEED combines the OPD loss with a group-relative policy optimization (GRPO) loss:

   $$\mathcal{L}_\text{SEED}(\theta) = \mathcal{L}_\text{rl}(\theta) + \lambda_\text{opd} \mathcal{L}_\text{opd}(\theta)$$

   The RL loss uses group-normalized advantages:

   $$A^\text{rl}_{q,n} = \frac{R(\tau_q^{(n)}) - \mu_q}{\sigma_q + \epsilon}$$

   with a clipped importance-ratio objective and KL regularization. After the update, the optimized policy becomes the new snapshot $\pi_{\theta_{\text{old}}}$, closing the loop.

5. Inference: Skills are removed; the agent acts only from the ordinary interaction history $a_t \sim \pi_\theta(\cdot \mid h_t)$.

Empirical Validation / Results

Benchmarks and Baselines

• ALFWorld (embodied household tasks, 6 categories), WebShop (web navigation), Search-based QA (7 datasets, following Search-R1 protocol).
• Backbones: Qwen2.5-3B-Instruct, Qwen2.5-7B-Instruct, Qwen3-1.7B-Instruct.
• Baselines: Vanilla (no training), Skill-Prompt (skills at test time), GRPO (outcome-only RL), Skill-GRPO (skills during RL training), OPSD, GRPO+OPSD, Skill-SD, RLSD, SDAR (static/offline skill distillation).

Main Results (Table 1)

• Across all backbones: SEED consistently outperforms GRPO (e.g., ALFWorld: +14.9 to +45.9 points), Skill-GRPO, and all static distillation baselines.
• Skill internalization beats skill prompting: SEED (no skills at test) beats Skill-GRPO* (skills at test) in 11/12 aggregate comparisons.
• Self-evolving beats static distillation: On ALFWorld, SEED outperforms the best static baseline (SDAR) by 7.4 (3B), 10.2 (7B), and 38.1 (1.7B) points.

Training Dynamics and Sample Efficiency

• Figure 3: On ALFWorld, SEED achieves higher success rates and shorter episode lengths than GRPO from early training steps, indicating more efficient exploration.
• Sample efficiency (Figure 4, Table 6): With only 60% of training data, SEED reaches 80.7% success on ALFWorld, exceeding GRPO with 100% data (75.0%). On WebShop, with 80% data SEED scores 75.0 vs. GRPO full-data 63.3.

Cross-Domain Generalization

• Figure 5, Table 7: On ALFWorld Unseen tasks, SEED improves average success rate from 70.9% (GRPO) to 86.2%, with largest gains on Heat (+35.0) and Look (+18.3).

Ablation Study (Table 2)

All three components are critical; the largest degradation comes from replacing on-policy skills with a static offline library.

Vision-Based Extension (Table 8)

• On Sokoban (grid pushing) and EZPoints (visual arithmetic) with Qwen2.5-VL-3B, SEED achieves 82.0% and 100.0%, improving over GRPO by 14.9 and 13.1 points respectively.

Key Equations from Theoretical Analysis (Appendix A)

• Proposition 1 (Occupancy-matched hindsight target): The expected OPD gradient is equivalent to KL divergence minimization toward a skill-reweighted target on the current policy’s own token-context occupancy:

  $$\nabla_\theta \mathcal{L}_{\text{opd},k}(\theta)\big|_{\theta=\theta_k} = \mathbb{E}_{c \sim d_k} \Big[ Z_k(c) \nabla_\theta \mathrm{D}_{\mathrm{KL}}( r_k(\cdot|c) \| \pi_\theta(\cdot|c) ) \big|_{\theta=\theta_k} \Big]$$

  where $r_k(v|c) \propto \pi_k(v|c) w_k(c,v)$.

• Proposition 2 (Dense signal under reward ties): When all group outcomes are identical, the OPD gradient is nonzero if the expected hindsight gate is non-constant across candidate tokens. The gradient magnitude satisfies

  $$\| \nabla_{z(c)} \mathcal{L}_{\text{opd},k,c} \|^2_{\pi_k^{-1}} = \mathrm{Var}_{v \sim \pi_k(\cdot|c)}[ w_k(c,v) ]$$

• Proposition 3 (Analyzer-staleness bound): The discrepancy between the OPD gradient induced by the synchronized analyzer vs. an older analyzer is bounded by

  $$\| U_k(A_k; \theta) - U_k(A_j; \theta) \|_2 \leq \frac{\beta_\text{opd} G}{4} \mathbb{E}_{x \sim \mu_k}[ | \Delta_{k,k}(x) - \Delta_{k,j}(x) | ]$$

Theoretical and Practical Implications

• Bridging the supervision gap: SEED provides a principled way to convert sparse outcome feedback into dense, decision-level guidance without requiring expensive process reward models or human annotations.
• Self-evolving supervision: The shared-parameter actor–analyzer design ensures that hindsight skills continually adapt to the policy’s changing behavior, avoiding the staleness problem of static skill libraries or fixed teachers.
• Practical deployability: Skills are used only during training; at inference time, the agent operates from ordinary interaction histories without extra memory, retrieval, or prompting overhead.
• Sample efficiency: By extracting more information from each trajectory, SEED achieves strong performance with substantially less training data than outcome-only RL.
• Generalization: The internalized skills transfer to unseen tasks and domains, as demonstrated on ALFWorld Unseen and vision-based benchmarks.

Conclusion

SEED introduces a self-evolving on-policy distillation framework for long-horizon agentic reinforcement learning. By extracting hindsight skills from completed on-policy trajectories and distilling their behavioral effects back into the policy via a dense token-level signal, SEED bridges the granularity gap between sparse trajectory rewards and token-level credit assignment. Joint optimization with outcome-based RL (GRPO) enables the policy to internalize reusable behavioral guidance without relying on skills at inference time.

Experiments across embodied interaction (ALFWorld), web navigation (WebShop), search-based QA, and vision-based tasks demonstrate consistent improvements in task performance, sample efficiency, and cross-domain generalization over outcome-only RL and static skill-distillation baselines. The three key components—hindsight skill SFT, self-evolving OPD, and on-policy skills—are all essential, with on-policy skill generation providing the largest individual contribution.

Future directions include scaling to longer and more complex benchmarks, mitigating self-generated supervision biases (e.g., through verifier anchoring or uncertainty-aware gating), and improving training efficiency with speculative decoding and selective trajectory analysis. The code is publicly available at jinyangwu/SEED.