Summary (Overview)

• PCSD (Persistent Consistency Self-Distillation) is a novel on-policy self-distillation framework for agentic reinforcement learning that estimates token-level teacher credibility from the local persistence of teacher-favoring signals, combining fine-grained positional discrimination with robustness to pointwise noise.
• It addresses the fundamental trade-off between token-level precision (sensitive to noise) and step-level robustness (obscures positional variation) in existing credibility-aware distillation methods.
• PCSD implements three complementary mechanisms: adaptive exponential-decay aggregation, one-sided trend modulation, and sigmoid gating to produce continuous token-level distillation weights.
• The PCSD-weighted distillation objective is jointly optimized with GRPO (Group Relative Policy Optimization), combining dense teacher guidance with sparse environmental feedback.
• Without inference-time skills, PCSD achieves the best ALFWorld Overall results among all baselines on both backbones, exceeding GRPO by 15.6 and 13.3 points and SDAR by 6.2 and 5.5 points, while remaining competitive on WebShop and gaining 15.8 points over GRPO on unseen ALFWorld split.

Introduction and Theoretical Foundation

Large language model (LLM) agents must make sequences of interdependent decisions over extended multi-turn interactions, where each action changes the subsequent environmental state. Reinforcement learning methods like GRPO provide a natural framework for optimizing such sequential behavior, but their effectiveness is constrained by reward sparsity: a trajectory spanning dozens of turns and hundreds of generated tokens may receive only a single scalar reward at its end.

On-policy self-distillation (OPSD) alleviates reward sparsity by using a privileged teacher—typically a frozen copy of the student augmented with additional context—to provide dense token-level supervision on trajectories sampled from the current policy. However, privileged information does not guarantee reliable guidance at every position: imperfect retrieval, noisy context, and task ambiguity may cause the teacher to favor suboptimal tokens.

Existing credibility-aware distillation methods typically estimate teacher reliability at either the token or step level:

• Token-level methods compute a distillation weight from an isolated teacher–student discrepancy or probability ratio at each position. While preserving fine-grained positional information, such pointwise estimates are sensitive to sampling variability.
• Step-level methods aggregate discrepancy signals over an entire interaction or reasoning step and assign a shared weight to all tokens within that step. This aggregation improves robustness to local noise but may obscure substantial variation in teacher credibility across positions.

The key insight of PCSD is that informative teacher advantage should persist across a local neighborhood, whereas isolated spikes are more likely to reflect sampling noise or incidental variation. PCSD therefore evaluates each token using both its pointwise discrepancy and the persistence of supporting evidence at nearby positions.

Methodology

Problem Formulation

In a multi-turn agent task, at turn $k$, the environment provides a state $s_k$, and the student policy $\pi_\theta$ generates a response $\mathbf{y}_k = (y_{k,1}, y_{k,2}, \dots, y_{k,T_k}) \sim \pi_\theta(\cdot | s_k)$. A complete trajectory is:

$$\boldsymbol{\tau} = (s_1, \mathbf{y}_1, s_2, \mathbf{y}_2, \ldots, s_k, \mathbf{y}_k)$$

Teacher–Student Log-Probability Gap

The teacher–student sampled log-probability gap is defined as:

$$\delta_{k,i} = \log \pi_T(y_{k,i} \mid h_{k,i}^T) - \log \pi_\theta(y_{k,i} \mid h_{k,i}^S)$$

where $\pi_T$ is the frozen privileged-context teacher policy, and $h_{k,i}^S$ and $h_{k,i}^T$ are the student-visible and teacher-augmented contexts, respectively. Larger values of $\delta_{k,i}$ indicate stronger relative teacher support for the sampled token $y_{k,i}$.

Persistent Consistency Assessment

PCSD aggregates gaps over a forward window of size $N$ with exponential weighting:

$$\bar{\delta}_{k,i}^{(N)} = \frac{\sum_{j=0}^{N-1} \alpha^j m_{k,i+j} \delta_{k,i+j}}{\sum_{j=0}^{N-1} \alpha^j m_{k,i+j}}$$

where $\alpha \in (0,1)$ is the decay factor and $m_{k,i}$ is the response mask.

Adaptive Aggregation Mechanism

A single fixed window may not adequately capture heterogeneous gap patterns. PCSD implements soft adaptive windowing by computing:

1. Local mean over a maximum analysis window of size $N_{\text{max}}$:

$$\mu_{k,i}^{(N_{\text{max}})} = \frac{\sum_{j=0}^{N_{\text{max}}-1} m_{k,i+j} \delta_{k,i+j}}{\sum_{j=0}^{N_{\text{max}}-1} m_{k,i+j}}$$

2. Local variance:

$$\sigma_{k,i}^2 = \frac{\sum_{j=0}^{N_{\max}-1} m_{k,i+j} (\delta_{k,i+j} - \mu_{k,i}^{(N_{\max})})^2}{\sum_{j=0}^{N_{\max}-1} m_{k,i+j}}$$

3. Interpolation coefficient:

$$r_{k,i} = \operatorname{clip}\left(\frac{\sigma_{k,i}^2 - \tau_{\text{low}}}{\tau_{\text{high}} - \tau_{\text{low}}}, 0, 1\right)$$

4. Adaptive persistent-consistency estimate:

$$\bar{\delta}_{k,i}^{\text{adaptive}} = (1 - r_{k,i}) \bar{\delta}_{k,i}^{(N_{\min})} + r_{k,i} \bar{\delta}_{k,i}^{(N_{\max})}$$

Trend Modulation and Continuous Gating

To address patterns of rapidly decreasing teacher support, PCSD applies one-sided trend modulation. A mask-aware OLS slope is estimated over the maximum analysis window:

$$\bar{j}_{k,i} = \frac{\sum_{j=0}^{N_{\max}-1} m_{k,i+j} j}{\sum_{j=0}^{N_{\max}-1} m_{k,i+j}}$$ $$\mathrm{slope}_{k,i} = \frac{\sum_{j=0}^{N_{\max}-1} m_{k,i+j} (j - \bar{j}_{k,i}) \delta_{k,i+j}}{\sum_{j=0}^{N_{\max}-1} m_{k,i+j} (j - \bar{j}_{k,i})^2 + \epsilon_{\mathrm{slope}}}$$

The trend modulation factor is:

$$\eta_{k,i} = \operatorname{clip}\left(1 - \gamma \cdot \operatorname{ReLU}\left(-\frac{\mathrm{slope}_{k,i}}{\delta_{\text{scale},k}}\right), 0, 1\right)$$

where $\delta_{\text{scale},k} = \frac{\sum_i m_{k,i} |\delta_{k,i}|}{\sum_i m_{k,i}} + \epsilon_{\text{scale}}$.

The final token-level distillation weight is:

$$w_{k,i} = \sigma\left(\beta_{\text{gate}} \cdot \bar{\delta}_{k,i}^{\text{adaptive}}\right) \cdot \eta_{k,i}$$

Training Objective

The PCSD distillation objective is:

$$\mathcal{L}_{\text{PCSD}} = \frac{1}{M} \sum_{k,i} m_{k,i} w_{k,i} \delta_{k,i}$$

where $M = \sum_{k,i} m_{k,i}$ is the total number of valid response tokens. The total objective combines this with GRPO:

$$\mathcal{L}_{\text{total}} = \mathcal{L}_{\text{GRPO}} + \lambda_{\text{PCSD}} \mathcal{L}_{\text{PCSD}}$$

Empirical Validation / Results

Benchmarks and Baselines

Experiments were conducted on ALFWorld (a text-based embodied AI benchmark across six household activity categories) and WebShop (simulating realistic e-commerce scenarios), using Qwen2.5-3B-Instruct and Qwen3-1.7B-Instruct backbones.

Baselines include: Vanilla, Skill-Prompt*, OPSD, GRPO, Skill-GRPO, Skill-GRPO*, GRPO+OPSD, RLSD, and SDAR.

Main Results

Key findings:

• PCSD achieves the highest ALFWorld Overall success rates of 90.6% and 59.4% with Qwen2.5-3B-Instruct and Qwen3-1.7B-Instruct, respectively.
• PCSD exceeds GRPO by 15.6 and 13.3 percentage points and SDAR by 6.2 and 5.5 percentage points.
• On WebShop, PCSD achieves the highest Score of 85.0 with Qwen2.5-3B-Instruct and ties with SDAR for the highest Acc on both backbones.

Generalization to Unseen Environments

On the ALFWorld unseen split, PCSD achieves an Overall success rate of 86.7%, well above GRPO's 70.9% and SDAR's 72.7%.

Ablation Studies

Theoretical and Practical Implications

• Theoretical contribution: PCSD introduces the concept of persistent consistency as a principled way to estimate token-level teacher credibility, resolving the fundamental trade-off between token-level precision and step-level robustness in self-distillation.
• Practical significance: PCSD achieves substantial improvements over outcome-only RL and existing self-distillation baselines without requiring inference-time skill retrieval, making it more practical for deployment.
• Training dynamics: The mean teacher–student gap remains negative but gradually increases during training, while 20%–28% of valid tokens have $w_{k,i} > 0.5$, indicating concentrated rather than hard-selective distillation.
• Robustness: PCSD generalizes robustly to unseen environment configurations, suggesting that persistent-consistency weighting captures fundamental patterns of reliable teacher guidance.

Conclusion

PCSD (Persistent Consistency Self-Distillation) is an on-policy self-distillation framework that weights token-level supervision based on persistent local support from the teacher. By integrating adaptive exponential-decay aggregation, one-sided trend modulation, and continuous sigmoid gating, PCSD incorporates evidence across multiple tokens while preserving positional specificity.

Experiments demonstrate consistent improvements over outcome-only RL and existing self-distillation baselines on ALFWorld and WebShop across two model backbones, with robust generalization to unseen environments. The work represents a significant step toward more effective credit assignment in long-horizon agentic tasks through selective, persistence-aware distillation.

Future directions include learning context-dependent aggregation and gating parameters from trajectory statistics, teacher uncertainty, and environmental feedback, as well as self-evolving distillation where the student, teacher, credibility estimator, and skill repository co-evolve through online interaction.