STOPD accelerates on-policy distillation training by up to 2.2x with early stopping and inverse probability weighting
Researchers propose STOPD, a method that accelerates on-policy distillation (OPD) training by randomly selecting stop positions before generation and using inverse probability weighting to correct truncation bias. In experiments, STOPD reduces generated tokens by 25%–48% and achieves 1.3–2.2x training speedup while maintaining accuracy comparable to vanilla OPD. The method also provides theoretical guarantees, including unbiased gradient sums and single-step update error bounds for SGD, Adam, and Muon.
Coverage timeline
机器之心机器之心
On-policy distillation(OPD)的做法很简单:先让学生自己生成回答,再由教师逐 token 提供反馈,指导学生更新。这样教师就能直接针对学生实际生成的内容进行指导,而不必依赖预先准备好的答案。问题在于,每一轮训练都需要学生生成完整的回答,之后教师还要对每个位置进行评估。随着回答变长,训练成本也会越来越大。 因此,我们提出了 STOPD:在生成前随机选好停止位置,然后让部分学生回答提前结束,最后再通过逆概率加权补偿截断带来的采样偏差。整个过程不需要评估回答质量,也不需要额外调用模型来判断何时停止。 在我们的实验中,STOPD 在准确率与 Vanilla OPD 相近的情况下,减少了 25%–48% 的生成 token ,实现了 1.3–2.2 倍训练加速 。在理论上,我们还额外证明了加权后的梯度总和在期望上与完整生成一致,并给出了 SGD、Adam 和 Muon 下的单步更新误差界。 图 1|STOPD 的方法与实验结果。 论文:https://arxiv.org/abs/2610.21337 代码:https://github.com/YouAreSpecialToMe/STOPD Hugging Face:https://huggingface.co/papers/2610.21337 已有方法如何减少计算 减少 OPD 开销通常有两种做法。 一种是 稀疏监督 :学生仍然生成完整的回答,但只选择部分 token 参与训练。这样的话如何选出值得监督的 token 就成了关键。有的方法关注学生在哪些位置更需要指导,比如 TIP [1] 根据学生的不确定性和师生分歧进行选择。有的方法关注教师的指导能否被学生有效吸收,比如 TA-OPD [2] 考虑教师信号的可学习性。还有的方法关注训练信号本身是否可靠,例如 IER [3] 考虑梯度估计的可靠性。这些方法虽然选择标准不同,但都通过减少了需要监督的 token 数量的方法削减了开销。 另一种是 直接截断生成 。最简单的做法是直接限制生成长度,比如 Early Stopping Rollout(ESR)[4] 只保留固定长度的前缀。在此基础上,Progressive OPD(POPD)[7] 进一步考虑训练进度,让生成长度随着训练逐步增加。除了预先设定长度规则,也有方法利用师生反馈动态调整生成过程。比如

Related stories
Tsinghua's ActionOPD Distills Agent Decisions, Not Tokens
Tsinghua University researchers propose ActionOPD, an action-level on-policy distillation method for training lightweight agent models. It supervises semantic actions in states actually visited by the student, decoupling teacher decision supervision from student token expression to avoid state distribution shift. The paper, code, and model weights have been released.
1 reports · 1 sources

Alibaba Qwen and OSU Propose TRACE to Speed Up FP4 RL Rollout by 5.4x
Alibaba Qwen and Ohio State University researchers proposed TRACE, a rollout-guided quantization-aware training method for FP4 reinforcement learning of mixture-of-experts (MoE) language models. By using the actual FP4 quantization results from the rollout side to guide rounding on the training side, TRACE reduces discrepancies between training and rollout paths, recovering performance close to BF16 while achieving up to 5.4x decoding throughput improvement in rollout with FP4 weights/activations and FP4 KV cache at 128K output length. The paper is available on arXiv and Hugging Face.
1 reports · 1 sources

Variational Method Proposed for Optimal Learning Rate Scheduling in Weight-Normalized Training
A blog post explains and extends a paper that estimates optimal learning rate schedules in weight-normalized training by simplifying training dynamics into a one-dimensional differential equation and applying variational principles to maximize a target under that constraint. The approach offers a method to translate training intuitions into schedule shapes rather than providing a single formula.
1 reports · 1 sources
Q Labs proposes Dust, a zeroth-order optimizer that rivals backprop in Transformer pretraining up to 243M params
Q Labs, a small research lab backed by Jeff Dean, published a paper introducing Dust, a zeroth-order optimization method that adds Gaussian noise to activations per token and estimates gradients via a virtual population. They claim it is the first such method to compete with backpropagation in Transformer pretraining, with experiments up to 243M parameters and 20M tokens. Third-party replication from SOTAAZ Lab confirmed basic results but found backprop leads when given better optimizers or more epochs, and Dust is far more computationally expensive, so its significance lies in demonstrating feasibility rather than replacing backprop.
1 reports · 1 sources

News updates every hour. Subscribe in your RSS reader to get new stories automatically.
Subscribe