Back to News

Ask-to-Distill: UIUC and others propose framework to optimize synthetic questions for on-policy distillation

#knowledge-distillation#on-policy-distillation#synthetic-data#llm-training

Researchers from UIUC, HKUST, and Texas A&M proposed Ask-to-Distill (A2D), a framework that optimizes synthetic questions for on-policy distillation by measuring the teacher supervision signal each question can elicit, without relying on standard answers or task success rates. Experiments in math reasoning, code generation, and agent tasks show A2D generates more effective distillation data and outperforms real and synthetic baselines, while reducing student response length by about 20% and increasing cumulative learning signal by about 28%.

Coverage timeline

  1. 机器之心机器之心

    一道好问题,可能比一千道普通问题更有价值。 在大语言模型的训练中,人们通常认为,要让模型变得更聪明,就需要更多高质量数据、更可靠的标准答案,以及更强大的教师模型。但近期关于 On-Policy Distillation(OPD,在线策略蒸馏)的研究揭示了一个值得关注的现象:即使只使用很少的训练问题,学生模型也有可能从教师模型中获得显著的性能提升。 这引出了一个新的问题: 如果少量问题就足以带来显著的知识迁移,那么我们是否能够让模型主动学习,生成那些最有利于自身学习的问题? 进一步来说,如果这些问题是模型自己生成的,没有标准答案,也没有外部验证器,又该如何判断哪些问题真正值得用于训练? 近日,来自伊利诺伊大学厄巴纳-香槟分校(UIUC)、香港科技大学和德克萨斯农工大学等机构的研究者提出了 Ask-to-Distill(A2D) ,一种面向在策略蒸馏的合成问题优化框架。不同于依赖标准答案或任务成功率评估合成数据的方法,A2D 从教师与学生的预测分布出发,直接衡量一道问题能够激发多少可利用的教师监督信号,并通过强化学习优化问题生成器。在数学推理、代码生成和智能体任务上的实验表明,A2D 能够在不依赖外部训练问题和正确答案的情况下,生成更有效的蒸馏数据,并在所评估的设置中取得优于多种真实数据及合成数据基线的平均性能。 更有意思的是,研究发现: A2D 生成的问题能够让学生模型的回答长度减少约 20%,同时激发的累计学习信号增加约 28%。 这意味着,真正有价值的训练问题,未必需要让模型进行更长、更复杂的推理,而可能是那些能够精准暴露其知识缺口的问题。 论文标题:Ask to Distill: Learning to Synthesize Data for On-Policy Distillation 论文地址:https://arxiv.org/abs/2610.27450 代码地址:https://github.com/GasolSun36/Ask_to_Distill 模型与数据:https://huggingface.co/collections/gasolsun/ask-to-distill-model-and-data-collection 从「如何回答问题」到「应该提出什么问题」 大语言模型的知识蒸馏,通常希望将强教师模型的能力迁移到相对较弱的学生模型中。传统的

Related stories

  • UCSB and Microsoft Propose ScholarEvolve for AI Agents to Read Papers and Self-Modify Code

    Researchers from UC Santa Barbara and Microsoft introduced ScholarEvolve, a method that automates the research process of reading literature, implementing algorithms, and evaluating results to enable AI agents to evolve their harness code without modifying underlying model parameters. The approach addresses the limitation of prior coding-agent-based harness optimization, which relied on the agent's existing knowledge and observed failures, by allowing agents to draw on the latest academic findings. The paper and code are publicly available.

    2 reports · 1 sources
  • Tsinghua's ActionOPD Distills Agent Decisions, Not Tokens

    Tsinghua University researchers propose ActionOPD, an action-level on-policy distillation method for training lightweight agent models. It supervises semantic actions in states actually visited by the student, decoupling teacher decision supervision from student token expression to avoid state distribution shift. The paper, code, and model weights have been released.

    1 reports · 1 sources
  • Reuters: 300+ Chinese docs show AI distillation from US models now widespread industrial practice

    A Reuters review of more than 300 Chinese documents found that AI distillation—using outputs from leading US models to train domestic rivals—has shifted from academic experiments to widespread industrial use. The review indicates Beijing is actively encouraging this practice, with Chinese entities adopting increasingly sophisticated methods.

    1 reports · 1 sources
  • Lev: Recurrent Decision Model Adapts Compute per Problem, No Chain-of-Thought

    Researchers from Lehigh University, UBC, and Vector Institute propose Lev, a decision model that uses recurrent computation instead of chain-of-thought to adaptively spend more compute on hard problems. Lev reads out option probabilities after each recurrent pass, trains jointly across depths, and uses a lightweight stopper to decide how many passes to run, enabling a single set of weights to answer at arbitrary depth. The authors describe this as System 1.5, preserving the direct interface of System 1 while allowing extra compute on difficult cases like System 2.

    1 reports · 1 sources

News updates every hour. Subscribe in your RSS reader to get new stories automatically.

Subscribe