Back to News

Tsinghua's ActionOPD Distills Agent Decisions, Not Tokens

#agent#distillation#on-policy#llm

Tsinghua University researchers propose ActionOPD, an action-level on-policy distillation method for training lightweight agent models. It supervises semantic actions in states actually visited by the student, decoupling teacher decision supervision from student token expression to avoid state distribution shift. The paper, code, and model weights have been released.

Coverage timeline

  1. 机器之心机器之心
    蒸馏Agent，学决策而非抄Token！清华提出动作级在线蒸馏ActionOPD

    在多轮工具调用与复杂环境交互中，大语言模型正加速以 Agent 形态落地。受限于前沿大模型高昂的推理成本，如何将其决策能力迁移至轻量级模型，成为当前 Agent 研究的重要课题。 然而，不同于单轮文本生成，Agent 的每一步输出都会改变环境状态并影响后续轨迹，这引出了两个基本问题： 1. 在什么状态监督？ 是在教师走出的预设轨迹上，还是在学生实际遇到的状态上？ 2. 以什么粒度对齐？ 是逐 Token 拟合词表分布，还是在更高层的 “动作语义” 上对齐决策？ 围绕这两个问题，清华大学研究团队提出了 ActionOPD（Action-level On-Policy Distillation） ：在 学生实际访问 的状态下，以高层 “ 语义动作 ” 作为监督单元，在规避状态分布偏移的同时，实现了教师决策监督与学生 Token 表达的彻底解耦。 论文标题：ActionOPD: Distilling Decisions, Not Tokens 作者团队：清华大学 Huining Yuan*、Xiangmin Yi*、Zelai Xu、Zixu Wang、Xinlei Chen、Xiao-Ping Zhang†、Yu Wang†、Chao Yu† 论文链接：https://arxiv.org/abs/2610.19820 项目主页：https://thu-nics.github.io/ActionOPD/ 开源代码：https://github.com/thu-nics/ActionOPD 模型权重：https://huggingface.co/collections/nics-efc/actionopd 一、从 “逐字模仿” 到 “决策对齐”：为什么 Token 级蒸馏存在局限？ 审视当前主流的两类 Agent 蒸馏范式（图 1），二者在多轮交互中各有掣肘： 离线知识蒸馏（KD/SFT） ：在教师轨迹上逐 Token 模仿。因训练未覆盖失误状态，学生测试时一旦偏离预设轨迹，便会因状态分布偏移而性能骤降。 Token 级在线策略蒸馏（Token-level OPD） ：在学生采样轨迹上对齐教师 Token 分布。虽缓解了状态偏移，却 强依赖教师白盒 log-prob 与同源 Tokenizer ，难以用于黑盒或跨家族蒸馏，且仍强求逐

Related stories

  • Tsinghua team's MGFlow and Elastic Forcing achieve SOTA one-step image generation and efficient 14B video model post-training

    Researchers at Tsinghua University and collaborators introduced distributional training, a method that supervises generation by comparing the overall distribution of generated samples with reference samples rather than assigning unique answers per sample. Their MGFlow method achieved state-of-the-art one-step generation on ImageNet 256×256 (pMF-H, JiT-H: 1.45, 1.64 FID) and outperformed the original 4-step FLUX.2 [klein] 4B model in text-to-image (GenEval 0.900, PickScore 21.98). Elastic Forcing, the video counterpart, scored 84.64 on VBench, surpassing Self Forcing, and post-trained a 14B streaming video model on 8 H200 GPUs in 23.2 hours.

    1 reports · 1 sources

  • UCSB and Microsoft Propose ScholarEvolve for AI Agents to Read Papers and Self-Modify Code

    Researchers from UC Santa Barbara and Microsoft introduced ScholarEvolve, a method that automates the research process of reading literature, implementing algorithms, and evaluating results to enable AI agents to evolve their harness code without modifying underlying model parameters. The approach addresses the limitation of prior coding-agent-based harness optimization, which relied on the agent's existing knowledge and observed failures, by allowing agents to draw on the latest academic findings. The paper and code are publicly available.

    2 reports · 1 sources

  • Tsinghua introduces SRHarness and MechBench for agentic symbolic regression and mechanism discovery

    Tsinghua University researchers introduced SRHarness, a runtime framework for agentic symbolic regression, and MechBench, a benchmark for mechanism discovery, aiming to improve AI's ability to derive formulas from data and understand underlying physical mechanisms. Experiments show SRHarness achieves 72.97% symbol recovery with DeepSeek-v4-flash-0731, far exceeding Codex's 20.72%, while MechBench reveals that even GPT-5.6-sol-powered Codex recovers mechanisms at only 13.75% on the core set, highlighting a significant gap between pattern discovery and mechanism understanding.

    1 reports · 1 sources

  • Redwood Research tests distillation for AI safety incrimination and capabilities

    Redwood Research released a paper empirically testing two distillation approaches for AI safety: Distillation for Incrimination (DFI) and Distillation for Capabilities (DFC). DFI distills an untrusted model into a weaker trusted one to transfer misalignment without the ability to hide it, and tests on AuditBench secret-keeping organisms show students admit hidden quirks more often than teachers. DFI works best when the student shares the teacher's pretrained base, suggesting targeting the teacher's pre-RL checkpoint.

    1 reports · 1 sources

News updates every hour. Subscribe in your RSS reader to get new stories automatically.

Subscribe