Meta FAIR and UW propose byte-level distillation, predicting up to 4-point accuracy gain
Researchers from Meta FAIR and the University of Washington propose a byte-level distillation method that converts a teacher model's token-level probability distribution into byte-level distributions by adding end-of-token markers. In distillation experiments using Llama 3-8B as the teacher, scaling-law predictions indicate that byte-level distillation will surpass token-level distillation as compute increases, with downstream average accuracy up to 4 percentage points higher.
Coverage timeline
量子位量子位
Meta新研究：字节模型蒸馏后，天花板破了 henry 2026-09-15 14:37:16 来源： 量子位 henry 发自 凹非寺 量子位 | 公众号 QbitAI Token词元替代了Byte字节，但不意味着大模型都得是Token。 字节模型，特别是蒸馏后，有点儿新说法。 最近，来自Meta FAIR和华盛顿大学的论文《突破Token天花板：蒸馏出更小、更强的字节模型》，提出了一个很有趣的想法—— 蒸馏的时候，能不能把学习单位从词元（Token）换成字节，让学生模型直接从 字节（byte）级别的概率分布 学起？ 结果还真可以～ 团队给每个Token加上结束标记，把教师的Token概率分布完整转换到字节级别。在以Llama 3-8B为教师的蒸馏实验中，团队根据缩放定律预测： 随着训练计算量增加，字节级蒸馏将反超Token级蒸馏，下游平均准确率上限高出 4个百分点 。 传统蒸馏让学生学习教师在庞大Token词表上的概率分布。相比之下，字节只有256种基础取值，单个位置的预测空间更小，基础取值也不随分词器改变。 换句话说， 学习的基本单位反而更小了，模型最终能学到的能力上限却更高了。 这是怎么做到的？ 字节级蒸馏 其实到今天，蒸馏已经不是什么陌生技术了。我们都知道，大的模型能力更强，但把它部署到实际应用中，显存、计算和响应速度都是成本。 于是，一个常见做法，就是让大模型当老师，把能力蒸馏给更小的学生模型。 和普通监督训练只告诉模型“下一个正确Token是什么”不同，蒸馏还会让学生学习教师对 各个候选Token的概率判断 。 但问题也出在这里。Token的候选空间实在太大。以Llama 3-8B为例，它的词表包含 128256个Token ，这意味着每一个预测位置，都对应十几万个候选概率。 如果做离线蒸馏，把完整概率分布全部保存下来，存储成本会非常高。因此实际操作中，通常只能保留概率最高的一部分，也就是 top-k截断 。 而字节模型，刚好把这个候选空间一下压小了。 一个字节由8个比特组成，共有256种可能的取值 。即使加上少量特殊符号，每个位置需要保存的概率也不过两百多个，完整分布自然更容易保留下来。 不过，教师预测的是整个Token，学生预测的是单个字节。要让两者对上，不能只把文本拆开，还得把教师的概率分布一起转换过去。 这正是论文首先解决的问题。 具体的，团队
Related stories
Extreme Sparse Supervision Improves Reasoning in On-Policy Distillation
Researchers from Amazon and Duke University found that retaining gradient signals from only one randomly selected token per rollout during on-policy distillation (OPD) significantly improves student model reasoning, challenging the assumption that dense token-level supervision is necessary. Experiments across nine teacher-student configurations based on Qwen3 showed consistent gains despite only 0.025% of tokens receiving supervision.
1 reports · 1 sources
On-Policy Distillation With One Training Example Recovers Over 70% of Full-Data Gains
Researchers from Tsinghua and collaborators show that on-policy distillation (OPD) with a single math problem can sustain performance gains over hundreds of training steps, recovering over 70% of the improvement achieved with the full 17,000-problem dataset. The second paper in the Rethinking On-Policy Distillation series argues that OPD is 'data-saturated but algorithm-starved,' shifting focus from data scale to algorithmic efficiency.
Meta releases Muse Spark 1.3, claims big coding gains at same price
Meta released Muse Spark 1.3 on Wednesday, available in Muse Code and the Meta Model API, claiming significant improvements in coding and agentic performance at the same price as Spark 1.2. The max reasoning variant, in limited preview for partners, scored 62 on the Artificial Analysis Intelligence Index, behind only Claude Fable 5.1 and Claude Opus 5. Mark Zuckerberg also said the Watermelon model and Muse Spark open weights are coming soon.
3 reports · 1 sources
Liquid AI Releases LFM2.5 Q4_0 GGUF Checkpoints with Quantization-Aware Distillation
Liquid AI has released LFM2.5 Q4_0 GGUF checkpoints trained with Quantization-Aware Distillation (QAD), a technique that recovers up to 97% of BF16 accuracy. The checkpoints are designed for faster edge deployment, offering a practical trade-off between model size and performance.
1 reports · 1 sources
News updates every hour. Subscribe in your RSS reader to get new stories automatically.
Subscribe