Q Labs proposes Dust, a zeroth-order optimizer that rivals backprop in Transformer pretraining up to 243M params
Q Labs, a small research lab backed by Jeff Dean, published a paper introducing Dust, a zeroth-order optimization method that adds Gaussian noise to activations per token and estimates gradients via a virtual population. They claim it is the first such method to compete with backpropagation in Transformer pretraining, with experiments up to 243M parameters and 20M tokens. Third-party replication from SOTAAZ Lab confirmed basic results but found backprop leads when given better optimizers or more epochs, and Dust is far more computationally expensive, so its significance lies in demonstrating feasibility rather than replacing backprop.
Coverage timeline
机器之心机器之心
编辑｜张倩 训练大模型，能不能不用反向传播？ 这事一直有人琢磨。只是反向传播太成功了：过去这些年，反向传播一直是训练神经网络的唯一主流算法。我们今天所有的架构、优化器、甚至 GPU 硬件，某种意义上都是围着它长出来的。 但一直有人不信这个邪：反向传播要求整个网络处处可微，这其实是个很强的限制，不可微的模块、离散的计算、外部程序，统统进不了训练回路。 那么，如果算力足够便宜，能不能干脆不「算」梯度了，改用一种最原始的思路 —— 随机把网络改一改，看它变好还是变坏，变好的方向就保留，变坏的就扔掉。这种办法笨是笨，但胜在什么都不挑：不管网络内部长什么样、可不可微，它都能用。 一家叫 Q Labs 的小型研究实验室最近尝试了类似方法，并将方法命名为「 Dust 」。 他们声称，这是 第一个在 Transformer 预训练任务上能与反向传播正面竞争的零阶（zeroth-order）优化方法 。 它的大致思路是：不碰权重，而是给每一层输出的激活值加高斯噪声，而且在每个 token 上独立地加。这样每个 token 都相当于一个「虚拟实验个体」，一次前向传播就能并行评估几千个个体。这就是他们所谓的「 虚拟种群 」（virtual population）。哪个噪声让 loss 降了，就给它记功；把记功后的噪声加权平均，就得到了梯度估计。 Q Labs 宣称，Dust 在大规模种群（即计算量显著更大）时能很好地近似反向传播，并且在多种设置下甚至超越了它。他们认为，「这暗示在计算资源充裕的条件下，我们或许能够超越反向传播。」 但瓶颈也很明显：scale。作者自己说得很直白 ——Dust 目前还远比反向传播贵，要再便宜几个数量级才有资格谈「替代」。论文里的实验最大也只到 243M 参数、20M token 的训练预算，和工业界动辄千亿 token 的预训练相去甚远。 来自 SOTAAZ Lab 的第三方复现也为 Dust 提供了一个更冷静的观察角度：在论文设定的一轮 SGD 比较下，Dust 的结果基本可以复现，但它与反向传播的差距很小，并未显示出论文所强调的明显优势；一旦给反向传播换上更合适的优化器，或允许其在同一数据上多训练几轮，反向传播便明显领先，而且计算成本远低于 Dust。换言之， Dust 的意义更在于证明零阶方法可以用于 Transformer 预训练，而不是在当下替代反
Related stories
Dust: Zeroth-Order Pretraining Method Rivals Backprop for Transformer LMs
A research correspondence introduces Dust, the first zeroth-order pretraining method for transformer language models that is competitive with backpropagation. Dust perturbs activations at every token, treating each token as a virtual population member and evaluating them in parallel in a single forward pass, and is orders of magnitude more efficient than weight-space evolutionary strategies. The method even exceeds backprop in some settings, and larger models (e.g., 243M parameters) appear more population-efficient, challenging the assumption that zeroth-order methods do not scale.
1 reports · 1 sources277 points · 82 comments
Tsinghua and Qujing Introduce KTransformers-FineTune for CPU-GPU MoE Fine-Tuning
Tsinghua University's MADSys lab and Qujing Technology, with LlamaFactory, introduced KTransformers-FineTune (KT-FT), a CPU-GPU co-computation method for fine-tuning mixture-of-experts (MoE) models. It claims 2.5-21.2x throughput gains over traditional offloading by having the CPU compute expert weights directly instead of transferring them to the GPU. The approach includes a semantic-aware proactive layout engine and dynamic thread scheduling for routing skew, and is fully open-sourced.
1 reports · 1 sources
LoopVL open-source VLM shows data efficiency with 260x less training data
Researchers from Renmin University, Baidu, and others released LoopVL, an open-source vision-language model based on the Loop Transformer. Despite using only 0.14T training tokens versus about 36T for Qwen3.5-2B and InternVL3.5-2B, LoopVL matches or exceeds them on benchmarks like LogicVista, MMMU-Pro, and MathVision, demonstrating high data efficiency. The team also observed cross-loop visual attention reorganization, termed 'Visual Aha Moment'.
DeepSeek details DSec sandbox infrastructure for agentic training in Zhihu post and arXiv paper
DeepSeek published a technical post on Zhihu detailing DeepSeek Elastic Compute (DSec), the sandbox infrastructure supporting all training, evaluation, and data preprocessing for DeepSeek-V4. The accompanying arXiv paper, co-authored with Tsinghua University and involving over 130 authors including Liang Wenfeng, describes DSec's design for agentic training workloads, including four execution backends (FnCall, Container, MicroVM, Full VM) accessed via a unified Python SDK (libdsec).
1 reports · 1 sources
News updates every hour. Subscribe in your RSS reader to get new stories automatically.
Subscribe