Back to News

Dust: Zeroth-Order Pretraining Method Rivals Backprop for Transformer LMs

277 points · 82 comments#zeroth-order#pretraining#backprop#evolutionary-strategies

A research correspondence introduces Dust, the first zeroth-order pretraining method for transformer language models that is competitive with backpropagation. Dust perturbs activations at every token, treating each token as a virtual population member and evaluating them in parallel in a single forward pass, and is orders of magnitude more efficient than weight-space evolutionary strategies. The method even exceeds backprop in some settings, and larger models (e.g., 243M parameters) appear more population-efficient, challenging the assumption that zeroth-order methods do not scale.

Coverage timeline

  1. Hacker NewsE-Reverance

    October 2026 Correspondence to s@qlabs.sh·Code· ## TL;DR * We present the first zeroth-order method that is competitive with backprop at pretraining transformer language models. Dust perturbs activations (node perturbation) independently at every token, so each token is a _virtual_ population member and one forward pass evaluates them all in parallel. * Dust approximates backprop closely at large population (i.e. substantially more compute) and in multiple settings even _exceeds_ it. This hints that in a compute-rich regime we might be able to surpass backprop. * Dust is orders of magnitude more efficient than weight-space ES. From 1M tokens up, Dust is on the order of $1 0^{3}$ to $1 0^{4}$ times more efficient than a transformer implementation of EGGROLL, a state-of-the-art ES method, based on our extrapolations. * Zeroth-order methods are widely believed not to scale to large networks. Strikingly, we find larger models are more population-efficient, not less: a 243M-parameter model

Related stories

  • Q Labs proposes Dust, a zeroth-order optimizer that rivals backprop in Transformer pretraining up to 243M params

    Q Labs, a small research lab backed by Jeff Dean, published a paper introducing Dust, a zeroth-order optimization method that adds Gaussian noise to activations per token and estimates gradients via a virtual population. They claim it is the first such method to compete with backpropagation in Transformer pretraining, with experiments up to 243M parameters and 20M tokens. Third-party replication from SOTAAZ Lab confirmed basic results but found backprop leads when given better optimizers or more epochs, and Dust is far more computationally expensive, so its significance lies in demonstrating feasibility rather than replacing backprop.

    1 reports · 1 sources

News updates every hour. Subscribe in your RSS reader to get new stories automatically.

Subscribe