# Revisiting the 16x Pretraining Compute Gap between Diffusion and Autoregressive Language Models

> The widely cited 16x compute gap between diffusion and autoregressive language models is overstated, shrinking to just 3x with exact likelihood scoring, block diffusion, and encoder-decoder architectures.

- **Source:** [arXiv](https://arxiv.org/abs/2610.17590)
- **Published:** 2026-10-17
- **Permalink:** https://picx.dev/p/f7e9QL
- **Whiteboard:** https://picx.dev/p/f7e9QL/image

## Summary

## Summary (Overview)

- **Revisits the widely cited "16×" compute gap** between diffusion language models (DLMs) and autoregressive language models (ARMs), showing that this figure substantially overstates the pretraining cost of DLMs.
- **Identifies three methodological flaws** in the original SMDM study (Nie et al., 2025): scoring with the evidence lower bound (ELBO) instead of exact likelihood, training full-sequence decoder-only models instead of block diffusion, and using an outdated FLOP count.
- **Proposes a revised protocol** incorporating exact likelihood evaluation (DUEL), block diffusion training (BD3LM), an encoder–decoder architecture (E2D2), and a sweep over practical block sizes $b \in \{4, 8, 16, 32\}$.
- **Reduces the compute multiple from 17.8× to 3.0×** through a combination of exact likelihood scoring, block diffusion, encoder–decoder architecture, and smaller block sizes.
- **Trains ~1,700 models** across ten model variants—the largest compute-matched scaling analysis of diffusion language models to date.

## Introduction and Theoretical Foundation

### Background

Masked diffusion models (MDMs) have emerged as the leading non-autoregressive alternative for language modeling, offering fast parallel generation—about **2.5× the generation throughput** of ARMs at comparable accuracy, with commercial systems reporting up to **10×**. However, their training efficiency has been widely believed to be far worse than ARMs, primarily due to the influential IsoFLOP study by Nie et al. (2025) (SMDM), which concluded that an MDM needs **~16× the training FLOPs** of an ARM to reach the same validation perplexity.

### Theoretical Foundation

The paper identifies three key methodological issues with the original 16× measurement:

**1. ELBO vs. Exact Likelihood.** The SMDM study scored MDMs using the negative evidence lower bound (ELBO):
$$\mathcal{L}_{\text{ELBO}}(x_{1:n}) = \mathbb{E}_{t \sim U(0,1)} \mathbb{E}_{z^t_{1:n} \sim q_t(\cdot|x_{1:n})} \left[ \frac{1}{t} \sum_{\ell: z^t_\ell = m} -\log p_\theta(x_\ell | z^t_{1:n}) \right] \geq -\log p_\theta(x_{1:n})$$

This is only an upper bound on the negative log-likelihood (NLL) and scores the model under uniformly random token orders it never uses at inference. The paper instead uses **DUEL** (Turok et al., 2026), which computes the exact NLL under the model's actual decoding order (top-1 confidence rule). Under a deterministic unmasking rule $\pi$, the exact likelihood factorizes as:
$$p^\pi_\theta(x_{1:n}) = \prod_{i=1}^{n} p_\theta(x_{\sigma_i} | x_{\sigma_{<i}})$$

**2. Full-sequence vs. Block Diffusion.** Nearly all large-scale DLMs now use block-wise factorization (BD3LM; Arriola et al., 2025) with block sizes between 4 and 32, rather than operating over the full sequence. Block diffusion partitions $x_{1:n}$ into contiguous blocks of $b$ tokens, factorizes autoregressively over blocks, and models each block with a masked diffusion model conditioned on the clean prefix.

**3. Decoder-only vs. Encoder–Decoder Architecture.** The encoder–decoder architecture (E2D2; Arriola et al., 2026) separates the processing of clean and noised tokens, splitting an $L$-layer transformer into an $L_{\text{enc}}$-layer encoder and an $L_{\text{dec}} = L - L_{\text{enc}}$-layer decoder, offering computationally cheaper training than the decoder-only network.

## Methodology

### Revised Scaling Law Protocol

The paper makes three advances and two corrections to the SMDM protocol:

**Advances:**
1. **Exact likelihood evaluation**: All diffusion models scored with DUEL under the top-1 confidence unmasking rule; ELBO also computed with 128-particle Monte-Carlo for comparison.
2. **Block diffusion**: Models trained at block sizes $b \in \{4, 8, 16, 32\}$, chosen based on the roofline model for GPU compute-bound serving.
3. **Encoder–decoder architecture**: E2D2 models trained at the same four block sizes, with decoder-depth sweeps at every grid point.

**Corrections:**
1. **Exact FLOP counting**: Instead of the coarse $6N$ rule (Kaplan et al., 2020), the paper counts actual training FLOPs including output head and attention score computation. Per-sample training FLOPs:
   - ARM: $6Pn + 6dL \cdot n(n+1) + 6dVn$
   - MDM: $6Pn + 12dL \cdot n^2 + 6dVn$
   - BD3LM: $12Pn + 12dL(n^2 + nb) + 6dVn$
   - E2D2: $6P_{\text{enc}}n + 6dL_{\text{enc}}(n^2 + nb) + 6P_{\text{dec}}n + 12dL_{\text{dec}} \cdot nb + 6dL_{\text{dec}}(n^2 - nb) + 6dVn$

2. **In-range compute multiple readout**: Compute multiples read within the measured range of the scaling laws rather than by extrapolation.

### Training Setup

- **Architecture**: LLaMA-style transformers, 19M–2.1B non-embedding parameters
- **Data**: SlimPajama, sequence length 2048, batch of 256×2048 tokens
- **Optimizer**: AdamW ($\beta = (0.9, 0.95)$, weight decay 0.1, gradient clipping at 1.0), peak LR $4 \times 10^{-4}$, linear warm-up over 1% of steps, cosine decay to 1/10 of peak
- **Budgets**: Five budgets $\{6 \times 10^{18}, 10^{19}, 3 \times 10^{19}, 6 \times 10^{19}, 10^{20}\}$ FLOPs
- **Total**: ~1,700 trained models, ~136k B200 GPU hours

### Fitting Procedure

For each variant and budget, a centered parabola is fitted in $\log_{10} N$ space to find the compute-optimal NLL $L^*(C)$. Across budgets, the power law is fitted:
$$L^*(C) = aC^\alpha$$

The compute multiple is read by finding the compute at which the ARM's fitted law reaches the diffusion variant's $L^*$ at $10^{20}$ FLOPs.

## Empirical Validation / Results

### Main Scaling Law Results

**Table 1: Fitted laws $L^*(C) = aC^\alpha$ and compute multiples**

| Variant | a | α | Multiple |
|---------|-----|---------|----------|
| ARM | 40.95 | −0.0618 | — |
| MDM (ELBO) | 49.93 | −0.0623 | 17.8 |
| MDM | 57.87 | −0.0659 | 12.8 |
| BD3LM b=32 | 55.54 | −0.0654 | 9.6 |
| E2D2 b=32 | 47.79 | −0.0624 | 7.8 |
| BD3LM b=16 | 55.56 | −0.0657 | 7.8 |
| E2D2 b=16 | 49.26 | −0.0634 | 6.3 |
| BD3LM b=8 | 54.34 | −0.0656 | 5.9 |
| E2D2 b=8 | 48.52 | −0.0635 | 4.6 |
| BD3LM b=4 | 52.99 | −0.0655 | 4.1 |
| E2D2 b=4 | 45.50 | −0.0627 | 3.0 |

Key findings:
- **ELBO baseline**: 17.8× multiple, comparable to the original 16× figure
- **Exact likelihood (DUEL)**: Reduces multiple to 12.8×—about a third of the gap is an evaluation artifact
- **Block diffusion**: BD3LM reduces to 9.6× at b=32, 7.8× at b=16, 5.9× at b=8
- **Encoder–decoder**: E2D2 further reduces to 7.8× at b=32, 6.3× at b=16, 4.6× at b=8, and 3.0× at b=4

**Table 2: Compute multiples at 10²⁰ FLOPs—DUEL vs. ELBO**

| | MDM | BD3LM 32 | BD3LM 16 | BD3LM 8 | BD3LM 4 | E2D2 32 | E2D2 16 | E2D2 8 | E2D2 4 |
|---|---|---|---|---|---|---|---|---|---|
| **DUEL** | 12.8 | 9.6 | 7.8 | 5.9 | 4.1 | 7.8 | 6.3 | 4.6 | 3.0 |
| **ELBO** | 17.8 | 16.5 | 13.6 | 10.1 | 6.3 | 12.0 | 9.8 | 7.1 | 4.2 |

### Optimal Decoder Depth

The optimal decoder fraction depends on the block size alone, following the empirical rule:
$$L^*_{\text{dec}}/L \approx 0.556 + 0.043 \log_2 b$$

This rule is near-optimal: at 90 of 104 grid points with $b \in \{8, 16, 32\}$, the DUEL of the rounded prediction lies within 5 m-nats of the best split found by the sweep.

### Held-out Budget Validation

At a held-out budget of $3 \times 10^{20}$ FLOPs (three times larger than the largest fitted budget), the gap between variants persists:
- ARM–E2D2 gap at b=16: **261.6 m-nats measured** vs. 257 predicted
- ARM–E2D2 gap at b=32: **296.8 m-nats measured** vs. 294 predicted

All variants show a common upward deviation of ~1.1% from pure power-law extrapolation, suggesting a mild misspecification (small irreducible term), but the gap between variants is insensitive to this.

### Downstream Benchmarks

**Table 3: DUEL vs. ELBO accuracy difference (mean ± s.d. over five budgets)**

| | b=4 | b=8 | b=16 | b=32 |
|---|---|---|---|---|
| **MDM** | | | | +0.4 ± 0.1 |
| **BD3LM** | +1.0 ± 0.4 | +0.9 ± 0.2 | +0.9 ± 0.2 | +0.8 ± 0.2 |
| **E2D2** | +1.0 ± 0.5 | +1.1 ± 0.4 | +1.0 ± 0.3 | +1.0 ± 0.2 |

DUEL yields statistically significantly higher accuracy than the ELBO for every diffusion variant. The order in benchmark accuracy broadly matches the order in NLL: ARM > E2D2 b=4 > E2D2 b=32 > BD3LM b=32 > MDM, confirming that likelihood improvements carry over to downstream accuracy.

## Theoretical and Practical Implications

### Implications for DLM Pretraining

1. **The 16× figure is substantially overstated**: The true compute gap between DLMs and ARMs is between 3.0× and 12.8× depending on the design choices, making pretraining DLMs from scratch far more viable than previously assumed.

2. **Evaluation methodology matters**: Using the ELBO instead of exact likelihood under the model's decoding order inflates the apparent gap by about a third. DUEL is both theoretically better (exact likelihood under deployment conditions) and empirically better (higher downstream accuracy).

3. **Architecture design choices are crucial**: The encoder–decoder architecture (E2D2) consistently outperforms decoder-only models at matched compute, suggesting that the conceptual split between encoding and decoding is beneficial for diffusion language models.

4. **Block size is a key hyperparameter**: Smaller block sizes reduce the compute gap, with practical implications for serving (block sizes 8–32 put GPU compute units to use during decoding).

### Practical Guidance

- **Optimal decoder depth rule**: Practitioners can use $L^*_{\text{dec}}/L \approx 0.556 + 0.043 \log_2 b$ to set the encoder–decoder split without expensive sweeps.
- **Exact FLOP accounting**: The $6N$ rule distorts scaling law comparisons across variants; attention score computation accounts for 5–30% of training FLOPs and differs markedly across variants.

## Conclusion

This paper revisits the scaling laws of diffusion language models and demonstrates that the widely cited 16× compute gap to ARMs substantially overstates the pretraining cost. Through a revised protocol incorporating exact likelihood evaluation (DUEL), block diffusion training (BD3LM), encoder–decoder architecture (E2D2), and a sweep over practical block sizes, the compute multiple is reduced from 17.8× to as low as **3.0×** (E2D2 at b=4). The paper also provides an empirical rule for optimal decoder depth, validates results at a held-out budget, and shows that likelihood improvements carry over to downstream accuracy.

**Future directions** include:
- Fitting scaling laws with an irreducible term (the consistent miss at the held-out budget suggests one is present)
- Exploring other unmasking rules beyond top-1 confidence
- Hyperparameter tuning at every scale (the paper keeps the SMDM recipe for consistency)
- Extending the held-out budget validation to b=8 and other block sizes

**Limitations**: single fixed random seed, only top-1 confidence sampler tested, single learning-rate and batch-size configuration, pure power-law fits without irreducible term.

---

_Markdown view of https://picx.dev/p/f7e9QL, served by PicX — AI-generated visual whiteboard summaries of research papers._