# Do Halting Heads Really Save Compute in Looped Language Models?

> Halting heads in looped language models save far less compute than reported because mixture-trained heads rarely stop tokens and batching overhead eliminates most remaining gains.

- **Source:** [arXiv](https://arxiv.org/abs/2610.32246)
- **Published:** 2026-11-21
- **Permalink:** https://picx.dev/p/m4Hdcb
- **Whiteboard:** https://picx.dev/p/m4Hdcb/image

## Summary

## Summary (Overview)

- This paper investigates whether halting heads in looped language models actually save compute as claimed, finding significant discrepancies between predicted savings and realized compute reductions.
- The authors train halting heads in looped models from 140M to 1.3B parameters and identify two fundamental problems: (1) heads trained with the mixture loss predict tokens need under half the iterations yet run almost every iteration under standard stopping rules, and (2) batching removes most of the remaining savings.
- The root cause is traced to the training loss: the mixture loss never scores the single iteration where a token stops, while per-step scoring works only with a strong prior penalty favoring early stops.
- Even when heads successfully stop tokens early, batched generation (where sequences run until the slowest token stops) eliminates most of the advantage.
- The authors provide a pre-training test that bounds the potential gain of a halting head and recommend standardized reporting of three numbers: expected halting steps, realized steps, and batched-pass iterations.

## Introduction and Theoretical Foundation

Looped language models apply the same transformer block multiple times to each token, with each application called an *iteration*. Unlike stacked transformers that use different layers, looped models share weights across depth, meaning extra depth costs no additional parameters. This architecture invites adaptive computation: easy tokens could stop after few iterations while hard tokens run the full cap.

To enable adaptive stopping, models add a **halting head**: a small layer that reads the token's state after each iteration and outputs the probability that the token should stop. Two standard designs exist:

- **Adaptive Computation Time (ACT)** (Graves, 2016; Dehghani et al., 2019)
- **PonderNet** (Banino et al., 2021)

The paper identifies a critical assumption underlying reported savings: that the average number of iterations per token equals the compute actually spent. This assumption fails for two reasons:

1. **Average vs. percentile**: The halting head outputs probabilities, but an exit rule stops tokens at a percentile of those probabilities, not their average. A token with expected halting steps of 3.7 iterations can run all 8 if more than ε of its probability mass sits on the last iteration (Figure 1b).

2. **Batching**: Tokens are processed in batches, and a batch runs until its *deepest* token stops. Every token in the batch pays for that slowest token's iterations (Figure 2a).

The paper formalizes this with notation: let $h_{i,t}$ be the state of token $i$ after $t$ iterations, and $N_i$ be the iteration at which token $i$ actually stops under the exit rule.

## Methodology

### Models and Training

The authors train four scales of looped models (140M to 1.3B parameters) on two corpora: FineWeb-Edu (FW) and Common Corpus (CC). They compare several methods:

| Method | Training depth | What the coda reads | Loss term on depth |
|--------|---------------|-------------------|-------------------|
| Dense | no sharing | final layer | — |
| Sampled-depth | T ~ Poisson, mean 8 | $h_{i,T}$ | — |
| ACT | fixed T=8 | mixture up to halting iteration | ponder cost, weight $\tau_p = 10^{-4}$ |
| Mixture | fixed T=8 | $\sum_t p_{i,t} h_{i,t}$ | KL to geometric $\kappa=4$, weight $\beta$ |
| Per-step | fixed T=8 | each $h_{i,t}$, weighted by $p_{i,t}$ | KL to geometric $\kappa=4$, weight $\beta$ |

The halting head outputs $\lambda_{i,t} = \sigma(w^\top h_{i,t})$, giving stopping probability $p_{i,t} = \lambda_{i,t} \prod_{u<t}(1-\lambda_{i,u})$.

Two loss functions are compared:

- **Mixture loss**: scores only the mixture prediction $\bar{y}_i$ decoded from $\bar{h}_i = \sum_t p_{i,t} h_{i,t}$
- **Per-step loss** (PonderNet's own): scores every iteration with prediction $\hat{y}_{i,t}$ from $h_{i,t}$ alone

$$\text{mixture: } L(y_i, \bar{y}_i) + \beta \text{KL}(p_i \| p_G)$$
$$\text{per-step: } \sum_t p_{i,t} L(y_i, \hat{y}_{i,t}) + \beta \text{KL}(p_i \| p_G)$$

where $p_G$ is a geometric prior favoring early iterations with mean ~3 (for $\kappa=4$, $T=8$).

### Three Numbers for Reporting

The paper introduces three metrics:
1. **$E[N]$**: expected halting steps (what the head predicts)
2. **$\bar{N}$**: realized steps (what the exit rule actually runs)
3. **$R\bar{N}$**: iterations a batched pass pays (with $R \geq 1$ being the overhead factor)

## Empirical Validation / Results

### Heads Trained on a Mixture Save Almost No Compute

The mixture-loss heads predict fewer than 4 of 8 iterations on average, yet tokens run 7.6–7.9 iterations under a 90% exit rule (ε=0.1). The sampled-depth baseline overtakes the mixture-loss model within one iteration of the cap.

**Table 1: What the head predicts is not what a batched pass pays**

| Scale | $E[N]$ | $\bar{N}$ | $R\bar{N}$ | ΔBPB at $\bar{N}$ | ΔBPB at $R\bar{N}$ (hidden) | ΔBPB at $R\bar{N}$ (reused) |
|-------|--------|-----------|------------|-------------------|---------------------------|---------------------------|
| Mixture, ε=0.1 | | | | | | |
| 140M | 3.81 | 7.86 | – | −0.0093 | – | – |
| 370M | 3.74 | 7.76 | 8.0 | −0.0063 | – | – |
| 770M | 3.63† | 7.67 | – | −0.0065 | – | – |
| 1.3B | 3.61† | 7.61 | – | −0.0048 | – | – |
| Per-step, β=0.1, ε=0.7 | | | | | | |
| 140M | 4.33 | 3.11 | 4.71 | −0.0205 | −0.0004 | +0.043 |
| 370M | 4.07 | 2.83 | 4.40 | −0.0219 | −0.0010 | +0.035 |
| 770M | 4.00 | 2.75 | 4.37 | −0.0246 | −0.0030 | +0.028 |

### The Training Loss Is the Cause

**Reading out one iteration costs the mixture-loss model ~0.2 BPB** (Table 3):

| Scale | Reads mixture | Reads state at cap | Readout gap |
|-------|--------------|-------------------|-------------|
| Mixture | | | |
| 140M | 1.2058 ± .0006 | 1.4091 ± .040 | +0.203 ± .040 |
| 370M | 1.0807 ± .0001 | 1.2559 ± .015 | +0.175 ± .015 |
| Per-step | | | |
| 140M | 1.2050 ± .0010 | 1.2048 ± .0009 | −0.0002 ± .0001 |
| 370M | 1.0924 ± .0015 | 1.0914 ± .0016 | −0.0010 ± .0004 |

**Scoring every iteration is not enough without a strong prior** (Table 4):

| Scale | Loss | BPB ↓ | change at β=0.1 | change at β=1.0 |
|-------|------|-------|-----------------|-----------------|
| 140M | Mixture | 1.813 | +0.038 | +0.052 |
| 140M | Per-step | 1.711 | −0.394 | −0.450 |
| 370M | Mixture | 1.610 | −0.002 | +0.031 |
| 370M | Per-step | 1.430 | −0.273 | −0.308† |

### Batching Removes Most of the Remaining Lead

When 32 sequences are decoded together, the per-step model's tokens stop after ~3 iterations on average, but a pass runs 4.4–4.7 iterations. The lead against the sampled-depth baseline shrinks from ~0.02 BPB to near zero (Table 1).

**Shuffling depths costs 0.008 BPB** (Table 5), showing that allocation (giving more iterations to harder tokens) matters, but depth-robustness is most of the lead.

### A Test Before Training a Head

The paper provides a pre-training test based on three quantities:
- $f$: share of per-token compute inside the core
- $R$: batching overhead factor
- **depth curve**: the sampled-depth baseline's BPB at each fixed depth

The cost of running $t$ of $T$ iterations is $c(t) = (1-f) + ft/T$, giving architecture speedup $S_{arch}(t) = 1/c(t)$. A head pays off only if $S_{arch}(R\bar{N}) > 1$ and the model beats the sampled-depth baseline at $R\bar{N}$ iterations.

For a head aiming at 3 iterations, the bound is ~0.03 BPB for single-sequence decoding, falling to ~0.01 BPB for 32-sequence batched decoding.

## Theoretical and Practical Implications

The paper's findings have significant implications for the design and evaluation of adaptive computation in language models:

1. **Reporting standards**: The authors recommend that looped models with halting heads report three numbers: (i) expected halting steps, (ii) realized steps under a stated exit rule, and (iii) iterations a batched pass runs. They also recommend stating how stopped tokens' keys and values are filled.

2. **Training methodology**: The mixture loss is fundamentally flawed for training halting heads because it never scores the single state an exit rule uses. The per-step loss with a strong prior is necessary but not sufficient.

3. **Batching overhead**: Reported average iterations per token can overstate savings by a factor of $R$ (which grows with batch size). This affects not just the authors' models but also released models like Ouro.

4. **Practical guidance**: The pre-training test (Section 8) can rule out halting heads before expensive training runs, potentially saving substantial compute.

## Conclusion

The paper demonstrates that for the halting heads they trained, the average number of iterations per token is not the compute a head saves. Heads trained with the mixture loss run tokens almost to the cap; heads scored at every iteration under a strong prior stop tokens early, but a batched pass pays for its deepest token, and at that cost most of their lead disappears.

**Key recommendations:**
1. Report three numbers with the exit rule's threshold and batch size
2. Report quality when generating (real or simulated decode), stating how early-stopped tokens' deeper keys/values are filled
3. Compare at $R\bar{N}$ iterations, not at the average, against sampled-depth baselines and specialists
4. Run the pre-training test before training a head

**Limitations** acknowledged by the authors:
- Results are for flat depth curves; steeper curves may change the test's verdict
- Evaluation uses BPB on validation splits, not downstream tasks
- Per-step model trained only up to 770M parameters
- Some single-seed cells in the experiments
- The mechanism by which key-value reuse works is not fully established

---

_Markdown view of https://picx.dev/p/m4Hdcb, served by PicX — AI-generated visual whiteboard summaries of research papers._
