# How Model Growth, Recursion, and Boundary Operators Influence Scaling Exponents

> Architectural interventions like model growth and boundary operators improve scaling exponents, with grown models matching GPT-3 13B using roughly 20x less compute.

- **Source:** [arXiv](https://arxiv.org/abs/2609.19107)
- **Published:** 2026-10-17
- **Permalink:** https://picx.dev/p/vOOi07
- **Whiteboard:** https://picx.dev/p/vOOi07/image

## Summary

# How Model Growth, Recursion, and Boundary Operators Influence Scaling Exponents

## Summary (Overview)

- **Contrary to conventional wisdom**, this paper demonstrates that architectural interventions can modify scaling exponents in pretraining, leading to power-law improvements in performance as computation increases—not merely constant-factor improvements.
- **Model growth** (increasing the number of core passes during training) provides the biggest changes to scaling exponents. A 7.4B parameter grown model matches GPT-3 13B on CORE with roughly **20× less compute**.
- **A simple boundary operator**—which normalizes and injects an earlier block—also improves the scaling exponent, though to a lesser extent than model growth.
- **In the data-constrained, multi-epoch setting**, looping has a useful regularizing effect, and it becomes compute-optimal to increase the number of loops with scale.
- The results are explained through the lens of **computational depth**: for a given computational budget, increasing the usable depth of the transformer leads to efficiency gains that increase with scale.

## Introduction and Theoretical Foundation

### Background

Scaling laws predict loss as a function of parameters and datapoints, providing a recipe for configuring the compute-optimal frontier. The standard form (Hoffmann et al., 2022) decomposes loss into three terms:

$$L(N, T) = E + (N/N_0)^{-\alpha} + (T/T_0)^{-\beta}$$

where $N$ is parameter count, $T$ is training tokens, $E$ is the irreducible loss floor, and $\alpha$, $\beta$ are fitted exponents. Training compute follows $C \approx 6NT$ (Kaplan et al., 2020).

The compute-optimal loss is itself a power law:

$$L(C) = E + A\left(\frac{C}{C_0}\right)^{-\gamma}$$

where $A$ and $\gamma$ are fitted, and $C_0$ is a chosen base compute.

### Key Concepts

**Compute multiplier**: To compare architectures, the paper defines the compute multiplier $\pi(\ell) = \hat{C}_A(\ell)/\hat{C}_B(\ell)$, where $\hat{C}_A(\ell)$ is the compute architecture $A$ needs to reach loss $\ell$. A rising multiplier indicates an exponent improvement; a flat multiplier indicates only a constant improvement.

**Computational depth**: Defined as the number of blocks that meaningfully influence the predictive distribution. Executed depth and computational depth need not coincide.

**The curse of depth**: In pre-norm transformers, the residual stream grows with depth, so each block's update becomes a shrinking fraction of the stream, and deeper blocks drift toward doing nothing.

### Motivation

The authors hypothesize that:
1. The boundary operator improves the exponent because it closes the gap between executed and computational depth (fixing the curse of depth).
2. Model growth improves the exponent because a fixed-depth model pays for depth it does not yet need early in training.
3. Both wastes (unused depth and premature depth) take up a larger fraction of depth at larger depths, and since compute-optimal depth rises with budget, both should grow with scale.

## Methodology

### Architecture Family: Prelude–Core–Coda

All models follow a three-stage network:

$$e = P(s), \quad h_k = R_{\theta_k}^{\phi}(h_{k-1}, e), \quad k = 1, \ldots, K, \quad y = C_{\rho}(h_K, e)$$

where $P$, $R$, and $C$ are prelude, core, and coda blocks; $\phi$ and $\rho$ are boundary operators; $\theta_k$ are weights of the $k$-th pass; and $K$ is the loop count.

### Boundary Operator

$$\text{BO}(h, e) = \text{Norm}(h) + \alpha e$$

This operator normalizes the residual stream and re-injects the prelude output $e$, addressing the curse of depth.

### Architecture Variants

Six primary architectures are compared, arranged as three matched comparisons:

| Comparison | Variants | What's Isolated |
|---|---|---|
| Boundary operator | Vanilla vs. Operator-1 | Effect of BO (matched params & FLOPs) |
| Weight sharing | Loop-2 vs. Untied-2 | Effect of tying (matched FLOPs & depth) |
| Model growth | Loop-Grow vs. Untied-Grow | Effect of growth (vs. fixed counterparts) |

### Training Procedure

Each architecture receives a **compute-optimal recipe** fitted in four stages:
1. **Base hyperparameters** tuned at depth 8 on 1B tokens
2. **Tokens per stored parameter** fitted via iso-compute sweeps
3. **Growth timing** (for growth variants) via transition-point sweeps
4. **Learning-rate scaling rule**: $\text{GLR}(N) = \text{GLR}_{d_8}(N/N_{d_8})^{\beta}$

### Experimental Setup

- **Data**: FineWeb and FineWeb-Edu (Penedo et al., 2024)
- **Tokenizer**: GPT-2 (50,257 vocabulary)
- **Context length**: 2048 tokens
- **Global batch size**: 524,288 tokens
- **Optimizer**: Muon for matrix parameters; AdamW for embeddings and LM head
- **Width-depth ratio**: Fixed at 128
- **Compute range**: ~10¹⁸ to 10²⁰ FLOPs for fitted ladders

## Empirical Validation / Results

### Single-Epoch Compute-Optimal Scaling

**Fitted exponents** (lower is better for loss, higher exponent γ is better):

| Architecture | Exponent γ | Compute Multiplier at 10²⁰ FLOPs |
|---|---|---|
| Vanilla | 0.111 | 1.00× (baseline) |
| Deep Vanilla | 0.111 | ~1.08× |
| Operator-1 | 0.114 | 1.25× |
| Loop-2 | 0.114 | — |
| Untied-2 | 0.114 | 1.34× |
| Loop-Grow | 0.116 | 1.36× |
| Untied-Grow | 0.117 | 1.55× |

### Key Findings

1. **The boundary operator improves the exponent**: Operator-1 matches Vanilla in parameters and FLOPs but its compute multiplier rises from 1.12× at 10¹⁸ to 1.25× at 10²⁰ FLOPs.

2. **Model growth improves the exponent**: Untied-Grow's multiplier widens from 1.30× at 10¹⁸ to 1.55× at 10²⁰ FLOPs. Growth's multiplier over Untied-2 grows from 1.09× to 1.16×.

3. **Untying improves only the constant**: Untied-2 sits ~1.06–1.08× above Loop-2 at all scales—a fixed factor, not a growing one.

4. **Extrapolation validation**: A 7.4B Untied-Grow model trained at 8× the largest fitted compute lands on the predicted loss curve (0.03 below forecast). It reaches 0.3865 CORE, matching GPT-3 13B's 0.3852, with ~20× less compute.

### Multi-Epoch, Data-Constrained Setting

- **Optimal loop count grows with compute**: At matched compute, the optimal loop count rises from ~1.4 at the smallest budget to ~6.7 at the largest.
- **Scaling loop count beats scaling model size**: Scaling loops at fixed model size and fixed weight decay matches the best tuned model-size-scaling loss with **2.2× less compute**.
- **Looping leaves optimal weight decay nearly unchanged**: The optimal weight decay rises with depth but is nearly flat in loop count.

### KL Effective Depth Results

- The boundary operator increases KL effective depth: Untied-2 reaches 24 vs. 20 for Deep Vanilla at 10²⁰ FLOPs (same executed blocks).
- Model growth raises computational depth: Untied-Grow reaches 36 vs. 24 for Untied-2 at 10²⁰ FLOPs.
- In multi-epoch training, scaling loop count raises KL effective depth faster than scaling depth at K=1 (18 vs. 14 layers at the largest budget).

## Theoretical and Practical Implications

### Theoretical Interpretation

The results support the **computational depth hypothesis**:

1. **Boundary operator**: Normalizing the residual stream and re-injecting the input lets every block write at full relative weight, closing the gap between executed and computational depth. Evidence: Operator-1 sits only slightly above Vanilla at shallow depths, and the gap widens as models get deeper.

2. **Model growth**: Networks fit simple structure early and complex structure late in training. Growth timing sweeps have interior minima—paying for depth too early or too late both cost loss. Grown models prefer smaller starting models trained on more tokens (TPP rising from 6 to 7–8).

3. **Looping as regularization**: Models overfit with stored parameters, not executed blocks. Optimal weight decay tracks stored parameters (not executed depth), and the marginal gains of looping are independent of stored parameters.

### Practical Implications

- **Architecture-specific tuning is crucial**: Transferring Vanilla's recipe to Operator-1 costs 7.8×10⁻³ loss at depth 8 and erases its exponent improvement along the ladder.
- **Simple recipe prescription**: Calibrate architecture and growth schedule at small scale, then reuse at larger budgets. Fixed growth fraction (~0.3) and rounded TPP values work well.
- **Weight tying**: Useful when parameter storage is the priority—retains exponent improvement from growth while trading a constant factor of compute efficiency.
- **Data-constrained regime**: Increasing loop count with scale becomes compute-optimal as data becomes a more constrained resource.

## Conclusion

This paper demonstrates, contrary to conventional wisdom, that **architectural interventions can influence scaling exponents** in pretraining. Both model growth and a simple boundary operator provide compute efficiency gains that increase with scale. The key insight is **computational depth**: for a given computational budget, we want to increase the usable depth of the transformer.

### Future Directions

- **Other growable dimensions**: Context length, number of experts in MoE models, and width all grow with compute but are fixed before training begins. Growing them on the schedule the network needs may change the exponent.
- **Staged schedules**: Two passes, then four, then six may increase the exponent further.
- **Evaluation standard**: The distinction between constant and exponent improvements should become a standard part of evaluating new architectures—a better constant saves the same factor at every scale, while a better exponent saves a factor that compounds as budgets grow.

### Key Equations Summary

The paper's central scaling law:

$$L(C) = E + A\left(\frac{C}{C_0}\right)^{-\gamma}$$

The boundary operator:

$$\text{BO}(h, e) = \text{Norm}(h) + \alpha e$$

The learning-rate scaling rule:

$$\text{GLR}(N) = \text{GLR}_{d_8}\left(\frac{N}{N_{d_8}}\right)^{\beta}$$

---

_Markdown view of https://picx.dev/p/vOOi07, served by PicX — AI-generated visual whiteboard summaries of research papers._