Summary (Overview)

• Novel Scaling Law: Introduces Loop Scaling Laws, the first unified scaling law that jointly models recurrence (looping), sparsity (MoE), model size, and data for looped Mixture-of-Experts (MoE) models.
• Bounded Recurrence Mapping: Proposes a bounded, sparsity-conditional effective-parameter mapping that captures diminishing returns from recurrence, unlike prior unbounded linear or power-law mappings.
• Unified Framework: The law subsumes standard dense, dense-looped, and non-looped MoE scaling laws as special cases.
• Practical Benefits: Demonstrates that sparsity delivers ~3× active-parameter efficiency, recurrence yields ~2× total-parameter efficiency on reasoning, and joint scaling advances the performance frontier.
• Trillion-Token Validation: At matched training compute, a looped MoE with law-derived recurrence matches a ~2× larger non-looped MoE on reasoning benchmarks while enabling test-time scaling.

Introduction and Theoretical Foundation

Background and Motivation

Scaling language models has conventionally relied on larger models and more data. Two additional axes enable efficient scaling:

1. MoE Sparsity: Expands total capacity at fixed active compute by routing tokens to a sparse subset of expert networks.
2. Looped Transformers: Increase computational depth by reusing shared weights across multiple passes, increasing depth at fixed stored parameters.

Two Central Questions

The paper addresses two key questions:

• (Q1) How much effective capacity does each recurrent pass add, and does this gain grow indefinitely?
• (Q2) Does greater MoE sparsity increase and sustain this gain across recurrent passes?

Key Observations

The paper identifies that:

• Increasing recurrence shows diminishing returns: loss drops sharply over the first few passes, then approaches a plateau.
• Greater sparsity (more experts) improves and sustains the gain: with more experts, looping achieves lower loss over more passes.

The intuition: in a dense looped transformer, every recurrent pass reuses the same weights; in a looped MoE, the router can send tokens to different experts across passes, allowing each pass to reach new parameters.

Methodology

Standard Scaling Law

The Chinchilla-style scaling law models loss $\mathcal{L}$ as:

$$\mathcal{L}(N, D) = A N^{\alpha} + B D^{\beta} + c,\tag{1}$$

where $A, B, \alpha, \beta, c$ are fitted coefficients, with $\alpha, \beta < 0$, and $c$ is the irreducible loss.

Bounded Recurrence Mapping

The paper proposes a bounded monotone mapping for effective parameters:

$$N_{\mathrm{eff}}^{\mathrm{bounded}}(R) = N + \kappa_{1} N_{\mathrm{loop}} \left(1 - e^{-(R-1)/\kappa_{2}}\right), \qquad \kappa_{1} > 0, \kappa_{2} > 0\tag{4}$$

where:

• $\kappa_{1}$ sets the asymptotic effective-parameter gain from looping: $\kappa_{1} N_{\mathrm{loop}}$
• $\kappa_{2}$ controls how quickly this limit is approached as $R$ increases

Boundary properties: Recovers the non-looped baseline at $R=1$, and approaches a finite bound as $R \to \infty$:

$$N_{\mathrm{eff}}^{\mathrm{bounded}}(1) = N, \quad \lim_{R \to \infty} N_{\mathrm{eff}}^{\mathrm{bounded}}(R) = N + \kappa_{1} N_{\mathrm{loop}}.\tag{5}$$

Loop Scaling Law

For a looped transformer with $R$ recurrence loops:

$$\mathcal{L}(N, D, R) = A N_{\mathrm{eff}}(R)^{\alpha} + B D^{\beta} + c,\tag{6}$$

Sparsity-Conditional Recurrence Mapping

Let $m = N_{\mathrm{act}}/N_{\mathrm{total}} \in (0, 1]$ denote the MoE active-parameter ratio. The extended mapping:

$$N_{\mathrm{eff}}(R, m) = N_{\mathrm{act}} + \kappa_{1}(m) N_{\mathrm{loop}} \left(1 - e^{-(R-1)/\kappa_{2}(m)}\right), \quad \kappa_{j}(m) = \kappa_{j} m^{-\theta} (j = 1, 2).\tag{7}$$

The fitted sparsity-scaling exponent $\theta \geq 0$ quantifies how strongly sparsity amplifies the parameter gain from recurrence.

MoE Loop Scaling Law

$$\mathcal{L}(N_{\mathrm{act}}, D, R, E, m) = A \hat{E}^{\delta} N_{\mathrm{eff}}(R, m)^{\alpha + \gamma \ln \hat{E}} + B \hat{E}^{\omega} D^{\beta + \zeta \ln \hat{E}} + c,\tag{9}$$

where $\hat{E}$ is a monotonic transformation of the expert count $E$.

Effective-Parameter Multiplier

$$\rho_{\mathrm{eff}}(R, m) = \frac{N_{\mathrm{eff}}(R, m)}{N_{\mathrm{act}}} = 1 + \kappa_{1}(m) \frac{N_{\mathrm{loop}}}{N_{\mathrm{act}}} \left(1 - e^{-(R-1)/\kappa_{2}(m)}\right).\tag{10}$$

Empirical Validation / Results

Comparison of Recurrence Mappings

The bounded mapping achieves the lowest held-out RMSE across all axes:

Key Findings

1. Scaling recurrence adds effective-parameter gain up to a finite, sparsity-dependent asymptote; scaling sparsity raises and sustains this gain over more recurrent passes.
2. Higher sparsity improves the IsoFLOP frontier at matched compute; recurrence pushes this frontier further at the same parameter size when given more compute.
3. Larger training compute budgets and greater sparsity favor higher recurrence.
4. Larger weight memory budgets and higher recurrence favor greater sparsity.
5. At fixed compute, tight memory favors recurrence; more memory favors sparsity.

Downstream Task Results

The paper evaluates across 14 benchmarks spanning five categories: reasoning (BBH, GSM8K), science (ARC-C/E, OpenBookQA), commonsense (HellaSwag, PIQA, SIQA, WinoGrande), reading (BoolQ, DROP), and knowledge (MMLU, Natural Questions, TriviaQA).

Key results at matched training compute (FLOPs ≈ 1.5×10²²):

Efficiency gains:

• Sparsity: ~3× active-parameter efficiency (0.3B MoE with E≥8 outperforms 1.0B dense)
• Recurrence: ~2× total-parameter efficiency on reasoning (0.3B with R≥4 matches 0.6B with R=1)
• Test-time scaling: Varying R from 1 to 5 boosts Overall performance by +10.3 points

Theoretical and Practical Implications

Theoretical Contributions

1. Unified Framework: Closes the gap between MoE and looped model scaling theories by jointly modeling recurrence and sparsity.
2. Bounded Capacity: Demonstrates that recurrence gains are bounded, with the bound depending on sparsity—a fundamental insight into weight-tied architectures.
3. Reduced Forms: The law recovers standard dense, dense-looped, and non-looped MoE scaling laws as special cases, providing a unified theoretical foundation.

Practical Implications

1. Architecture Design: Provides principled recipes for selecting compute- and memory-optimal recurrence and expert count under resource constraints.
2. Resource-Constrained Deployment:
   • Tight memory budgets favor higher recurrence and smaller expert counts
   • Larger memory budgets favor greater sparsity and less recurrence
3. Test-Time Scaling: Looped MoE models enable on-demand test-time scaling through recurrence variation, useful for adaptive inference.

Conclusion

The paper introduces Loop Scaling Laws, a unified predictive law for looped MoE models that jointly models recurrence, sparsity, model size, and data. Key takeaways:

• Bounded, sparsity-conditional mapping accurately captures diminishing recurrence gains and how sparsity modulates the effective-capacity asymptote.
• Complementary axes: Sparsity and recurrence provide distinct but complementary routes to parameter-efficient scaling.
• Practical validation: At trillion-token scale, the law-derived recurrence selection enables a looped MoE to match a 2× larger non-looped MoE on reasoning tasks.

Future Directions

1. Modeling: Explore more advanced looping strategies to enrich recurrent latent representations.
2. Scaling: Generalize looped MoE models across broader scales, from edge to cloud.
3. Inference: Optimize recurrent states or add early-exit gating to scale recurrence only when necessary.