# Scaling Laws for Looped Mixture of Experts

> Loop scaling laws unify recurrence and sparsity in MoE models, showing sparsity yields ~3x active-parameter efficiency and recurrence ~2x total-parameter efficiency on reasoning tasks.

- **Source:** [arXiv](https://arxiv.org/abs/2609.40316)
- **Published:** 2026-10-12
- **Permalink:** https://picx.dev/p/H7MLem
- **Whiteboard:** https://picx.dev/p/H7MLem/image

## Summary

## Summary (Overview)

- **Novel Scaling Law**: Introduces **Loop Scaling Laws**, the first unified scaling law that jointly models recurrence (looping), sparsity (MoE), model size, and data for looped Mixture-of-Experts (MoE) models.
- **Bounded Recurrence Mapping**: Proposes a bounded, sparsity-conditional effective-parameter mapping that captures diminishing returns from recurrence, unlike prior unbounded linear or power-law mappings.
- **Unified Framework**: The law subsumes standard dense, dense-looped, and non-looped MoE scaling laws as special cases.
- **Practical Benefits**: Demonstrates that sparsity delivers ~3× active-parameter efficiency, recurrence yields ~2× total-parameter efficiency on reasoning, and joint scaling advances the performance frontier.
- **Trillion-Token Validation**: At matched training compute, a looped MoE with law-derived recurrence matches a ~2× larger non-looped MoE on reasoning benchmarks while enabling test-time scaling.

## Introduction and Theoretical Foundation

### Background and Motivation

Scaling language models has conventionally relied on larger models and more data. Two additional axes enable efficient scaling:

1. **MoE Sparsity**: Expands total capacity at fixed active compute by routing tokens to a sparse subset of expert networks.
2. **Looped Transformers**: Increase computational depth by reusing shared weights across multiple passes, increasing depth at fixed stored parameters.

### Two Central Questions

The paper addresses two key questions:
- **(Q1)** How much effective capacity does each recurrent pass add, and does this gain grow indefinitely?
- **(Q2)** Does greater MoE sparsity increase and sustain this gain across recurrent passes?

### Key Observations

The paper identifies that:
- Increasing recurrence shows **diminishing returns**: loss drops sharply over the first few passes, then approaches a plateau.
- Greater sparsity (more experts) **improves and sustains** the gain: with more experts, looping achieves lower loss over more passes.

The intuition: in a dense looped transformer, every recurrent pass reuses the same weights; in a looped MoE, the router can send tokens to different experts across passes, allowing each pass to reach new parameters.

## Methodology

### Standard Scaling Law

The Chinchilla-style scaling law models loss $\mathcal{L}$ as:

$$
\mathcal{L}(N, D) = A N^{\alpha} + B D^{\beta} + c,\tag{1}
$$

where $A, B, \alpha, \beta, c$ are fitted coefficients, with $\alpha, \beta < 0$, and $c$ is the irreducible loss.

### Bounded Recurrence Mapping

The paper proposes a bounded monotone mapping for effective parameters:

$$
N_{\mathrm{eff}}^{\mathrm{bounded}}(R) = N + \kappa_{1} N_{\mathrm{loop}} \left(1 - e^{-(R-1)/\kappa_{2}}\right), \qquad \kappa_{1} > 0, \kappa_{2} > 0\tag{4}
$$

where:
- $\kappa_{1}$ sets the asymptotic effective-parameter gain from looping: $\kappa_{1} N_{\mathrm{loop}}$
- $\kappa_{2}$ controls how quickly this limit is approached as $R$ increases

**Boundary properties**: Recovers the non-looped baseline at $R=1$, and approaches a finite bound as $R \to \infty$:

$$
N_{\mathrm{eff}}^{\mathrm{bounded}}(1) = N, \quad \lim_{R \to \infty} N_{\mathrm{eff}}^{\mathrm{bounded}}(R) = N + \kappa_{1} N_{\mathrm{loop}}.\tag{5}
$$

### Loop Scaling Law

For a looped transformer with $R$ recurrence loops:

$$
\mathcal{L}(N, D, R) = A N_{\mathrm{eff}}(R)^{\alpha} + B D^{\beta} + c,\tag{6}
$$

### Sparsity-Conditional Recurrence Mapping

Let $m = N_{\mathrm{act}}/N_{\mathrm{total}} \in (0, 1]$ denote the MoE active-parameter ratio. The extended mapping:

$$
N_{\mathrm{eff}}(R, m) = N_{\mathrm{act}} + \kappa_{1}(m) N_{\mathrm{loop}} \left(1 - e^{-(R-1)/\kappa_{2}(m)}\right), \quad \kappa_{j}(m) = \kappa_{j} m^{-\theta} (j = 1, 2).\tag{7}
$$

The fitted sparsity-scaling exponent $\theta \geq 0$ quantifies how strongly sparsity amplifies the parameter gain from recurrence.

### MoE Loop Scaling Law

$$
\mathcal{L}(N_{\mathrm{act}}, D, R, E, m) = A \hat{E}^{\delta} N_{\mathrm{eff}}(R, m)^{\alpha + \gamma \ln \hat{E}} + B \hat{E}^{\omega} D^{\beta + \zeta \ln \hat{E}} + c,\tag{9}
$$

where $\hat{E}$ is a monotonic transformation of the expert count $E$.

### Effective-Parameter Multiplier

$$
\rho_{\mathrm{eff}}(R, m) = \frac{N_{\mathrm{eff}}(R, m)}{N_{\mathrm{act}}} = 1 + \kappa_{1}(m) \frac{N_{\mathrm{loop}}}{N_{\mathrm{act}}} \left(1 - e^{-(R-1)/\kappa_{2}(m)}\right).\tag{10}
$$

## Empirical Validation / Results

### Comparison of Recurrence Mappings

The bounded mapping achieves the lowest held-out RMSE across all axes:

| Evaluation | Linear | Power law | Bounded ($N_{\text{eff}}(R)$) | Bounded ($N_{\text{eff}}(R,m)$) |
|---|---|---|---|---|
| Held-out $R$ | 0.2847 | 0.0480 | 0.0190 | **0.0100** |
| Held-out $E$ | 0.0896 | 0.0113 | 0.0075 | **0.0047** |
| Held-out $N$ | 0.0656 | 0.0077 | 0.0065 | **0.0050** |
| Held-out $D$ | 0.0824 | 0.0074 | 0.0058 | **0.0043** |

### Key Findings

1. **Scaling recurrence** adds effective-parameter gain up to a finite, sparsity-dependent asymptote; scaling sparsity raises and sustains this gain over more recurrent passes.
2. **Higher sparsity** improves the IsoFLOP frontier at matched compute; recurrence pushes this frontier further at the same parameter size when given more compute.
3. **Larger training compute budgets and greater sparsity favor higher recurrence.**
4. **Larger weight memory budgets and higher recurrence favor greater sparsity.**
5. **At fixed compute, tight memory favors recurrence; more memory favors sparsity.**

### Downstream Task Results

The paper evaluates across 14 benchmarks spanning five categories: reasoning (BBH, GSM8K), science (ARC-C/E, OpenBookQA), commonsense (HellaSwag, PIQA, SIQA, WinoGrande), reading (BoolQ, DROP), and knowledge (MMLU, Natural Questions, TriviaQA).

**Key results at matched training compute (FLOPs ≈ 1.5×10²²):**

| Model | R | Relative $F_{inf}$ | BBH | GSM8K | Overall (Δ) |
|---|---|---|---|---|---|
| A0.6B-2.9B MoE | 1 | 1.0× | 29.8 | 42.9 | 50.2 |
| A0.3B-1.3B LoopMoE | 1 | 0.4× | 18.6 | 10.5 | 36.6 |
| A0.3B-1.3B LoopMoE | 3 | 1.1× | 30.1 | 41.0 | 46.0 (+9.4) |
| A0.3B-1.3B LoopMoE | 4 | 1.5× | 31.5 | 41.5 | 46.7 (+10.1) |
| A0.3B-1.3B LoopMoE | 5 | 1.8× | 31.8 | 41.2 | 46.9 (+10.3) |

**Efficiency gains:**
- **Sparsity**: ~3× active-parameter efficiency (0.3B MoE with E≥8 outperforms 1.0B dense)
- **Recurrence**: ~2× total-parameter efficiency on reasoning (0.3B with R≥4 matches 0.6B with R=1)
- **Test-time scaling**: Varying R from 1 to 5 boosts Overall performance by +10.3 points

## Theoretical and Practical Implications

### Theoretical Contributions

1. **Unified Framework**: Closes the gap between MoE and looped model scaling theories by jointly modeling recurrence and sparsity.
2. **Bounded Capacity**: Demonstrates that recurrence gains are bounded, with the bound depending on sparsity—a fundamental insight into weight-tied architectures.
3. **Reduced Forms**: The law recovers standard dense, dense-looped, and non-looped MoE scaling laws as special cases, providing a unified theoretical foundation.

### Practical Implications

1. **Architecture Design**: Provides principled recipes for selecting compute- and memory-optimal recurrence and expert count under resource constraints.
2. **Resource-Constrained Deployment**: 
   - Tight memory budgets favor higher recurrence and smaller expert counts
   - Larger memory budgets favor greater sparsity and less recurrence
3. **Test-Time Scaling**: Looped MoE models enable on-demand test-time scaling through recurrence variation, useful for adaptive inference.

## Conclusion

The paper introduces **Loop Scaling Laws**, a unified predictive law for looped MoE models that jointly models recurrence, sparsity, model size, and data. Key takeaways:

- **Bounded, sparsity-conditional mapping** accurately captures diminishing recurrence gains and how sparsity modulates the effective-capacity asymptote.
- **Complementary axes**: Sparsity and recurrence provide distinct but complementary routes to parameter-efficient scaling.
- **Practical validation**: At trillion-token scale, the law-derived recurrence selection enables a looped MoE to match a 2× larger non-looped MoE on reasoning tasks.

### Future Directions

1. **Modeling**: Explore more advanced looping strategies to enrich recurrent latent representations.
2. **Scaling**: Generalize looped MoE models across broader scales, from edge to cloud.
3. **Inference**: Optimize recurrent states or add early-exit gating to scale recurrence only when necessary.

---

_Markdown view of https://picx.dev/p/H7MLem, served by PicX — AI-generated visual whiteboard summaries of research papers._