Scaling Multilingual Mixture-of-Experts: How Language Coverage Changes Capacity and Compute Allocation

Authors: Joel Dag, Nikit Srivastava, Hamada M. Zahera, Axel-Cyrille Ngonga Ngomo (Paderborn University)

---

Summary (Overview)

• Core contribution: Introduces a repetition-aware joint scaling law that relates language-modeling loss to the number of active parameters ($N$), training tokens per language ($D_t$), number of experts ($E$), and language coverage ($K$) for multilingual Mixture-of-Experts (MoE) models.
• Key finding (RQ1): Increasing experts from $E=1$ to $E=32$ reduces the loss penalty of expanding from 12 to 200 languages by 83% at matched training tokens per language, with consistent direction across seeds and external evaluations.
• Key finding (RQ2): Broader language coverage extends the compute range over which expert-heavy configurations are preferred, delaying the switch from more experts to fewer experts by up to 2.7× in compute.
• Key finding (RQ3): Preserving average multilingual quality after language expansion can still leave many individual languages worse off—only 36% of original languages match or improve at the mean-preserving point.
• Validation: The law reduces prediction error by 28% relative to the strongest evaluated baseline (JointMoE) on held-out configurations and by 45% on a larger-model stress test, using 26,300 per-language loss observations from 64 pretrained configurations.

---

Introduction and Theoretical Foundation

Background and Motivation

Multilingual pretraining enables a single model to support many languages through cross-lingual transfer (Conneau et al., 2020; Xue et al., 2021). However, extending language coverage introduces the curse of multilinguality: adding languages worsens performance on already-supported languages due to capacity competition.

Two existing approaches to mitigate this penalty:

• Language-specific capacity: X-MOD adds language-specific modules; X-ELM trains separate models on language subsets (Pfeiffer et al., 2022; Blevins et al., 2024). These tie capacity to pre-selected languages.
• Sparse Mixture-of-Experts (MoE): Models like mSwitch and LOLA store many experts while activating only a small subset per token, allowing capacity growth at approximately fixed per-token computation (Fedus et al., 2022; Srivastava et al., 2025).

The Gap

Existing scaling laws fail to jointly address expert capacity and language coverage:

• Multilingual laws (He et al., 2025b; Longpre et al., 2026) study language sampling, transfer, and coverage costs but ignore expert capacity.
• MoE laws (Ludziejewski et al., 2025) study active parameter vs. expert allocation under compute/memory limits but treat data as a single distribution.

Research Questions

1. RQ1: Does adding experts reduce what each additional language costs?
2. RQ2: How does broader language coverage change the preferred allocation between active and expert capacity under fixed resource budgets?
3. RQ3: When adding languages preserves average quality, how does this affect individual languages?

Theoretical Foundation

The standard Chinchilla scaling law (Hoffmann et al., 2022) decomposes loss as:

$$L(N, D) = L_\infty + \frac{A}{N^\alpha} + \frac{B}{D^\beta}$$

where $L_\infty$ is the asymptotic entropy floor, $A, B$ are coefficients, and $\alpha, \beta$ are scaling exponents. JointMoE extends this with expert count, while ATLAS adds training tokens per language and language coverage.

---

Methodology

Problem Setting

A training configuration fixes the number of active parameters $N$, number of experts $E$, and number of training languages $K$. Each target language $t$ receives $D_t$ training tokens. The goal is to predict validation loss $L_t(N, D_t, E, K)$ for each language.

Core Joint Law

The proposed core joint law models per-language loss as:

$$L_{\infty,t} + \frac{A_t E^\delta K^{\phi(E)}}{N^{\alpha(E)}} + \frac{B_t E^\omega K^{\psi(E)}}{D_t^{\beta(E)}}$$

where:

• $L_{\infty,t}$, $A_t$, $B_t$ are language-specific coefficients
• $E^\delta$, $E^\omega$ scale loss terms with expert count (following JointMoE)
• $K^{\phi(E)}$, $K^{\psi(E)}$ scale with language coverage (following ATLAS)
• Exponents vary with experts via log-linear parameterization:

$$q(E) = a_q + b_q \ln(E), \quad q \in \{\alpha, \beta, \phi, \psi\}$$

Predictive Refinements

Three refinements address simplifying assumptions:

1. Parameter–data coupling: Add factor $N^\zeta$ to the data-limited term, allowing capacity to change the benefit of additional training tokens.

2. Coverage-dependent parameter scaling: Allow the exponent of $N$ to vary with coverage:

$$\alpha(E, K) = \alpha(E) + \gamma_N g(K)$$

where $g(K) = \max\{0, \log(K/K_0)\}$ grows with log coverage above reference $K_0$.

3. Data repetition: Replace $D_t$ with effective exposure $D_t^{eff}$:

$$D_t^{eff} = \min(D_t, U_t) + U_t \cdot r\left(1 - e^{-rR_t}\right)$$

where $U_t$ is unique tokens, $R_t = \max(D_t/U_t - 1, 0)$ is repeated passes, and $r = \lambda_0 P^\rho$ depends on total stored parameters $P$.

Selected Law

Combining refinements while keeping $\phi, \psi$ independent of $E$:

$$L_t(N, D_t, E, K) = L_{\infty,t} + \frac{A_t E^\delta K^\phi}{N^{\alpha(E,K)}} + \frac{B_t E^\omega K^\psi N^\zeta}{(D_t^{eff})^{\beta(E)}}$$

This has 12 shared coefficients plus 3 per-language coefficients.

Experimental Setup

• Architectures: Decoder-only Transformers with SwiGLU feed-forward layers; Top-1 MoE for sparse configurations
• Grid: $E \in \{1, 4, 16, 32\}$, $N \in \{93, 190, 349, 590\}$M active parameters, $K \in \{1, 12, 50, 200\}$ languages → 64 configurations, 400 endpoints, 26,300 loss observations
• Data: FineWeb (English) + FineWeb2 (other languages); temperature sampling at exponent 0.3
• Evaluation: Unigram-adjusted validation loss:

$$L_t = -\frac{1}{T_t} \sum_{i=1}^{T_t} \log \frac{p_\theta(w_i | w_{<i})}{p_{uni,t}(w_i)}$$

• Held-out evaluation: Three splits withholding $N \approx 590$M, $E = 32$, or $K = 200$; plus a stress test with 1.15B active / 21.23B stored parameters

---

Empirical Validation / Results

Scaling Law Evaluation

Table 1: Prediction RMSE across held-out configurations

Key results:

• The repetition-aware law outperforms the strongest baseline (JointMoE with $D_t$) by 28% on held-out configurations
• On the larger-model stress test, accounting for repetition reduces RMSE from 0.31586 to 0.17298 (45% improvement)
• The law transfers to changed language mixtures with 74% RMSE reduction vs. strongest baseline

RQ1: Language Coverage Cost

At matched training tokens per language, the mean loss penalty of expanding from 12 to 200 languages:

83% reduction in coverage penalty from $E=1$ to $E=32$. Across alternative 12-language mixtures, expert expansion reduces the loss increase by 76.6–93.3%. The direction is consistent across three seeds (83.6%, 82.6%, 66.4% reductions at ~190M active parameters).

External evaluation confirms the interaction: expert gains on FLORES (22–57%) and NTREX (20–62%) at $K=200$ relative to $K=12$.

RQ2: Capacity Allocation

• Under a 0.4B total-parameter budget, the preferred switch from 92.8M active/16 experts to 190.3M active/4 experts occurs at:
  • $1.70 \times 10^{18}$ FLOPs for $K=12$
  • $2.75 \times 10^{18}$ FLOPs for $K=200$ (1.62× delay)
• At 5.1B total-parameter budget, the predicted $E=32 \rightarrow 16$ switch occurs at:
  • $4.51 \times 10^{19}$ FLOPs for $K=1$
  • $1.22 \times 10^{20}$ FLOPs for $K=200$ (2.70× delay)

RQ3: Per-Language Preservation

Starting from a 50-language reference at $2 \times 10^{18}$ FLOPs (2.2B total parameters, 32 experts):

At the mean-preserving point, only 36% of original languages match or improve upon their reference loss.

---

Theoretical and Practical Implications

Theoretical Contributions

1. Joint scaling law: First law to simultaneously model active parameters, training tokens per language, expert count, and language coverage for multilingual MoEs, with per-language (not just average) prediction.

2. Repetition awareness: Demonstrates that distinguishing total training tokens from unique data is critical for extrapolation to longer training runs—the core law fails on larger models but the repetition-aware refinement succeeds.

3. Coverage-dependent capacity allocation: Shows that language coverage should be treated as a capacity-allocation variable, not just a data-distribution property.

Practical Implications

1. Model design guidance: Broader language coverage favors expert-heavy configurations across wider compute ranges, informing architecture choices for multilingual models.

2. Evaluation standards: Mean loss is insufficient as a target—capacity planning must specify whether the goal is preserving average quality or per-language quality.

3. Training efficiency: Expert expansion provides a compute-efficient way to offset the curse of multilinguality without increasing per-token computation.

---

Conclusion

The paper demonstrates that expert expansion changes the effect of increased language coverage rather than merely lowering loss overall. The key findings:

1. Expert capacity mitigates coverage costs: $E=32$ reduces the loss penalty of expanding from 12 to 200 languages by 83% at matched training tokens per language.

2. A repetition-aware joint scaling law over $N$, $D_t$, $E$, and $K$ improves held-out prediction by 28% over the strongest baseline and transfers to larger models and changed mixtures.

3. Coverage shifts capacity allocation: Broader language coverage extends the compute range favoring expert-heavy configurations.

4. Average quality ≠ per-language quality: Preserving mean loss can leave many individual languages worse off, making per-language evaluation essential.

Future Directions

• Extending to different architectures, corpora, and training regimes
• Incorporating hardware-specific costs (communication, memory, parallelism)
• Investigating expert granularity and routing strategies beyond Top-1
• Developing more nuanced per-language preservation criteria

---

Limitations

• Main factorial grid uses one training seed (though key comparisons are replicated across 3 seeds)
• Conclusions apply to decoder-only, Top-1 MoE with $E \leq 32$, fixed tokenizer, and specific corpus/sampling
• Allocation results use logical compute ($F \approx 6ND$); actual hardware costs depend on communication and memory overheads
• Web-derived corpora may inherit biases; checkpoints not evaluated for deployment safety