# Scaling Multilingual Mixture-of-Experts: How Language Coverage Changes Capacity and Compute Allocation > Scaling experts from 1 to 32 cuts the multilingual loss penalty of expanding language coverage from 12 to 200 languages by 83%, with broader coverage favoring expert-heavy models. - **Source:** [arXiv](https://arxiv.org/abs/2610.20247) - **Published:** 2026-10-24 - **Permalink:** https://picx.dev/p/dMzeuK - **Whiteboard:** https://picx.dev/p/dMzeuK/image ## Summary # Scaling Multilingual Mixture-of-Experts: How Language Coverage Changes Capacity and Compute Allocation **Authors:** Joel Dag, Nikit Srivastava, Hamada M. Zahera, Axel-Cyrille Ngonga Ngomo (Paderborn University) --- ## Summary (Overview) - **Core contribution:** Introduces a **repetition-aware joint scaling law** that relates language-modeling loss to the number of active parameters ($N$), training tokens per language ($D_t$), number of experts ($E$), and language coverage ($K$) for multilingual Mixture-of-Experts (MoE) models. - **Key finding (RQ1):** Increasing experts from $E=1$ to $E=32$ reduces the loss penalty of expanding from 12 to 200 languages by **83%** at matched training tokens per language, with consistent direction across seeds and external evaluations. - **Key finding (RQ2):** Broader language coverage extends the compute range over which expert-heavy configurations are preferred, delaying the switch from more experts to fewer experts by up to **2.7×** in compute. - **Key finding (RQ3):** Preserving average multilingual quality after language expansion can still leave many individual languages worse off—only 36% of original languages match or improve at the mean-preserving point. - **Validation:** The law reduces prediction error by **28%** relative to the strongest evaluated baseline (JointMoE) on held-out configurations and by **45%** on a larger-model stress test, using 26,300 per-language loss observations from 64 pretrained configurations. --- ## Introduction and Theoretical Foundation ### Background and Motivation Multilingual pretraining enables a single model to support many languages through cross-lingual transfer (Conneau et al., 2020; Xue et al., 2021). However, extending language coverage introduces the **curse of multilinguality**: adding languages worsens performance on already-supported languages due to capacity competition. Two existing approaches to mitigate this penalty: - **Language-specific capacity:** X-MOD adds language-specific modules; X-ELM trains separate models on language subsets (Pfeiffer et al., 2022; Blevins et al., 2024). These tie capacity to pre-selected languages. - **Sparse Mixture-of-Experts (MoE):** Models like mSwitch and LOLA store many experts while activating only a small subset per token, allowing capacity growth at approximately fixed per-token computation (Fedus et al., 2022; Srivastava et al., 2025). ### The Gap Existing scaling laws fail to jointly address expert capacity and language coverage: - **Multilingual laws** (He et al., 2025b; Longpre et al., 2026) study language sampling, transfer, and coverage costs but ignore expert capacity. - **MoE laws** (Ludziejewski et al., 2025) study active parameter vs. expert allocation under compute/memory limits but treat data as a single distribution. ### Research Questions 1. **RQ1:** Does adding experts reduce what each additional language costs? 2. **RQ2:** How does broader language coverage change the preferred allocation between active and expert capacity under fixed resource budgets? 3. **RQ3:** When adding languages preserves average quality, how does this affect individual languages? ### Theoretical Foundation The standard Chinchilla scaling law (Hoffmann et al., 2022) decomposes loss as: $$L(N, D) = L_\infty + \frac{A}{N^\alpha} + \frac{B}{D^\beta}$$ where $L_\infty$ is the asymptotic entropy floor, $A, B$ are coefficients, and $\alpha, \beta$ are scaling exponents. JointMoE extends this with expert count, while ATLAS adds training tokens per language and language coverage. --- ## Methodology ### Problem Setting A training configuration fixes the number of active parameters $N$, number of experts $E$, and number of training languages $K$. Each target language $t$ receives $D_t$ training tokens. The goal is to predict validation loss $L_t(N, D_t, E, K)$ for each language. ### Core Joint Law The proposed **core joint law** models per-language loss as: $$L_{\infty,t} + \frac{A_t E^\delta K^{\phi(E)}}{N^{\alpha(E)}} + \frac{B_t E^\omega K^{\psi(E)}}{D_t^{\beta(E)}}$$ where: - $L_{\infty,t}$, $A_t$, $B_t$ are language-specific coefficients - $E^\delta$, $E^\omega$ scale loss terms with expert count (following JointMoE) - $K^{\phi(E)}$, $K^{\psi(E)}$ scale with language coverage (following ATLAS) - Exponents vary with experts via log-linear parameterization: $$q(E) = a_q + b_q \ln(E), \quad q \in \{\alpha, \beta, \phi, \psi\}$$ ### Predictive Refinements Three refinements address simplifying assumptions: **1. Parameter–data coupling:** Add factor $N^\zeta$ to the data-limited term, allowing capacity to change the benefit of additional training tokens. **2. Coverage-dependent parameter scaling:** Allow the exponent of $N$ to vary with coverage: $$\alpha(E, K) = \alpha(E) + \gamma_N g(K)$$ where $g(K) = \max\{0, \log(K/K_0)\}$ grows with log coverage above reference $K_0$. **3. Data repetition:** Replace $D_t$ with effective exposure $D_t^{eff}$: $$D_t^{eff} = \min(D_t, U_t) + U_t \cdot r\left(1 - e^{-rR_t}\right)$$ where $U_t$ is unique tokens, $R_t = \max(D_t/U_t - 1, 0)$ is repeated passes, and $r = \lambda_0 P^\rho$ depends on total stored parameters $P$. ### Selected Law Combining refinements while keeping $\phi, \psi$ independent of $E$: $$L_t(N, D_t, E, K) = L_{\infty,t} + \frac{A_t E^\delta K^\phi}{N^{\alpha(E,K)}} + \frac{B_t E^\omega K^\psi N^\zeta}{(D_t^{eff})^{\beta(E)}}$$ This has 12 shared coefficients plus 3 per-language coefficients. ### Experimental Setup - **Architectures:** Decoder-only Transformers with SwiGLU feed-forward layers; Top-1 MoE for sparse configurations - **Grid:** $E \in \{1, 4, 16, 32\}$, $N \in \{93, 190, 349, 590\}$M active parameters, $K \in \{1, 12, 50, 200\}$ languages → **64 configurations, 400 endpoints, 26,300 loss observations** - **Data:** FineWeb (English) + FineWeb2 (other languages); temperature sampling at exponent 0.3 - **Evaluation:** Unigram-adjusted validation loss: $$L_t = -\frac{1}{T_t} \sum_{i=1}^{T_t} \log \frac{p_\theta(w_i | w_{