Summary (Overview)

  • Novel formulation: The paper rethinks Mixture-of-Experts (MoE) load balancing as a feedback control problem, framing per-expert load deviation as the control error and per-expert routing bias as the control input.
  • PID-Balancer family: The authors define a family of controllers, PID-Balancer, based on proportional, integral, and derivative terms of classical PID control, unifying existing methods: the auxiliary balancing loss acts as a proportional controller, and ALF-LB (loss-free bias update) acts as a relay integral controller.
  • Key theoretical insight: The integral term is the only one that both removes persistent imbalance (e.g., dead experts) and averages out step-to-step load fluctuations from micro-batch noise; the proportional and derivative terms pass or amplify this noise into the bias.
  • PID (I) performance: The pure-integral controller with a single gain KIK_I achieves the lowest steady-state imbalance (MaxVio) in both pretraining (with 16–64 experts per layer through a mid-run mixture switch) and continual pretraining (on OLMoE-1B-7B), at comparable perplexity to baselines.
  • Drop-in applicability: PID (I) modifies neither model weights nor the training objective, making it directly applicable to existing pretrained models without retuning.

Introduction and Theoretical Foundation

Background and Motivation

Mixture-of-Experts (MoE) architectures decouple model capacity from per-token compute by replacing dense feed-forward layers with sparsely-activated expert sub-networks. However, without intervention, MoE routers exhibit a "rich-get-richer" dynamic: early-trained experts attract more routing, concentrate gradient signal, and leave other experts under-trained (Eigen et al., 2013; Shazeer et al., 2017).

Key Problem: Non-Stationary Training

Existing load balancing methods were developed and evaluated under stationary training distributions. However, training is often non-stationary:

  • Data mixture changes at curriculum boundaries
  • Continual pretraining extends models into new domains
  • The corpus itself is refreshed

Each change disrupts the routing the model had converged to, creating persistent new imbalances.

Existing Methods and Their Limits

  1. Auxiliary loss (Fedus et al., 2022): Penalizes uneven expert usage via a tunable coefficient, entering the router through the same backward pass as the language-modeling objective.
  2. Auxiliary-Loss-Free (ALF-LB) (Wang et al., 2024): A per-expert bias, updated at each step in proportion to the sign of the load deviation, shifts expert selection while leaving router weights to the language-modeling objective alone.

Both families were largely evaluated under stationary distributions and fail to maintain balance through distribution shifts.

Methodology

Control Formulation

Error definition: Perfect balance indicates TT tokens each activate KK of the EE experts, with every expert receiving an equal share r=TK/Er = TK/E. The per-expert deviation is:

ei(t)=r−Li(t)r(1)e_i(t) = \frac{r - L_i(t)}{r} \tag{1}

where Li(t)L_i(t) is the load of expert ii at optimizer step tt. A positive ei(t)e_i(t) marks an under-loaded expert; negative marks over-loaded. Normalization by rr makes the error dimensionless and bounds it to [1−E/K,1][1 - E/K, 1] regardless of batch size.

Control input: A per-expert bias ui(t)u_i(t) applied with one-step delay:

Tj(t)=Top−Ki(sij(t)+ui(t−1)),sij(t)=exp⁡zij(t)∑k=1Eexp⁡zkj(t)(2)\mathcal{T}_j(t) = \mathrm{Top-}K_i\big(s_{ij}(t) + u_i(t-1)\big), \quad s_{ij}(t) = \frac{\exp z_{ij}(t)}{\sum_{k=1}^{E}\exp z_{kj}(t)} \tag{2}

where zij(t)z_{ij}(t) is the router logit, sij(t)s_{ij}(t) is the softmax score, and Tj(t)\mathcal{T}_j(t) is the set of KK experts selected for token jj. The bias affects only selection; the gate weight remains the unbiased score sij(t)s_{ij}(t).

Closed loop:

u(t−1)→routerL(t)→measuremente(t)→balanceru(t)→…(3)u(t-1) \xrightarrow{\text{router}} L(t) \xrightarrow{\text{measurement}} e(t) \xrightarrow{\text{balancer}} u(t) \to \dots \tag{3}

The PID-Balancer

The discrete-time PID controller applied to the error is:

ui(t)=KPei(t)+KI∑τ=0tei(τ)+KD(ei(t)−ei(t−1))(4)u_i(t) = K_P e_i(t) + K_I \sum_{\tau=0}^{t} e_i(\tau) + K_D\big(e_i(t) - e_i(t-1)\big) \tag{4}

Each member is named by the terms it keeps; PID (I) sets KP=KD=0K_P = K_D = 0.

Existing Methods as PID Members

Aux-loss as Proportional mechanism: The gradient on logits simplifies to:

∂Laux∂zkj(t)=αTskj(t)(eˉj(t)−ek(t)),eˉj(t)=∑isij(t)ei(t)(6)\frac{\partial \mathcal{L}_{\mathrm{aux}}}{\partial z_{kj}(t)} = \frac{\alpha}{T} s_{kj}(t) (\bar{e}_j(t) - e_k(t)), \quad \bar{e}_j(t) = \sum_{i} s_{ij}(t) e_i(t) \tag{6}

This raises the logit of every expert whose error exceeds the softmax-weighted mean by an amount linear in that difference, depending on no error before step tt—the defining property of proportional control. A persistent disturbance is offset only by a nonzero steady-state error that scales as 1/α1/\alpha but never vanishes.

ALF-LB as relay-Integral: The update

ui(t)=ui(t−1)+ηsign⁡(eiALF(t))(8)u_i(t) = u_i(t-1) + \eta \operatorname{sign}\big(e_i^{\mathrm{ALF}}(t)\big) \tag{8}

unrolls to ui(t)=η∑τ=0tsign⁡(ei(τ))u_i(t) = \eta \sum_{\tau=0}^{t} \operatorname{sign}(e_i(\tau)), which is Equation (4) with KP=KD=0K_P = K_D = 0 and KI=ηK_I = \eta applied to the sign of the error. The relay discards error magnitude, weakening the integral guarantee to the sign of the error.

Term-by-Term Analysis

  • Persistent error (e.g., dead expert with ei=1e_i = 1): The integral term ramps the bias by KIK_I per step, eventually bringing the expert back. The proportional term contributes only a constant offset KPK_P that cannot remove imbalance completely. The derivative term contributes nothing (constant error has no change).
  • Step-to-step fluctuation (e.g., alternating error ei(t)=(−1)te_i(t) = (-1)^t): The integral term averages out fluctuations (errors cancel pairwise in the sum). The proportional and derivative terms copy the current error or its change, flipping sign exactly as the error does.

Consequence: The integral term is the only one that both removes persistent error completely and ignores step-to-step fluctuation, leading to the adoption of PID (I) with KIK_I as the only hyperparameter.

Empirical Validation / Results

Experimental Setup

  • Pretraining: OLMoE backbone with E∈{16,32,64}E \in \{16, 32, 64\} experts per layer (Top-2, Top-4, Top-8 routing, respectively), trained from scratch for 33,000 steps with a single domain switch at step 16,500 to GitHub alone. Three pre-shift mixtures: Books alone (1→1), all domains except GitHub (6→1), all seven domains (7→1). Metrics averaged over 6 seeds.
  • Continual Pretraining: Released OLMoE-1B-7B-0125 checkpoint, continued on codeparrot (CPT corpus). Two settings: cold-start (directly on CPT) and warm-start (16,500 steps on Annealing corpus first, then switch to CPT).

Key Metrics

  • MaxVio: MaxVio(t)=(max⁡iLi(t)−r)/r\mathrm{MaxVio}(t) = (\max_i L_i(t) - r)/r — relative overload of the busiest expert
  • DER/UTR: Fraction of experts with load below αr\alpha r (α=0.01\alpha = 0.01 for DER, α=0.25\alpha = 0.25 for UTR)
  • PPL: Validation perplexity (must be preserved)

Pretraining Results (Table 1, selected)

EMethodpostMV (1→1) ↓PPL (1→1) ↓postMV (7→1) ↓PPL (7→1) ↓
16Vanilla4.443.294.773.21
16Aux-loss0.80 ± 0.033.31 ± 0.010.89 ± 0.103.20 ± 0.02
16ALF-LB0.33 ± 0.033.32 ± 0.010.40 ± 0.013.24 ± 0.03
16PID (I)0.30 ± 0.023.32 ± 0.010.36 ± 0.093.23 ± 0.02
64Vanilla6.083.306.103.23
64Aux-loss1.27 ± 0.103.29 ± 0.021.50 ± 0.073.21 ± 0.03
64ALF-LB0.80 ± 0.033.31 ± 0.021.20 ± 0.203.22 ± 0.02
64PID (I)0.40 ± 0.013.30 ± 0.020.47 ± 0.013.21 ± 0.02

Key findings:

  • PID (I) achieves the lowest MaxVio both before and after the shift in every setting
  • ALF-LB improves balance substantially but falls further behind PID as expert count grows
  • Aux-loss balances better than Vanilla but less effectively than bias-based methods
  • Perplexity remains nearly identical across balancers in every setting

Continual Pretraining Results (Table 2, selected)

MethodSettingpreMV ↓postMV ↓PPL-CPT ↓PPL-Annealing ↓
Base---2.405.46
Vanillacold-5.832.197.23
Aux-losscold-4.432.197.21
ALF-LBcold-1.012.197.36
PID (I)cold-0.902.197.39
Vanillawarm1.145.762.246.07
Aux-losswarm1.094.512.246.07
ALF-LBwarm0.721.032.246.28
PID (I)warm0.660.872.236.28

Key findings:

  • PID (I) attains the lowest postMV in both settings
  • Adaptation to the new domain (PPL-CPT) is insensitive to balancer choice
  • Bias-based methods forget more on the Annealing corpus (higher PPL-Annealing) — a trade-off: stronger balancing pushes tokens onto experts that may overwrite prior-domain knowledge
  • Both PID (I) and ALF-LB serve as effective drop-in replacements on pretrained models

Theoretical and Practical Implications

Theoretical Contributions

  1. Unified control framework: The paper provides a rigorous mathematical connection between classical PID control and MoE load balancing, showing that two widely used methods (auxiliary loss and ALF-LB) are specific members of the PID-Balancer family.

  2. Principled justification for integral control: The analysis demonstrates that the integral term uniquely satisfies both requirements of an ideal balancer—removing persistent error and averaging out step-to-step noise. This provides a theoretical justification for why ALF-LB works and why the auxiliary loss leaves a residual steady-state error.

  3. Router as static map: By treating the router as a memoryless, monotone map with a one-step delay, the analysis isolates all memory in the balancer, enabling local analysis without explicitly modeling the router's complex behavior.

Practical Implications

  1. Single hyperparameter: PID (I) requires tuning only KIK_I, simplifying deployment. Section C.2 provides an empirical rule for transferring KIK_I across expert counts.

  2. Drop-in deployment: Since PID (I) modifies neither weights nor objective, it can be inserted into existing pretrained models without retraining, as demonstrated in the continual pretraining experiments.

  3. Balance-specialization trade-off: The paper reveals a fundamental tension: bias-based balancers (ALF-LB, PID (I)) achieve better balance but may cause more forgetting in continual pretraining because they force tokens onto experts that lack matching knowledge. Aux-loss reduces domain dependence of routing but leaves more imbalance.

  4. Latency implications: Since MRI=(MaxVio+1)⋅K/E\mathrm{MRI} = (\mathrm{MaxVio} + 1) \cdot K/E, the balance results convert to estimates of worst-case latency under expert parallelism, connecting load balance to deployment efficiency.

Conclusion

The paper rethinks MoE load balancing as a feedback control problem, defining the PID-Balancer family and showing that:

  1. The auxiliary loss acts as a proportional controller (through gate weights), leaving a non-vanishing steady-state error.
  2. ALF-LB acts as a relay integral controller (on routing decisions), which removes persistent error but discards error magnitude.
  3. The pure-integral PID (I) with a single gain KIK_I achieves the lowest steady-state imbalance in both pretraining (through distribution shifts) and continual pretraining, at comparable perplexity and downstream accuracy.

The control-theoretic perspective provides a principled basis for understanding and designing load balancers, highlighting that the integral term's memory is essential for both removing persistent imbalance and averaging out micro-batch noise.

Future directions (Section D.4): The paper discusses limitations and suggests exploring the derivative term's potential in anticipatory balancing, adaptive gain scheduling for more complex distribution shifts, and extending the framework to other routing architectures and deployment scenarios.

Related papers