Issue 3: Causal Decomposition of Routing Drift and Scaling Evidence for Sparsity and Granularity
Highlights
This issue features two main threads: one provides a causal-level decomposition and counterintuitive negative results on "routing drift," and the other delivers the most systematic decoupling evidence to date on the scaling behavior of sparsity and expert granularity.
On the routing drift front, Launch-Bound and Substitutable uses causal routing replay interventions to separate "routing changes" from "weight errors": INT4 quantization changes an average of 0.53 out of 8 selected experts per token, but replaying the quantized routing decisions on full-precision weights reproduces only 2.7% of the quality degradation—experts are substitutable rather than specialized. More critically, exempting the router from quantization reduces drift by 20% yet increases loss, proving that routing fidelity and output quality are separable objectives. This contrasts with last issue's VeXact numerical-scale mismatch diagnosis: if routing drift alone explains only a small fraction of quality loss, then routing-replay methods such as PR² and ReLibra warrant re-examination of what they actually optimize. A related mechanistic study, Routers Learn the Geometry of Their Experts, shows from a gradient-structure perspective that routers and their selected experts accumulate gradients along the same input direction (geometric coupling), and that auxiliary load-balancing losses break this coupling by injecting input-direction gradients into all router weights (directional cosine similarity increases nearly 3×)—providing a mechanistic explanation for why bias-only balancing preserves specialization without auxiliary losses.
On the scaling front, Slicing and Dicing jointly exhausts expert count, granularity, heterogeneous expert pools, and shared experts across 2000+ pretraining runs (up to 6.6B total parameters). With FLOPs matched, it decouples expert count from granularity: performance improves monotonically with total MoE parameters (even at extreme sparsity s=128), the optimal expert granularity depends almost solely on activated parameter count and not on total parameters, and heterogeneous expert pools offer no gain over well-configured homogeneous MoEs. This directly corrects prior scaling-law conclusions that assumed a fixed optimal number of activated experts and the necessity of shared experts, and also addresses the field's concern that loss-optimal configurations may not be reasoning-optimal.
On the serving side, several systems papers are equally dense. FreeBalance moves online load balancing from post-routing to pre-routing: it exploits the similarity of hidden representations across adjacent layers to predict routing loads ahead of time, overlaps expert migration with attention computation, and achieves lossless balancing. Gimbal is the first to jointly optimize DP-engine request scheduling and expert placement: a front-end module uses online backend pressure signals (KV cache, remaining prefill, queue backlog, MoE expert pressure) for fine-grained request dispatch, while a back-end module uses source-DP-to-expert routing statistics and a MINLP heuristic for source-aware expert placement, closing the loop between front-end and back-end pressure. EasyBalance introduces a cross-layer load-balancing axis orthogonal to replication/migration: it exploits natural redundancy across layers and uses micro-batch scheduling to jointly execute multi-layer MoE loads so that imbalances cancel out, without modifying expert-to-device mappings or incurring migration overhead, and with worst-case safety guarantees. Finally, Mixture-of-Experts Serving is the first to formalize the latency-reconfiguration trade-off in MoE serving as an online optimization problem, providing a polynomial-time O(√log k)-competitive online algorithm, a matching Ω(√log k) lower bound, a constant-factor offline approximation, and NP-hardness/FPTAS-impossibility results.
Community and Dynamics
On the routing-specialization tension, From Pseudo-Balancing to True Specialization: Memory-Aware Routing explicitly distinguishes "pseudo-balancing" (uniform load) from "true specialization" (semantic consistency), and proposes a memory-aware routing mechanism that uses long-term token-preference vectors per expert to guide routing. The authors claim that with half the experts, they maintain performance and improve specialization by 35%. Their "pseudo-balancing" concept resonates with this issue's Routers Learn the Geometry finding that auxiliary losses break geometric coupling, though the scale and mechanistic details await full-paper confirmation.
On routing failure-mode diagnosis, Sparse Mixture-of-Experts Routing in Visual Diffusion Transformers provides a systematic diagnosis of dense-to-MoE conversion in DiT models: linear routers fall into global soft saturation, causing expert homogenization (cosine similarity >0.99); MLP routers avoid global deadlock but introduce "selective deadlock"—about one-third of layers degrade to single-expert mode, and even raising the auxiliary loss coefficient to 0.2 cannot prevent this. The deadlocked layers exhibit a U-shaped distribution, concentrated in early visual-processing layers and late semantic-integration layers. The paper also reports a numerical pitfall where bf16 mixed precision truncates tiny expert weight updates to zero, echoing the numerical-mismatch theme of last issue's VeXact.
Open Questions
-
Launch-Bound and Substitutable shows that routing drift explains only 2.7% of quantization quality loss and that routing fidelity and output quality are separable—if routing drift is not the main culprit, what exactly are routing-replay/compensation methods such as PR² and ReLibra optimizing? Should their objective be repositioned from "quality" to "stability"?
-
Slicing and Dicing shows that optimal granularity depends only on activated parameter count and that heterogeneous/shared experts provide no gain—how does this reconcile with frontier designs such as Kimi K3's 896 experts and DeepSeek's shared experts? Does the conclusion hold under reasoning-optimal rather than loss-optimal criteria?
-
Routers Learn the Geometry proposes geometric coupling as a unified explanation for why bias-only balancing preserves specialization—can this mechanism be validated at frontier scales beyond 1B parameters, and can it be placed in the same causal framework as the expert-level causal tracing protocol of Expert-Aware Causal Tracing?
Papers in this issue
Scaling MoE performance depends primarily on maximizing total expert parameters and setting expert granularity based on active parameter scale, not on complex routing, heterogeneous experts, or generalists.
Editor's noteMust-read. The most systematic decoupling of sparsity and granularity to date: 2000+ pretraining runs (up to 6.6B total parameters) jointly exhaust expert count, granularity, heterogeneous pools, and shared experts. With FLOPs matched, performance improves monotonically with total MoE parameters (even at s=128 extreme sparsity), optimal granularity depends only on activated parameter count, and heterogeneous/shared experts offer no gain over well-tuned homogeneous MoEs. Directly corrects prior scaling-law conclusions about fixed optimal activated-expert counts and the necessity of shared experts, and provides new empirical baselines for sparsity and granularity research.
Fused kernels, INT4 quantization, and torch.compile fail on MoE models because they are launch-bound, not arithmetic-bound, with routing drift causing only 2.7% of quality loss.
Editor's noteMust-read. Uses causal routing replay interventions to separate routing changes from weight errors: INT4 changes 0.53/8 experts per token on average, but replaying quantized routing on full-precision weights reproduces only 2.7% of quality loss—experts are substitutable, not specialized. Exempting the router from quantization reduces drift by 20% yet increases loss, proving routing fidelity and output quality are separable. Also reveals the model is launch-bound rather than arithmetic-bound: eliminating all graph breaks makes it 3× slower. Directly challenges the premise of routing-replay/compensation methods.
Router and expert weights in sparse mixture-of-experts models naturally align through shared gradient updates, but auxiliary load-balancing losses disrupt this coupling, while a parameter-free K-Means router preserves it with minimal perplexity cost.
Editor's noteMust-read. First to show from a gradient-structure perspective that routers and their selected experts accumulate gradients along the same input direction (geometric coupling), and that auxiliary load-balancing losses break this coupling by injecting input-direction gradients into all router weights (directional cosine similarity increases nearly 3×). Provides a mechanistic explanation for why bias-only balancing preserves specialization without auxiliary losses. Compared to last issue's #1 From Observation to Intervention (token-level causal auditing) and #2 Beyond Geometric Complementarity (geometric overlap and functional value), this paper offers a gradient-level mechanism and theoretical grounding for auxiliary-loss-free balancing.
FreeBalance predicts expert workloads by reusing the frozen router on earlier hidden states, enabling lossless, proactive load balancing that reduces max-to-mean rank load ratios by 32.8% and prefill latency by 13.1%.
Editor's noteMust-read. Moves online load balancing from post-routing to pre-routing: exploits cross-layer hidden-representation similarity to predict routing loads ahead of time, overlaps expert migration with attention computation, and achieves lossless balancing. Compared to last issue's #1 UltraEP (post-routing reactive rebalancing) and PR² (predictive replay for train-inference consistency), this is the first to apply predictive replay to online expert migration scheduling. Limitation: validated only on a single node with EP=8, but the approach has a clear scaling path.
Gimbal's coordinated cross-level scheduling, jointly optimizing DP-engine dispatch and source-aware expert placement, cuts MoE LLM serving TTFT by 42.9% and TPOT by 33.3% versus vLLM.
Editor's noteFirst to jointly optimize DP-engine request scheduling and MoE expert placement: front-end uses online backend pressure signals (KV cache, remaining prefill, queue backlog, MoE expert pressure) for fine-grained request dispatch; back-end uses source-DP-to-expert routing statistics and a MINLP heuristic for source-aware expert placement, closing the loop between front-end and back-end pressure. Compared to last issue's #1 UltraEP/TEMPO (which focus on intra-expert-parallel rebalancing), Gimbal adds a cross-engine request-side orthogonal axis, reducing average TTFT by 42.9% and TPOT by 33.3% on vLLM.
Expert-aware causal tracing reveals factual recall localizes to a single expert in Qwen3 but distributes across expert coalitions in Mixtral, showing localization is architecture-dependent.
Editor's noteExtends causal tracing from dense transformers to expert level: first locates layers via MoE-block patching, then uses ablation-difference patching with active-random controls to decompose signals into single-expert contributions, revealing that layer-wise rescue can concentrate in a single expert (Qwen3's L44E069) or distribute across a routing coalition (Mixtral). Compared to last issue's #1 From Observation to Intervention and #2 When Are Experts Misrouted, this provides a two-stage protocol distinguishing block-level, single-expert, and coalition-level localization. Mixtral's negative result and the protocol itself are valuable for setting evidence standards for expert specialization claims.
EasyBalance reduces GPU under-utilization by over 40% in distributed MoE inference by cross-layer scheduling of micro-batches, without modifying expert-device mappings.
Editor's noteProposes a cross-layer load-balancing axis orthogonal to replication/migration: exploits natural cross-layer expert redundancy and uses micro-batch scheduling to jointly execute multi-layer MoE loads so imbalances cancel out, without modifying expert-device mappings or incurring migration overhead, and with worst-case safety guarantees. Compared to last issue's #1 UltraEP and #2 ELDR (replication/migration methods), EasyBalance offers an instant-adaptation, zero-overhead complementary path, reducing GPU idle rate by over 40%.
This paper introduces the first formal model for Mixture-of-Experts serving, providing an optimal online algorithm with a tight competitive ratio and a 2-approximation for the static case.
Editor's noteFirst to formalize the latency-reconfiguration trade-off in MoE serving as an online optimization problem: provides a polynomial-time O(√log k)-competitive online algorithm, a matching Ω(√log k) lower bound, a constant-factor offline approximation, and NP-hardness/FPTAS-impossibility results. Compared to last issue's #1 TEMPO and UltraEP (heuristic system designs), this paper provides a provable competitive-ratio framework, setting a theoretical baseline for dynamic expert rebalancing. Pure theory without experimental validation; the realism of its abstractions (linear reconfiguration costs, max-load latency) in real deployments remains to be tested.
