Issue 1: Verifiability of Sparse Expert Structures—Load Balancing, Causal Auditing, and System Measurements

Papers in this issue

1. 1

   Kimi K3: Open Frontier Intelligence

   Kimi K3, a 2.8T-parameter MoE model with 104B active parameters, achieves frontier performance across coding, reasoning, and vision tasks via architectural innovations yielding 2.5x scaling efficiency over its predecessor.

   Editor's note

   At the extreme sparse frontier scale of 2.8T parameters and 896 experts (16 active), proposes Quantile Balancing (QB)—setting expert biases directly from router-score quantiles to exactly match target load, replacing fixed-step sign-based bias updates, paired with RMSNorm and SiTU-GLU for stable optimization. This is the first scalable scheme for auxiliary-loss-free load balancing under extreme sparsity, reporting full routing statistics and a 2.5× scaling efficiency improvement, making it a must-read for load balancing and sparse stability directions.
2. 

   2

   TEMPO: Makespan-Aware Expert-Parallel Load Balancing Across Memory- and Compute-Bound Regimes

   TEMPO replaces token-count or activation-count proxies with a calibrated two-regime time model, achieving within 1.02% of optimal makespan at millisecond cost and 4-6% throughput gains in mapped win regions.

   Editor's note

   First to model expert costs in EP distribution as a max-affine cost function with memory-bound (flat) and compute-bound (linear) regimes, formalized as a fixed-charge makespan problem (NP-hard with additive approximation guarantees). On real batches, 92–100% contain both regimes, and token-count proxies systematically fail at regime mixtures; TEMPO yields 4–6% throughput and ~15.6% p99 latency improvements on Qwen3-235B, and uses phase diagrams to predict when adaptive distribution is worth the investment. A significant methodological contribution to expert-parallel serving and load balancing.
3. 

   3

   How Modular Is a Frontier Mixture-of-Experts? A Pre-registered Causal Test in Which Apparent Expert Modularity Mostly Dissolves

   In a pre-registered causal test of Command A+, only 1 of 6 expert families (Arabic) is a robust module, with apparent modularity fragile to corpus, metric, and statistical bar.

   Editor's note

   On Command A+, performs causal ablations of six pre-registered expert families, treating corpus, metric, and statistical criteria as variables. The conclusion is cautionary: only the Arabic family is a robust module among the six, while the apparent modularity of the others flips with corpus/metric/statistical criteria. This is a benchmark work for the causal evidence threshold of expert specialization, and its "measurement-as-variable" protocol should become standard for subsequent modularity claims.
4. 

   4

   PR²: Predictive Routing Replay for MoE-Based LLM Reinforcement Learning

   PR² introduces a lightweight evolution predictor to anticipate router drift in MoE LLMs, reducing route-induced gradient deviation and improving reasoning accuracy by up to 13% over existing replay methods.

   Editor's note

   Addressing the staleness issue of routing replay in MoE RL, proposes Predictive Routing Replay (Pr2): using a lightweight evolutionary predictor to anticipate short-term router drift, using predicted distributions for top-k during rollout, and replaying predicted routes during training to maintain determinism. Theoretically provides bounds linking replay staleness to routing-induced gradient bias, and experiments on models like Qwen3-30B-A3B significantly reduce routing mismatch and improve inference performance. An important comparison for routing replay alternatives.
5. 

   5

   Compute-Optimal Is Not Cluster-Optimal: Systems-Aware Scaling for Sparse Mixture-of-Experts

   Compute-optimal MoE configurations are not cluster-optimal: optimizing for hardware-deliverable FLOPs, not model FLOPs, yields interior sparsity optima and flips loss rankings.

   Editor's note

   Jointly optimizes sparse MoE architecture and parallel layout under a hardware-deliverable FLOPs budget, proving that model-FLOPs-optimal sparsity is boundary-seeking, and only when cluster system efficiency is accounted for does an interior optimal sparsity emerge. Fits joint scaling laws and calibrated performance models from ~150 from-scratch pretraining runs, validated up to 250B total parameters. Directly addresses the scaling behavior concern that "loss-optimal is not necessarily cluster-optimal."
6. 

   6

   From Observation to Intervention: A Causal Audit of Expert Importance in Mixture-of-Experts Models

   No observational routing metric predicts causal expert importance at the token level across three MoE architectures, with pruning success explained by early-layer redundancy rather than metric validity.

   Editor's note

   Conducts token-level causal auditing of three high-redundancy MoE architectures, finding that observed statistics such as utilization, activation norm, and routing weights cannot predict token-level causal expert importance (all 60 metric-layer combinations have effect sizes below Cohen's d=0.23). This is a strong negative result against implicit assumptions in MoE pruning literature, and introduces a reusable intervention auditing protocol (Definition 2.1), serving as a benchmark for causal evidence in expert specialization.
7. 

   7

   Diagnosing Overhead in Dispatch Operations: Cross-architecture Observatory

   Routing imbalance in Mixture-of-Experts models is a model-and-data property, not a scaling issue, with architectures splitting into data-resilient and persistently concentrated classes.

   Editor's note

   First to systematically characterize MoE routing imbalance behavior with EP degree across architectures and data conditions: EP scaling barely changes per-expert load concentration (stragglers are intrinsic to routing decisions), and mock-token benchmarks systematically overestimate routing imbalance by up to 2.35×. Proposes a taxonomy of architectures into data-elastic vs. persistently concentrated, providing calibrated benchmarks for routing algorithm and expert-parallel system design.
8. 

   8

   Beyond Routing: Decoupling Expert Dispatch and Aggregation in Sparse Mixture-of-Experts

   ]] scores poorly weight selected experts in sparse MoE models, yet a lightweight post-hoc head trained directly on the language modeling objective recovers substantial gains.

   Editor's note

   Decomposes MoE routing causally into distribution (which experts are selected) and aggregation (how to weight computed outputs), and via counterfactual fixing of distribution finds that the router's intra-set weights are far from optimal. Proposes FDAA, a lightweight post-hoc head that learns token-adaptive aggregation under LM objectives, replicated across OLMoE and DeepSeek-V2-Lite. Provides an alternative path for routing-commitment mismatch without re-routing.
9. 

   9

   UltraEP: Unleash MoE Training and Inference on Rack-Scale Nodes with Near-Optimal Load Balancing

   UltraEP achieves 94.6% of ideal throughput in MoE training by using exact-load, real-time expert replication and rerouting, outperforming prediction-based balancers by 1.42x.

   Editor's note

   First system to achieve precise load and real-time expert load balancing on rack-scale nodes (RSN), rebalancing at every microbatch and layer, jointly optimizing expert replication and token re-routing, with RSN-native communication (persistent tile streaming and relay fan-out mitigation). Achieves 94.3% of force-balanced ideal throughput on 106B–671B models with up to 256 GPUs, reducing final rank-level imbalance to 1.01–1.04. A strong baseline for dynamic expert rebalancing.