Issue 2: Route Replay Moves to the System Level, Expert Specialization Gains Positive Structural Evidence
Highlights of This Issue
This issue features two main threads: route replay upgrades from an algorithmic stability measure to a system-level load-balancing oracle, and evidence for expert specialization shifts from cautionary negative results to positive structural patterns.
On the RL stability front, ReLibra seizes the unique opportunity in the RL rollout-training workflow that "routes are known before training," extending route replay from the algorithmic layer (last issue's PR² predictive replay) to the system layer: performing expert reordering at the inter-batch time scale and expert replication at the intra-batch time scale, matching hierarchical network bandwidth, and absorbing hot expert fluctuations at the microbatch granularity. VeXact isolates the training-inference mismatch from the route decision layer, building a zero-mismatch rollout engine as a diagnostic baseline, proving that token-level numerical mismatch alone can independently cause RL training collapse—providing a calibration benchmark for evaluating various route replay and compensation methods.
On the expert specialization evidence front, Beyond Geometric Complementarity uses the Expert Subspace Separation Index and a prefix-controlled 2×2 factorial design to separate route consistency, candidate quality, and candidate-context interaction, consistently finding "coherent overlap" across 39 factorial units: geometric overlap coexists with functional value, routes select token-relevant experts from shared geometric neighborhoods, and the value of multi-expert computation does not depend on disjoint linear coverage. This is an important positive complement to last issue's causal audit negative results. When Are Experts Misrouted turns to the quality of route selection itself, revealing that standard top-k routers align well on confident tokens but systematically fail on fragile tokens driving hard reasoning, and that updating only the final-layer router (<0.001% parameters) can shift pass@K.
The serving side is equally dense. MoE-Prefill proposes a prefill-only AsyncEP paradigm, collecting experts by weight rather than activation, liberating expert parallelism from the synchronous activation routing of the decoding era; ExpertPlex shares MoE experts across stages while separating attention modules, eliminating over 95% of duplicate model weights; ELDR adds an expert locality routing axis orthogonal to load balancing for PD-disaggregated serving.
Community and Developments
On the engineering side, LMSYS's DeepSeek-V4-Pro serving report provides measured route skew and placement statistics on frontier workloads: recording route affinity from representative requests and using it to configure EPLB redundant experts yields a 13.5% per-GPU throughput improvement—new empirical evidence beyond simulations or small-scale studies. NVIDIA's Router Replay (R3) guide operationalizes route replay as an engineering solution for distributed RL training, documenting the route inconsistency between rollout and training passes.
On the theoretical side of routing dynamics, Uncertainty-Aware Routing proposes "frustrated exploration" as a three-stage phase transition for MoE routing (exploration, symmetry breaking, stabilization), pointing out that uniformity constraints structurally interfere with specialization—providing a mechanistic framework for when static load balancing fails. ERPO offers importance sampling and expert selection entropy rewards from the expert perspective as an alternative path to route replay/constraint stabilization. Additionally, an engineering blog post reports negative results of forced topic-based specialization failing or collapsing, noting that routes become fixed within the first 2% of training—worth attention as a hypothesis-generating finding, though model scale and baseline details are limited.
Open Questions
-
Given that VeXact proves numerical-level training-inference mismatch alone can cause RL collapse, to what extent are existing route replay/prediction methods (e.g., ReLibra's system-level replay) compensating for route decision staleness rather than being masked by lower-level numerical mismatch? Should the zero-mismatch baseline become standard for evaluating all route stabilization methods?
-
Beyond Geometric Complementarity's "coherent overlap" shows that geometric overlap does not imply functional redundancy—so should pruning and route design rely on counterfactual route utility (as measured by When Are Experts Misrouted) rather than geometric similarity? How can the two be unified under a single protocol?
-
MoE-Prefill's AsyncEP collects experts by weight, and ExpertPlex shares experts across stages—how do these "non-activation-routing" paradigms coexist with synchronous activation routing in the decoding phase within mixed prefill-decode serving? Is there a unified execution abstraction?
-
How do Uncertainty-Aware Routing's phase transition framework and ReLibra's microbatch-level replay interact—when routing is in the "frustrated exploration" stage, does microbatch-level load balancing amplify instability instead?
Papers in this issue
ReLibra exploits routing replay in MoE RL training to decompose load balancing by timescale, achieving up to 1.58× throughput gains over Megatron-LM with negligible overhead.
Editor's noteMust-read. Extends route replay from the algorithmic layer (last issue #1's PR² predictive replay) to the system layer: leveraging the property that routes are known before training in RL rollout-training, performing expert reordering across batches and expert replication within batches, matching hierarchical network bandwidth, and absorbing hot expert fluctuations at the microbatch granularity. Compared to PR²'s focus on training-inference consistency, ReLibra uses replay information for system scheduling, reporting full route statistics and dense baselines, with throughput improvements up to 1.6× over Megatron-LM and 1.2× over oracle-load EPLB—a significant increment in the route replay direction.
Sparse MoE routing works through coherent overlap, not geometric complementarity, as selected experts share subspaces yet still improve prediction, making geometric similarity alone insufficient for pruning decisions.
Editor's noteMust-read. Uses ESSI and a prefix-controlled 2×2 factorial design to separate candidate quality, context opportunity, and their interaction, consistently finding the interaction to be negative across 39 factorial units—actual prefixes narrow rather than amplify the geometric advantage of selected experts. This is a positive structural complement to last issue #1's causal audit (From Observation to Intervention) and modular causal tests: "coherent overlap" shows geometric overlap coexists with functional value, providing key nuance for pruning and route design.
Training-inference mismatch (TIM) between rollout and training engines, not just algorithmic choices, can independently cause catastrophic RL training collapse, as shown by the new VeXact engine.
Editor's noteIsolates training-inference mismatch from the route decision layer, building the zero-mismatch rollout engine VeXact as a diagnostic baseline, proving that token-level numerical mismatch alone can independently cause RL training collapse, and revealing via zero-centered loss contribution analysis that it disrupts training through sign-imbalanced gradient distortion. Compared to last issue #1's PR² focusing on route staleness, this work addresses lower-level numerical implementation differences and serves as a calibration benchmark for evaluating all route replay/compensation algorithms.
MoE-Prefill decouples expert placement from activation routing, asynchronously gathering expert weights to eliminate AllToAll and achieve 1.35–1.37× throughput over baselines.
Editor's noteProposes a prefill-only AsyncEP paradigm: collecting experts by weight (background AllGather) rather than activation (layer-wise AllToAll), eliminating on-path communication and route-imbalanced stragglers, with front-end co-enforced physically derived saturation thresholds. Compared to last issue #1's UltraEP and TEMPO, this is a fundamental alternative to synchronous activation routing, achieving 1.35–1.37× throughput gains on Qwen3-235B-A22B across four hardware configurations, formalizing prefill-only as a distinct service category.
ExpertPlex shares MoE expert weights across prefill and decode while disaggregating attention, achieving up to 5.65x goodput gains via tile-level preemptive GPU scheduling.
Editor's noteProposes a hybrid split-co-located architecture: sharing MoE experts across stages while separating attention modules, eliminating over 95% of duplicate model weights, with adaptive persistent kernels for tile-level preemption and attention-initiated one-sided communication. Compared to last issue #1's UltraEP/TEMPO, the increment lies in cross-stage expert sharing and fine-grained tile scheduling, achieving up to 2.01× goodput improvement on MiniMax-M2.7 and GLM-5.1-FP8.
ELDR reduces median decode latency by 5.9–13.9% via expert locality routing, grouping requests by prefill-predicted expert activations.
Editor's noteAdds an expert locality routing axis orthogonal to load balancing for PD-disaggregated serving: constructing expert signatures from prefill activations to predict decoding-phase expert overlap, paired with block-granularity signature caches co-indexed with KV caches to maintain prefix cache consistency. Compared to last issue #1's TEMPO (load balancing only) and UltraEP (rebalancing only within EP ranks), ELDR reduces median TPOT by 5.9–13.9% on 40-GPU deployments without changing model outputs.
Standard top-k MoE routers are systematically misaligned on the low-confidence tokens driving hard reasoning, leaving better equal-compute routes unused, yet router-only updates can recover them.
Editor's noteFirst counterfactual route quality evaluation of trained top-k routers: comparing standard routing against sampled alternative routes with equal compute within frozen models, finding that routers align well on confident tokens but systematically fail on fragile tokens driving hard reasoning, with the pattern consistent across four model families. Compared to last issue #1's From Observation to Intervention focusing on expert importance, this work turns to the quality of route selection itself, proving that updating only the final-layer router (<0.001% parameters) can shift pass@K—high diagnostic value.
