Symmetric Recovery: A New Principle of Routing Consistency for FP4 MoE RL
Highlights of This Issue
The main theme of this issue is new evidence for "training-inference consistency" at both the quantization and execution levels. On the FP4 RL side, Aligning Training and Rollout Policies for End-to-End FP4 RL on Mixture-of-Experts LLMs is the first to systematically prove that recovery recipes effective on dense models (RolloutResQ) are not only ineffective but harmful on MoE, and attributes the root cause to the asymmetry of quantization recovery—only when residual correction is applied simultaneously on both the training and inference sides (symmetric recovery) can routing consistency be restored, reducing the accuracy gap from 20+ points to within 1 point on Qwen3-30B-A3B, and providing quantitative evidence of routing alignment via PCC and top-k matching rate. This echoes several community engineering efforts that treat "eliminating numerical inconsistency" as an alternative path to routing replay (vLLM's bitwise-consistent training-inference consistency, IsoExec, vime × RL-Kernel × AMD's ROCm bit-level consistency): one path performs symmetric correction within the quantization domain, the other achieves bit-level alignment at the kernel layer, and together they advance alternative paths beyond "routing replay" to the level of execution details.
The second line is the raised evaluation bar for inference-time expert deactivation and reduction. Matching the Training-Time Weight Rule in Inference-Time Expert Deactivation is the first to conduct a controlled comparison of four deactivation implementations on the same set of disabled experts, directly linking "high-loss implementations" to the training-time weight rule (original probabilities vs. normalized weights) via a weight-sum model, providing quantitative evidence of perplexity gaps ranging from 14 to 28,740 times across six checkpoints of OLMoE/DeepSeek-V2-Lite/Granite/Qwen3; SafeStart is the first to extend the evaluation of expert reduction from accuracy to safety behavior, finding that the cost of reduction is concentrated in the prefill and first-token stages, directly challenging the recommended operating point of You Only Need 2/3 of the Chosen Experts from Issue 12 in safety scenarios.
On the serving side, the debate over the benefits of AFD has seen both positive and negative empirical evidence. FastAFD conducts an end-to-end comparison of AFD with workload-tuned wide EP on rack-level NVLink in GB200 NVL72, yielding a conditional conclusion of 0.99–1.17×, and provides a three-factor decomposition and an analytical model with a prediction error of 2.9%; Should You Split Decode? gives a negative result under a unified power budget, showing no robust deployment-level advantage for AFD. Meanwhile, Accelerating Offloading-Based MoE Inference by Interleaving Concurrent Requests is the first to falsify the benefits of the "expert cache + prefetch" combination, proposing request-level runahead scheduling to replace predictive prefetching.
Community and Updates
Why Does Training-Inference Mismatch Destabilize LLM RL? quantifies training-inference numerical inconsistency as a mechanism that flips routing decisions layer by layer and compounds them, pointing out that it disrupts the importance sampling assumptions of GRPO/GSPO, and proposes routing replay as a remedy—complementing the symmetric recovery of MoEResQ and the bit-level consistency of vLLM as three complementary repair paths. Why Speculative Decoding Works Better Than Predicted on Sparse MoE Models is the first to quantify the token-level expert overlap brought by fine-grained experts (20–45% vs. expected 4–12%), providing a previously unquantified new statistical dimension for the scaling behavior of sparse models, and directly explaining the phenomenon in FastAFD where speculative decoding benefits light-expert models the most.
Open Questions
-
MoEResQ proves that symmetric recovery is a key principle for FP4 MoE RL, but the routing layer remains in BF16 (only 0.041% of parameters)—if the routing layer is also compressed into FP4, can symmetric recovery still maintain routing consistency? Can the symmetry principle of quantization recovery be generalized to other precision domains (e.g., NVFP4) and larger scales?
-
Matching the Training-Time Weight Rule proves that the training-time weight rule is the first-order factor explaining differences in deactivation implementations, but the deactivation set is based on a fixed C4 calibration corpus—when the deactivation set is determined by dynamic routing statistics (rather than a fixed corpus), does the prediction of the weight-sum model still hold?
-
FastAFD and Should You Split Decode? give opposite conclusions on the benefits of AFD—the former measures 0.99–1.17× on NVLink within a rack, the latter gives a negative result with no robust advantage under a unified power budget. Does the divergence stem from hardware topology, power accounting methodology, or baseline tuning intensity? Can the benefit boundary of AFD be consolidated into an analytical criterion?
-
Accelerating Offloading-Based MoE Inference proves that the larger the cache, the lower the prefetch accuracy, and runahead scheduling replaces prefetching—when the cache policy itself is dynamic (such as Rimuru's elastic page pool in Issue 12), will new coupling emerge between runahead and cache replacement?
Papers in this issue
Symmetric residual correction on both training and inference sides, unlike one-sided rollout correction, restores near-FP16 accuracy in FP4 RL for Mixture-of-Experts models.
Editor's noteMust-read. The first to systematically prove that the FP4 RL recovery recipe effective on dense models (RolloutResQ) is not only ineffective but harmful on MoE, and attributes the root cause to the asymmetry of quantization recovery—only when residual correction is applied simultaneously on both the training and inference sides (symmetric recovery) can routing consistency be restored, reducing the accuracy gap from 20+ points to within 1 point on Qwen3-30B-A3B. Compared to TRACE's rollout-guided quantization-aware training in Issue 11, this paper explicitly takes symmetry as the principle and reports complete routing alignment metrics with dense baselines via PCC and top-k matching rate.
Matching the training-time weight rule is critical: mismatched expert deactivation implementations can inflate perplexity by up to 28,740x, with the most damaging rule predicted by model type.
Editor's noteMust-read. The first to conduct a controlled comparison of four inference-time expert deactivation implementations on the same set of disabled experts, directly linking 'high-loss implementations' to the training-time weight rule (original probabilities vs. normalized weights) via a weight-sum model, providing quantitative evidence of perplexity gaps ranging from 14 to 28,740 times across six checkpoints of OLMoE/DeepSeek-V2-Lite/Granite/Qwen3. Compared to the controlled comparison in Issue 11's Don't Skip the Baseline and the normalization decoupling in Issue 5's Training-Free Halving, this paper establishes 'whether the training-time weight rule is matched' as the first-order factor explaining implementation differences, and extends to predictions of width adjustments with varying numbers of active experts.
FastAFD shows attention-FFN disaggregation beats per-workload-tuned vLLM wide EP by up to 1.17x on GB200 NVL72, while an analytical model predicts its speedups within 2.9% mean error.
Editor's noteMust-read. The first to conduct an end-to-end empirical comparison of AFD with workload-tuned wide EP on rack-level NVLink in GB200 NVL72, providing a three-factor decomposition (batch gain × overlap gain × disaggregation cost) and a zero-refit analytical model with a prediction error of 2.9%, and the first to measure the impact of speculative decoding on the optimal A:F ratio and benefits of AFD. Compared to Afore's micro-batch-level demand prefetching in Issue 10, this paper answers with empirical measurements rather than analytical models whether AFD is worthwhile on rack-level NVLink, yielding a conditional conclusion of 0.99–1.17×.
MoE router states encode reasoning behaviors transferable across domains, enabling training-free steering that cuts generated tokens by a median of 30.4% without hurting accuracy.
Editor's noteMust-read. The first to use inference behaviors (planning/reflection, etc., six categories) as explicit readable signals for MoE routing, using behavior-associated experts for inference-time expert suppression to reduce overthinking, reducing the median number of generated tokens by 30.4% (up to 63.3%) across four models, and providing a mechanism analysis that suppressing planning rather than reflection is effective. Compared to RouterInterp's SAE feature micro-domain proxy in Issue 9, this paper advances routing readability to the level of inference behaviors, reporting complete routing statistics with dense baselines.
Bevy replaces expert prefetching with runahead inference, scheduling ready requests to reveal actual expert needs and achieving 1.26–3.46x throughput over llama.cpp.
Editor's noteMust-read. The first to systematically falsify the benefits of the 'expert cache + prefetch' combination: the larger the cache, the lower the prefetch accuracy (the cache already contains easily predictable experts, making remaining candidates harder to predict), and proposes runahead inference—letting ready requests run ahead to subsequent layers to expose real expert demands, replacing predictive prefetching with request scheduling. Compared to Rimuru's elastic expert cache in Issue 12, the increment here is at the scheduling layer rather than the memory layer, and does not change model outputs, reporting complete routing statistics such as expert miss counts and prefetch waste rates.
Expert reduction in MoE serving weakens model refusal by up to 18.4 percentage points, but running all experts for the prompt and first 8 tokens restores safety while keeping most speedup.
Editor's noteMust-read. The first to systematically audit the safety cost of expert reduction (top-k truncation, top-p, threshold skipping, LExI, ACE) in serving engines, locating the cost in the prefill and first-token stages, and proposing SafeStart (full experts at the beginning, reduction afterward) as an engine-level fix. Compared to Issue 12's You Only Need 2/3 of the Chosen Experts, which focuses only on accuracy and compute matching, this paper adds a safety dimension, directly challenging the applicability of its recommended operating point in safety scenarios, and provides a position-dependent budget allocation scheme.
SiliconForge shows attention/FFN disaggregation yields no robust advantage over prefill/decode disaggregation, with prefill provisioning, memory bandwidth, and pool granularity driving LLM serving performance more than the split itself.
Editor's noteWorth reading. The first to treat serving decomposition (P/D vs. P/D+AFD) and hardware allocation as two independent axes, conducting a forward-looking comparison under a unified power budget and calibrated model, finding that AFD has no robust deployment-level advantage, and that losses from early hardware commitments can be reduced from 36% to 9% through re-optimized mapping and pool size. Compared to Afore's AFD micro-batch-level demand prefetching in Issue 10, this is the first systematic negative-result study questioning the benefits of AFD, and its two-stage test (9% envelope + sign robustness) provides a reusable burden of proof for evaluating serving decomposition.
Softmax routing collapse in MoE models is transient under trainable experts, with recovery time scaling logarithmically or polynomially depending on gradient structure.
Editor's noteWorth reading. The first to formalize the collapse and recovery of softmax routing as continuous-time gradient flow dynamics, providing a face-selection theorem for the stationary-loss barrier under fixed experts, a separation of finite-time concentration and internal limits under trainable experts, and exact scaling laws for recovery time (logarithmic vs. power-law). Compared to the control-theoretic perspective of ID Balancing in Issue 8, this is the first theoretical work providing provable time scales for routing collapse-recovery, and its recovery time scaling can serve as a new theoretical benchmark for evaluating routing replay or intervention methods.
