Routing Foresight Takes Over the System Layer: Predictive Placement and Micro-Step Scheduling
Highlights
This issue focuses on the concentrated burst of "predictive" and "micro-step" ideas across the MoE system layer: routing replay extends from training-inference consistency to server-side expert placement and RL post-training micro-step scheduling, while scaling behavior gains a new axis of loop depth.
On the serving side, Director advances expert placement from "reactive afterthought" to "online foresight": it predicts routing based on queued requests (via cascaded predictors or low-bit quantized replicas) to optimize expert layout in advance, overlaps migration with computation to achieve near-zero-downtime reconfiguration, and centers on a polynomial-time (1+ε) approximate placement optimizer. This complements Gimbal (cross-engine request scheduling) from Issue 3—Gimbal focuses on cross-engine coordination, while Director focuses on intra-engine predictive placement. The related TAOT is the first to model dynamic replica placement as a topology-aware optimal transport problem, explicitly adding cross-node communication costs to the objective and replacing hard graph constraints with soft topological preference flows.
On the RL post-training side, ForeMoE exploits the structural property that "rollout routing is known before training": it proactively guides expert reconfiguration during recompute and policy update stages, addressing the load pattern of stable steps but volatile micro-steps, and decomposes the NP-hard problem into a four-stage planner. RoutePack further uses routing replay signals simultaneously for intra-layer expert reordering and attention-aware sample packing, explicitly co-optimizing attention and expert load for the first time. Both are direct increments over ReLibra from Issue 2.
On the scaling behavior front, SMELT is the first to study MoE looped Transformers under three simultaneous budgets—per-token FLOPs, total parameters, and KV cache—introducing loop depth as a new axis orthogonal to sparsity and granularity, and provides a mechanism explanation for reduced attention sink on the second pass. On the routing consistency front, RARE projects representation-engineering perturbations onto the null space of the router matrix, enabling behavior control while keeping the native routing unchanged, and empirically demonstrates that "routing is sensitive to semantics, not behavior."
Community and Trends
On the debate over whether load balancing is necessary, Least-Loaded Expert Parallelism takes a contrarian stance: unbalanced routing is a natural and desirable property of well-trained MoE, and aggressively forcing balance (via auxiliary losses or bias corrections) risks destroying expert specialization. Instead, it proposes LLEP, which only migrates excess tokens and corresponding expert weights to the least-loaded devices during overload. This directly tensions with the default "auxiliary-loss-free bias balancing" direction and cross-validates the conclusion from Issue 3's Routers Learn the Geometry that "auxiliary losses disrupt geometric coupling."
On replication granularity, CRAFT advances expert replication from EPLB's uniform per-layer allocation to memory-budget-aware, per-layer on-demand allocation, assigning replicas based on each layer's replication benefit, achieving 1.14× throughput improvement with roughly half the memory budget at 100B–1T scale. In the community, a discussion on the DeepSeek-V3 repo proposes replacing learned unbounded logits with profile-based bounded routing, simultaneously eliminating z-loss and bias balancing, and reinterprets the bias term as a "staleness sentinel"—a speculative idea but a valuable hypothesis generator for auxiliary-loss-free balancing and routing stability.
Open Questions
-
Director and ForeMoE both rely on the accuracy of routing prediction—when request patterns shift rapidly or prediction errors grow, where is the boundary of predictive benefit? Is there a quantifiable trade-off curve between prediction error and migration cost?
-
SMELT adds loop depth as a scaling axis and reports that the second pass reuses a small core subset of routing—does deeper looping imply increasingly concentrated routing choices, thereby weakening expert specialization? How do loop depth, sparsity, and granularity interact?
-
Can ReBA's composition-shift law generalize to load drift in pure-text MoE (long context, mixed tasks)? How does its "modality-separated" design reconcile with auxiliary-loss-free bias balancing?
-
RARE demonstrates that routing is sensitive to semantics, not behavior—does this imply that representation interventions can only change "behavior" but never "semantic assignment"? How does this reconcile with causal evidence on expert specialization?
Papers in this issue
DIRECTOR reduces MoE serving latency by up to 60.9% using proactive expert placement with a lightweight predictor and a provably near-optimal (1+ε)-approximation algorithm.
Editor's noteAdvances expert placement from "reactive afterthought" to "online foresight": predicts routing based on queued requests (via cascaded predictors or low-bit quantized replicas) to optimize expert layout in advance, overlaps migration with computation for near-zero-downtime reconfiguration, and centers on a polynomial-time (1+ε) approximate placement optimizer. Compared to Gimbal (cross-engine request scheduling) from Issue 3 and UltraEP (reactive rebalancing) from Issue 1, Director is the first to apply predictive replay ideas to intra-engine predictive placement, reducing end-to-end latency by 11–55% on Mistral/DeepSeek/Qwen.
ForeMoE leverages the foreseeability of routing decisions in RL post-training to achieve up to 1.45x speedup via micro-step-level expert load balancing.
Editor's noteTargets the unique load characteristics of RL post-training—stable at step level but volatile at micro-step level—by exploiting the fact that rollout routing is known before training. It proactively guides expert reconfiguration during recompute and policy update stages, decomposes the NP-hard problem into a four-stage planner, and uses CPU-assisted and GPU-direct complementary transmission paths to overlap migration. Compared to ReLibra from Issue 2 (which only optimizes policy update stage and uses GPU-direct only), ForeMoE's increment is optimizing the recompute stage and supporting expert migration, achieving up to 1.45× speedup over veRL on 64 GPUs.
The SMELT architecture, looping the middle half of MoE layers twice, beats matched baselines by saving 6.8-18% of training compute.
Editor's noteFirst to study MoE looped Transformers under three simultaneous budgets—per-token FLOPs, total parameters, and KV cache—fitting an independent Chinchilla-style scaling law and showing that looping still saves 6.8–18.0% training FLOPs under matched budgets. Compared to Slicing and Dicing from Issue 3 (sparsity-granularity decoupling), this paper introduces loop depth as a new scaling axis and provides a mechanism explanation (second-pass visits reduce attention sink, routing reuses a small core subset), extending sparse-structure scaling from "how wide" to "how deep."
RARE projects behavioral perturbations onto the router's null space, enabling effective representation steering in MoE LLMs without disrupting expert allocation.
Editor's noteProjects representation-engineering perturbations onto the null space of the router matrix, enabling behavior control while keeping native routing unchanged, and further corrects downstream routing drift. Compared to Launch-Bound from Issue 3 (causal routing replay separating routing and weight errors), this paper is the first to treat the "routing-invisible subspace" as an explicit constraint for representation interventions, systematically compares five perturbation estimators on six heterogeneous MoE models, and empirically demonstrates that "routing is sensitive to semantics, not behavior"—providing a new alternative path for training-inference routing consistency.
ReBA's geometry-guided load balancing, with separate image and text losses and image-level routing, cuts load imbalance up to fivefold across token mixes while preserving accuracy.
Editor's noteMoves load balancing from "mixed token distribution" down to "modality-conditional distribution": proves that the standard Switch auxiliary loss, under vision-language mixed inputs, learns modality-complementary load errors that cancel out only at a single mixing ratio, and derives an analytical composition-shift parabola law (curvature determined by the image-text conditional load gap). Compared to FreeBalance (pre-routing load prediction) from Issue 3 and ReLibra (routing replay scheduling) from Issue 2, this is the first load-balancing training objective targeting multi-modal token mixing drift, with direct system implications for service load drift caused by image resolution/patch changes.
TAOT jointly optimizes expert-replica placement and communication cost via optimal transport, achieving 42.82% end-to-end training speedup over Megatron-LM with up to 74% lower communication cost.
Editor's noteFirst to model dynamic expert replica placement as a topology-aware optimal transport problem: the objective simultaneously optimizes peak-load reduction and cross-node communication cost, using Sinkhorn-Knopp iterations to generate soft topological preference flows instead of hard topology constraints. Compared to UltraEP from Issue 1 (load balancing only) and ELDR from Issue 2 (expert locality routing), this paper explicitly introduces a topology cost matrix into replica placement, achieving the best balance quality and lowest communication cost (74% reduction vs. LPLB) at EP=32.
RoutePack jointly optimizes expert placement and data packing using routing replay, improving MoE reinforcement learning token throughput by up to 14.89%.
Editor's noteFirst to use routing replay signals simultaneously for intra-layer expert reordering and attention-aware sample packing, forming a hierarchical joint planner: first reorders experts by aggregating demand over optimizer windows, then uses population annealing to search for EDP-shard-aware rectangular packing, with the objective explicitly combining attention linear/quadratic terms and physical EP rank tail. Compared to ReLibra from Issue 2 (inter-batch reordering + intra-batch replication), the increment is that the packing objective explicitly models attention and expert load and constructs capacity-feasible execution rows, improving token throughput by up to 14.89% on Ling-3.0.
