Summary (Overview)

• ISER-MoE (In-Switch Expert Replication for MoE) is a novel system for load balancing in expert-parallel Mixture-of-Experts (MoE) layers, built in TensorRT-LLM, that replicates "hot" (heavily loaded) experts using NVLink in-switch multicast.
• The key innovation is fixing the multicast groups rather than the replicas: experts are replicated to pre-defined aligned GPU groups, enabling each byte of an expert's weights to leave its owner GPU only once, regardless of replica count.
• The HALO-Q scheduler (Hot-Aligned, Locality-Optimal expert replication with Quota repair) removes 98–99% of excess load in trace replay, balancing ranks per MoE layer invocation.
• SAMI (Scalable Asynchronous Multi-port In-switch broadcast) spreads injection over group members' ports, overcoming the bandwidth bottleneck of single-port multicast.
• End-to-end results on GB300 show 17.4% throughput improvement on GSM8K and 7.2% on agentic prompts at EP8, exceeding gains from perfectly balanced routing (13.5% and 6.0%).

Introduction and Theoretical Foundation

Background

Expert-parallel MoE layers run at the pace of the most loaded GPU. Under disaggregated serving of large models like DeepSeek-V4-Pro, prefill workloads exhibit significant rank imbalance, with the rank imbalance ratio $R$ (max/mean routed tokens per rank) averaging 1.21–1.45 at EP4–8, and reaching 1.69 at EP8 on GSM8K.

Key Observations

1. Long-tailed expert load: The four busiest of 384 experts receive 13.02% of a layer's assignments on average (vs. 1.04% if uniform). Replicating these few hot experts onto underloaded ranks can absorb most of the imbalance.
2. Static plans are insufficient: History-based plans remove standing skew but not drift between batches, motivating per-invocation replication.

Three Core Motivations

1. Motivation 1: Routed-expert computation hides only 1–2 expert copies at peak rates, but the shared expert (which needs no token traffic) provides a window for copies to all helper slots.
2. Motivation 2: Point-to-point fan-out makes the owner a broadcast hot spot—a hot rank may need to send 7–10 payloads per invocation.
3. Motivation 3: Baseline broadcasts (software rings, scatter+all-gather, single-port in-switch multicast) all pay latency or bandwidth overheads.

Methodology

HALO-Q Scheduler

A single GPU kernel per MoE layer invocation with two objectives:

• Objective 1 (Balance): Bring every rank's routed load down to the water level $T = \lceil \sum_r L_r / P \rceil$
• Objective 2 (Locality): Maximize locally-calculated tokens among balanced routings

Five stages:

1. Count exchange: Ranks exchange route counts via symmetric NVLink buffer
2. Placement: Greedy selection of busiest rank's heaviest home expert, assigned to aligned groups with free slots
3. Slot coloring: Each placed expert gets one helper index free on all group members
4. Quota repair: Circle-method tournament over disjoint rank pairs to cancel opposite token flows
5. Route rewrite: Local-first assignment of surplus routes to hosts' spare quotas

Multi-Level Symmetric Multicast

• Level $\ell$ splits $P$ ranks into $2^\ell$ aligned groups of $g_\ell = P/2^\ell$ consecutive ranks
• Groups are fixed at initialization; HALO-Q picks a level per expert
• Symmetric helper slots: If every rank belongs to groups of at most $S$ broadcasts, coloring from coarse to fine with the smallest free index never fails

SAMI Multi-Port Broadcast

Each member pulls a slice from the owner and stores it through its own ingress port:

• Member $i$ pulls slices $j \equiv i \pmod{g}$ from the owner
• Stores each slice through its own port to the multicast address
• The switch delivers to every member

For an expert of $W$ bytes, the owner sends $W$ and each member injects $W/g$, reducing single-port output from $gW$ to $W$.

Runtime Overlap

• COPY stream (highest priority) runs HALO-Q and replication beside the shared expert
• MAIN stream computes the shared expert, then waits only for the plan
• Only helper tiles wait at a READY gate for the replication kernel's completion

Key Formula: Math/Comm Ratio

$$\frac{\text{math}}{\text{comm}} = \frac{6\tau k H_e I}{\underbrace{2\frac{P-1}{P}\tau k H_e b}_{\text{dispatch} + \text{combine}} + \underbrace{nW}_{\text{copies}}}$$

where $\tau$ = tokens per rank, $k$ = top-k routing, $H_e$ = expert input width, $I$ = intermediate width, $b$ = bytes per element (0.5625 B for NVFP4), $n$ = copies, $W = 3H_e I b$ = expert weight size.

Empirical Validation / Results

E1: Simulated Scheduler Rank Imbalance

E2: SAMI Broadcast Performance

• vs. single-port multicast: Speedup grew from 1.13× at g=4 to 2.03× at g=8 and 3.76× at g=16
• vs. tuned ncclBroadcast: SAMI was 1.07/1.21/1.34× faster for one expert at 8/12/16 GPUs
• Overlap with shared expert: SAMI lengthened the span by only 2.0/0.6/0.3% at 8K/16K/32K tokens/rank vs. 36.3/14.7/2.9% for owner-direct multicast

E3: MegaMoE Kernel Time

• Power-law routing: kernel time reduced by 17.2–34.1% (1.21–1.52×), within 0.3–3.5% of balanced routing
• Trace routing: mean time fell from 1,999.3 to 1,624.3 µs (1.23×), 2.1% above balanced routing
• Every layer improved, from 1.3% (R=1.12) to 42.8% (R=2.36)

E4: End-to-End Prefill Serving (GB300)

E5: Disaggregated Serving (GB300 NVL72)

• P90 request latency improved by 3.1–4.3%
• Output throughput per GPU improved by 1.4–1.9% across all four configurations

Theoretical and Practical Implications

1. In-switch multicast for weights: ISER-MoE is the first to replicate experts per invocation via in-switch multicast, demonstrating that NVLink SHARP can be effectively used for weight distribution when groups are fixed and replicas are flexible.

2. Shared expert as copy window: The insight that the shared expert provides a natural hiding window for replication is broadly applicable to MoE architectures with shared experts.

3. Group alignment principle: Fixing multicast groups at initialization (rather than per-invocation) sidesteps the MLID limitation and membership constraints, providing a practical template for in-network weight distribution.

4. SAMI generalizes in-network collectives: Applying scatter+all-gather principles to weight replication with an owner possibly outside the group extends NVLS capabilities.

5. Practical deployment: DynamicEPLB in TensorRT-LLM is available as an open pull request, making the approach accessible for production MoE serving.

Conclusion

ISER-MoE demonstrates that per-invocation replicas of hot experts can remove most rank imbalance in expert-parallel MoE layers when weights arrive via in-switch multicast to fixed aligned groups. Key findings:

• HALO-Q removes 98–99% of excess load in trace replay
• Throughput gains exceed those from perfectly balanced routing (17.4% vs. 13.5% on GSM8K)
• SAMI overcomes single-port multicast bandwidth limits with speedups up to 3.76× at g=16

Future work includes:

• Hardware comparison with static plans
• Application to generation and training phases
• Attention-side imbalance handling
• Replication across NVLink domains
• Reducing the 12 µs planning time at EP32 (2× its EP8 time)