# ISER-MoE: Dynamic In-Switch Replication of Hot Experts for Load Balancing in Expert-Parallel

> ISER-MoE uses in-switch multicast to replicate hot experts per invocation, removing 98-99% of MoE rank imbalance and boosting prefill throughput by 17.4%, exceeding perfectly balanced routing.

- **Source:** [arXiv](https://arxiv.org/abs/2611.09008)
- **Published:** 2026-11-14
- **Permalink:** https://picx.dev/p/lnxp6F
- **Whiteboard:** https://picx.dev/p/lnxp6F/image

## Summary

## Summary (Overview)

- **ISER-MoE** (In-Switch Expert Replication for MoE) is a novel system for load balancing in expert-parallel Mixture-of-Experts (MoE) layers, built in TensorRT-LLM, that replicates "hot" (heavily loaded) experts using NVLink in-switch multicast.
- The key innovation is **fixing the multicast groups** rather than the replicas: experts are replicated to pre-defined aligned GPU groups, enabling each byte of an expert's weights to leave its owner GPU only once, regardless of replica count.
- The **HALO-Q scheduler** (Hot-Aligned, Locality-Optimal expert replication with Quota repair) removes 98–99% of excess load in trace replay, balancing ranks per MoE layer invocation.
- **SAMI** (Scalable Asynchronous Multi-port In-switch broadcast) spreads injection over group members' ports, overcoming the bandwidth bottleneck of single-port multicast.
- End-to-end results on GB300 show **17.4% throughput improvement on GSM8K** and **7.2% on agentic prompts** at EP8, exceeding gains from perfectly balanced routing (13.5% and 6.0%).

## Introduction and Theoretical Foundation

### Background
Expert-parallel MoE layers run at the pace of the most loaded GPU. Under disaggregated serving of large models like DeepSeek-V4-Pro, prefill workloads exhibit significant rank imbalance, with the rank imbalance ratio $R$ (max/mean routed tokens per rank) averaging 1.21–1.45 at EP4–8, and reaching 1.69 at EP8 on GSM8K.

### Key Observations
1. **Long-tailed expert load**: The four busiest of 384 experts receive 13.02% of a layer's assignments on average (vs. 1.04% if uniform). Replicating these few hot experts onto underloaded ranks can absorb most of the imbalance.
2. **Static plans are insufficient**: History-based plans remove standing skew but not drift between batches, motivating per-invocation replication.

### Three Core Motivations
1. **Motivation 1**: Routed-expert computation hides only 1–2 expert copies at peak rates, but the **shared expert** (which needs no token traffic) provides a window for copies to all helper slots.
2. **Motivation 2**: Point-to-point fan-out makes the owner a broadcast hot spot—a hot rank may need to send 7–10 payloads per invocation.
3. **Motivation 3**: Baseline broadcasts (software rings, scatter+all-gather, single-port in-switch multicast) all pay latency or bandwidth overheads.

## Methodology

### HALO-Q Scheduler
A single GPU kernel per MoE layer invocation with two objectives:
- **Objective 1 (Balance)**: Bring every rank's routed load down to the water level $T = \lceil \sum_r L_r / P \rceil$
- **Objective 2 (Locality)**: Maximize locally-calculated tokens among balanced routings

Five stages:
1. **Count exchange**: Ranks exchange route counts via symmetric NVLink buffer
2. **Placement**: Greedy selection of busiest rank's heaviest home expert, assigned to aligned groups with free slots
3. **Slot coloring**: Each placed expert gets one helper index free on all group members
4. **Quota repair**: Circle-method tournament over disjoint rank pairs to cancel opposite token flows
5. **Route rewrite**: Local-first assignment of surplus routes to hosts' spare quotas

### Multi-Level Symmetric Multicast
- Level $\ell$ splits $P$ ranks into $2^\ell$ aligned groups of $g_\ell = P/2^\ell$ consecutive ranks
- Groups are fixed at initialization; HALO-Q picks a level per expert
- **Symmetric helper slots**: If every rank belongs to groups of at most $S$ broadcasts, coloring from coarse to fine with the smallest free index never fails

### SAMI Multi-Port Broadcast
Each member pulls a slice from the owner and stores it through its own ingress port:
- Member $i$ pulls slices $j \equiv i \pmod{g}$ from the owner
- Stores each slice through its own port to the multicast address
- The switch delivers to every member

For an expert of $W$ bytes, the owner sends $W$ and each member injects $W/g$, reducing single-port output from $gW$ to $W$.

### Runtime Overlap
- COPY stream (highest priority) runs HALO-Q and replication beside the shared expert
- MAIN stream computes the shared expert, then waits only for the plan
- Only helper tiles wait at a READY gate for the replication kernel's completion

### Key Formula: Math/Comm Ratio
$$\frac{\text{math}}{\text{comm}} = \frac{6\tau k H_e I}{\underbrace{2\frac{P-1}{P}\tau k H_e b}_{\text{dispatch} + \text{combine}} + \underbrace{nW}_{\text{copies}}}$$

where $\tau$ = tokens per rank, $k$ = top-k routing, $H_e$ = expert input width, $I$ = intermediate width, $b$ = bytes per element (0.5625 B for NVFP4), $n$ = copies, $W = 3H_e I b$ = expert weight size.

## Empirical Validation / Results

### E1: Simulated Scheduler Rank Imbalance
| Metric | EP4 | EP8 | EP16 | EP24 |
|--------|-----|-----|------|------|
| R without balancing | 1.20 | 1.44 | 1.84 | 2.24 |
| R with HALO-Q (S=4) | 1.002 | 1.006 | 1.010 | 1.020 |
| Excess load removed | 99.0% | 98.6% | 98.8% | 98.4% |

### E2: SAMI Broadcast Performance
- **vs. single-port multicast**: Speedup grew from 1.13× at g=4 to 2.03× at g=8 and 3.76× at g=16
- **vs. tuned ncclBroadcast**: SAMI was 1.07/1.21/1.34× faster for one expert at 8/12/16 GPUs
- **Overlap with shared expert**: SAMI lengthened the span by only 2.0/0.6/0.3% at 8K/16K/32K tokens/rank vs. 36.3/14.7/2.9% for owner-direct multicast

### E3: MegaMoE Kernel Time
- Power-law routing: kernel time reduced by 17.2–34.1% (1.21–1.52×), within 0.3–3.5% of balanced routing
- Trace routing: mean time fell from 1,999.3 to 1,624.3 µs (1.23×), 2.1% above balanced routing
- Every layer improved, from 1.3% (R=1.12) to 42.8% (R=2.36)

### E4: End-to-End Prefill Serving (GB300)
| Workload | ON gain | FB gain |
|----------|---------|---------|
| GSM8K (DEP8) | **17.4%** | 13.5% |
| GSM8K (DEP6) | 9.5% | 10.8% |
| AgentPerf (DEP8) | **7.2%** | 6.0% |
| AgentPerf (DEP6) | 1.6% | 3.8% |

### E5: Disaggregated Serving (GB300 NVL72)
- P90 request latency improved by 3.1–4.3%
- Output throughput per GPU improved by 1.4–1.9% across all four configurations

## Theoretical and Practical Implications

1. **In-switch multicast for weights**: ISER-MoE is the first to replicate experts per invocation via in-switch multicast, demonstrating that NVLink SHARP can be effectively used for weight distribution when groups are fixed and replicas are flexible.

2. **Shared expert as copy window**: The insight that the shared expert provides a natural hiding window for replication is broadly applicable to MoE architectures with shared experts.

3. **Group alignment principle**: Fixing multicast groups at initialization (rather than per-invocation) sidesteps the MLID limitation and membership constraints, providing a practical template for in-network weight distribution.

4. **SAMI generalizes in-network collectives**: Applying scatter+all-gather principles to weight replication with an owner possibly outside the group extends NVLS capabilities.

5. **Practical deployment**: DynamicEPLB in TensorRT-LLM is available as an open pull request, making the approach accessible for production MoE serving.

## Conclusion

ISER-MoE demonstrates that per-invocation replicas of hot experts can remove most rank imbalance in expert-parallel MoE layers when weights arrive via in-switch multicast to fixed aligned groups. Key findings:

- HALO-Q removes 98–99% of excess load in trace replay
- Throughput gains exceed those from perfectly balanced routing (17.4% vs. 13.5% on GSM8K)
- SAMI overcomes single-port multicast bandwidth limits with speedups up to 3.76× at g=16

**Future work** includes:
- Hardware comparison with static plans
- Application to generation and training phases
- Attention-side imbalance handling
- Replication across NVLink domains
- Reducing the 12 µs planning time at EP32 (2× its EP8 time)

---

_Markdown view of https://picx.dev/p/lnxp6F, served by PicX — AI-generated visual whiteboard summaries of research papers._