# Launch-Bound and Substitutable: Why Three Inference Optimizations Fail to Pay Off in Mixture-of-Experts Models

> Fused kernels, INT4 quantization, and torch.compile fail on MoE models because they are launch-bound, not arithmetic-bound, with routing drift causing only 2.7% of quality loss.

- **Source:** [arXiv](https://arxiv.org/abs/2608.26612)
- **Published:** 2026-08-29
- **Permalink:** https://picx.dev/p/nJ5NaI
- **Whiteboard:** https://picx.dev/p/nJ5NaI/image

## Summary

## Summary (Overview)

- This paper empirically evaluates three standard inference optimizations—fused Triton kernels, INT4/INT8 quantization, and torch.compile—on three Mixture-of-Experts (MoE) models: OLMoE-1B-7B, DeepSeek-V2-Lite, and Qwen3-30B-A3B.
- **Key finding 1 (Launch-bound)**: Fused Triton kernels achieve 5.6x–9.0x speedup in isolation but only 0.999x end-to-end against a measured 1.072x Amdahl ceiling, because models spend time waiting on ~1000 kernel launches per forward pass, not on arithmetic.
- **Key finding 2 (Substitutable experts)**: INT4 quantization changes on average 0.53 of 8 selected experts per token, but a causal route-replay intervention shows routing changes account for only 2.7% of quality loss—97.3% comes from weight error.
- **Key finding 3 (Graph breaks are not a performance metric)**: Eliminating all 23 torch.compile graph breaks makes the model 3× slower, not faster.
- **Key finding 4 (Separable objectives)**: Leaving routers in FP16 lowers routing drift by 20% but raises loss, proving routing fidelity and output quality are separable objectives that can move in opposite directions.

---

## Introduction and Theoretical Foundation

### Background
Mixture-of-Experts (MoE) models decouple model capacity from per-token compute cost via a learned gating network that selects a few expert feed-forward networks per token. This enables models like OLMoE, DeepSeek-V2, and Qwen3 to hold billions of parameters while activating only a fraction per forward pass.

### The Core Problem
MoE inference relies on **dynamic, data-dependent routing**, which violates assumptions underlying standard optimizations:
- **Custom GPU kernels** assume fixed tensor shapes
- **Quantization** assumes small weight perturbations produce small output changes
- **Graph compilers** (torch.compile) work best when the traced graph doesn't change between calls

The paper's central question: *What is the per-device cost of a forward pass actually made of?* Prior work (MoE-squared, Liu et al., SP-MoE) schedules around per-device costs; this paper asks whether those costs are correctly attributed.

### Theoretical Foundation
- **Amdahl's Law** provides the theoretical ceiling for kernel optimizations:
$$S = \frac{1}{(1 - f) + f/s} \tag{1}$$
where $f$ is the fraction of the forward pass spent in optimized operations and $s$ their isolated speedup.
- **Routing drift metrics** quantify how quantization perturbs the discrete top-k selection function. A perturbation smaller than the smallest logit margin changes nothing; one larger changes the expert set entirely.

---

## Methodology

### Models Under Study
Three models with deliberately different routing configurations:

| Property | OLMoE-1B-7B | DeepSeek-V2-Lite | Qwen3-30B-A3B |
|---|---|---|---|
| Total parameters | 7B | 16B | 30B |
| Active per token | 1B | 2.4B | 3B |
| Routed experts | 64 | 64 + 2 shared | 128 |
| top-k | 8 | 6 | 8 |
| MoE layers | 16 | 26 | 48 |
| Selection density | 12.5% | 9.4% | 6.3% |
| Gate module type | nn.Linear | nn.Parameter | nn.Linear |
| norm_topk_prob | false | false | true |

### Sub-study 1: Triton Kernel Engineering
- **FusedRMSNorm**: Merges variance computation and normalization into a single SRAM-resident pass (block size 1024 for hidden ≤ 2048, 2048 above)
- **FusedSoftmax**: Collapses four dispatches into one tile using online max-subtract-expsum formulation; gate probabilities accumulated in FP32 before top-k
- Kernels installed via monkey-patching (class-name matching for RMSNorm, forward wrapping for router softmax)
- **Amdahl ceiling** computed via profiler `record_function` ranges over 81 wrapped modules, giving $f = 7.87\%$ and ceiling of 1.072x

### Sub-study 2: Routing Drift and Causal Intervention
- Forward hooks on every gate module record selected expert indices per layer and token position
- 100 MMLU-style prompts → 119,952 token positions per configuration
- **Drift metrics**:

| Metric | Definition | Interpretation |
|---|---|---|
| Routing Similarity | Tokens with identical top-k set | 1.0 = perfect agreement |
| Jaccard Drift | $1 - \|A \cap B\|/\|A \cup B\|$ | 0.0 = no drift |
| Overlap@k | Mean $\|A \cap B\|/k$ | 1.0 = same experts |
| Selection Shift | Fraction of slots changed | 0.0 = no change |
| Swaps per token | $k \times$ Selection Shift | Comparable across k |

- **Causal route-replay intervention**: Record quantized model's expert selections, then run FP16 model with routing overridden to follow those selections while weights stay full-precision. Control replays FP16's own routes (must return baseline exactly).

### Sub-study 3: Graph Breaks and Compilation
- Graph breaks counted via `torch._dynamo.explain` on the real 16-layer OLMoE checkpoint
- Five configurations timed at identical shapes: eager, eager+Triton kernels, compile default, compile with `capture_dynamic_output_shape_ops`, and that + kernels

---

## Empirical Validation / Results

### Triton Kernel Results

**Isolated RMSNorm benchmark (A100):**

| Hidden | Baseline | Triton | Speedup | GB/s |
|---|---|---|---|---|
| 512 | 0.0581 ms | 0.0103 ms | 5.62x | 609 |
| 1024 | 0.0755 ms | 0.0132 ms | 5.72x | 953 |
| 2048 | 0.1365 ms | 0.0188 ms | 7.27x | 1342 |
| 4096 | 0.2789 ms | 0.0311 ms | 8.98x | 1620 |

**Forward-pass share and Amdahl ceiling:**

| Quantity | Value |
|---|---|
| RMSNorm share | 7.70% |
| Router softmax share | 0.17% |
| Combined f | 7.87% |
| Effective kernel speedup | 6.88x |
| Amdahl ceiling | 1.072x |

**End-to-end OLMoE latency:**

| seq | batch | Baseline | Kernels | Speedup |
|---|---|---|---|---|
| 128 | 1 | 246.8 ms | 314.5 ms | 0.785x |
| 512 | 4 | 342.5 ms | 342.7 ms | 0.999x |
| 1024 | 4 | 383.4 ms | 373.4 ms | 1.027x |

**Key evidence**: Going from 128 to 4096 tokens (32× work) raises latency only 1.55× (246.8 → 383.4 ms). The model is **launch-bound**, not arithmetic-bound.

### Routing Drift Results

**OLMoE at top-8, 119,952 token positions:**

| Precision | Jaccard drift | 95% CI | Sel. shift | dNLL |
|---|---|---|---|---|
| INT8 | 0.0488 | [0.0477, 0.0501] | 0.0277 | +0.0031 |
| INT4 | 0.1142 | [0.1123, 0.1163] | 0.0660 | +0.0866 |

**Causal route replay:**

| Configuration | NLL |
|---|---|
| FP16 baseline | 2.276696 |
| FP16 weights, FP16 routes (control) | 2.276696 |
| FP16 weights, INT4 routes | 2.279028 |
| INT4 | 2.363338 |

**Attribution**: Routing accounts for $+0.002332$ of $+0.086642$ total NLL increase = **2.7%**. Weight error accounts for 97.3%.

**Router exemption (quantize everything except 16 routers):**

| Configuration | Routers | Jaccard drift | dNLL |
|---|---|---|---|
| nf4 | quantized | 0.1140 | +0.0872 |
| nf4_gate_fp16 | FP16 | 0.0910 | +0.0979 |

Drift falls 20% but quality gets worse—**routing fidelity and output quality move in opposite directions**.

**Cross-architecture drift (top-k corrected):**

| Model | k | Prec | Jaccard | Swaps/token | dNLL |
|---|---|---|---|---|---|
| DeepSeek-V2-Lite | 6 | INT8 | 0.0419 | 0.1475 | +0.00118 |
| OLMoE-1B-7B | 8 | INT8 | 0.0488 | 0.2214 | +0.00312 |
| Qwen3-30B-A3B | 8 | INT8 | 0.0690 | 0.3171 | +0.00512 |
| DeepSeek-V2-Lite | 6 | INT4 | 0.1303 | 0.4686 | +0.02524 |
| OLMoE-1B-7B | 8 | INT4 | 0.1142 | 0.5281 | +0.08664 |
| Qwen3-30B-A3B | 8 | INT4 | 0.1657 | 0.7961 | +0.05810 |

The top-k correction **reverses** the apparent INT4 ordering between DeepSeek and OLMoE.

### torch.compile Results

| Configuration | 512x4 | 1024x4 | Breaks | First fwd |
|---|---|---|---|---|
| eager | 1.000x | 1.000x | 0 | 1.0 s |
| eager + kernels | 0.979x | 1.033x | 0 | 2.8 s |
| compile, default | 0.822x | 0.878x | 19/36 | 30–61 s |
| compile + capture | 0.613x | 0.327x | 0 | 37–40 min |

**Zero graph breaks is 3× slower than eager.** Unbacked symbolic shapes force Inductor to generate size-agnostic code with expensive guards and fallbacks.

---

## Theoretical and Practical Implications

### Theoretical Implications
1. **Amdahl's Law is insufficient**: A ceiling of 1.072x with achieved 0.999x at the same shape reveals the model is *integration-bound*, not Amdahl-bound. The gap is explained by launch overhead, not arithmetic.
2. **Correlation ≠ mechanism**: Drift correlates with quality at Pearson +0.907, but is 98% collinear with gate KL divergence. The causal intervention is required to separate them—correlation alone cannot.
3. **Drift is a symptom, not a cause**: Routing changes track the damage without causing it. Experts are *substitutable*—sending a token to its 9th-best expert instead of 8th-best costs very little.
4. **Top-k correction is essential for cross-model comparison**: Raw Jaccard drift inflates models with smaller k for the same physical event. Swaps per token ($k \times$ selection shift) is the comparable quantity.

### Practical Implications
1. **Kernel optimization is misdirected**: The binding constraint is the launch structure of expert dispatch (a Python loop over 64 experts × 16 layers ≈ 1000 launches). The optimization that matters is batching dispatch into a single grouped matrix multiplication.
2. **Graph-break counts are a traceability diagnostic, not a performance target**: Reducing breaks to zero actively harms performance by 3×.
3. **Quantization effort should focus on weight error, not routing preservation**: 97.3% of INT4's quality loss comes from weight error. Exempting routers from quantization is actively harmful.
4. **Deployment scheduling inherits misattributed costs**: If single-device costs are misattributed, every schedule built on top inherits the error.

---

## Conclusion

### Main Takeaways
1. **The model is launch-bound**: 32× tokens cost 1.55× time; the device waits on ~1000 kernel launches, not arithmetic.
2. **The experts are substitutable**: INT4 changes 0.53 of 8 experts per token, but replaying those changes through full-precision weights reproduces only 2.7% of the loss.
3. **Graph-break elimination is counterproductive**: Zero breaks = 3× slower than eager.
4. **Routing fidelity and output quality are separable objectives** that can trade against each other in the wrong direction.

### Limitations
- Causal replay run on OLMoE alone (single-model result)
- Correlation rests on 16 configurations from one checkpoint/prompt set
- Cross-model ordering rests on 3 points with capacity uncontrolled and DeepSeek's gate unquantized
- All models quantized by a single library (bitsandbytes)

### Future Work
1. **Hand-quantize DeepSeek's nn.Parameter gate** to confirm/eliminate the confound behind its apparent advantage
2. **Batch the expert dispatch** into a single grouped matrix multiplication to attack the launch-bound structure
3. **Test quantization objectives penalizing routing flips** against the router-exemption result, since lowering drift can raise loss

---

## Code Availability

The complete source code, Modal execution harness, and raw per-token route dumps are open source at: https://github.com/GokuHashira/moe-ceilings

---

_Markdown view of https://picx.dev/p/nJ5NaI, served by PicX — AI-generated visual whiteboard summaries of research papers._