Summary (Overview)

• This paper empirically evaluates three standard inference optimizations—fused Triton kernels, INT4/INT8 quantization, and torch.compile—on three Mixture-of-Experts (MoE) models: OLMoE-1B-7B, DeepSeek-V2-Lite, and Qwen3-30B-A3B.
• Key finding 1 (Launch-bound): Fused Triton kernels achieve 5.6x–9.0x speedup in isolation but only 0.999x end-to-end against a measured 1.072x Amdahl ceiling, because models spend time waiting on ~1000 kernel launches per forward pass, not on arithmetic.
• Key finding 2 (Substitutable experts): INT4 quantization changes on average 0.53 of 8 selected experts per token, but a causal route-replay intervention shows routing changes account for only 2.7% of quality loss—97.3% comes from weight error.
• Key finding 3 (Graph breaks are not a performance metric): Eliminating all 23 torch.compile graph breaks makes the model 3× slower, not faster.
• Key finding 4 (Separable objectives): Leaving routers in FP16 lowers routing drift by 20% but raises loss, proving routing fidelity and output quality are separable objectives that can move in opposite directions.

---

Introduction and Theoretical Foundation

Background

Mixture-of-Experts (MoE) models decouple model capacity from per-token compute cost via a learned gating network that selects a few expert feed-forward networks per token. This enables models like OLMoE, DeepSeek-V2, and Qwen3 to hold billions of parameters while activating only a fraction per forward pass.

The Core Problem

MoE inference relies on dynamic, data-dependent routing, which violates assumptions underlying standard optimizations:

• Custom GPU kernels assume fixed tensor shapes
• Quantization assumes small weight perturbations produce small output changes
• Graph compilers (torch.compile) work best when the traced graph doesn't change between calls

The paper's central question: What is the per-device cost of a forward pass actually made of? Prior work (MoE-squared, Liu et al., SP-MoE) schedules around per-device costs; this paper asks whether those costs are correctly attributed.

Theoretical Foundation

• Amdahl's Law provides the theoretical ceiling for kernel optimizations:

$$S = \frac{1}{(1 - f) + f/s} \tag{1}$$

where $f$ is the fraction of the forward pass spent in optimized operations and $s$ their isolated speedup.

• Routing drift metrics quantify how quantization perturbs the discrete top-k selection function. A perturbation smaller than the smallest logit margin changes nothing; one larger changes the expert set entirely.

---

Methodology

Models Under Study

Three models with deliberately different routing configurations:

Sub-study 1: Triton Kernel Engineering

• FusedRMSNorm: Merges variance computation and normalization into a single SRAM-resident pass (block size 1024 for hidden ≤ 2048, 2048 above)
• FusedSoftmax: Collapses four dispatches into one tile using online max-subtract-expsum formulation; gate probabilities accumulated in FP32 before top-k
• Kernels installed via monkey-patching (class-name matching for RMSNorm, forward wrapping for router softmax)
• Amdahl ceiling computed via profiler record_function ranges over 81 wrapped modules, giving $f = 7.87\%$ and ceiling of 1.072x

Sub-study 2: Routing Drift and Causal Intervention

• Forward hooks on every gate module record selected expert indices per layer and token position
• 100 MMLU-style prompts → 119,952 token positions per configuration
• Drift metrics:

• Causal route-replay intervention: Record quantized model's expert selections, then run FP16 model with routing overridden to follow those selections while weights stay full-precision. Control replays FP16's own routes (must return baseline exactly).

Sub-study 3: Graph Breaks and Compilation

• Graph breaks counted via torch._dynamo.explain on the real 16-layer OLMoE checkpoint
• Five configurations timed at identical shapes: eager, eager+Triton kernels, compile default, compile with capture_dynamic_output_shape_ops, and that + kernels

---

Empirical Validation / Results

Triton Kernel Results

Isolated RMSNorm benchmark (A100):

Forward-pass share and Amdahl ceiling:

End-to-end OLMoE latency:

Key evidence: Going from 128 to 4096 tokens (32× work) raises latency only 1.55× (246.8 → 383.4 ms). The model is launch-bound, not arithmetic-bound.

Routing Drift Results

OLMoE at top-8, 119,952 token positions:

Causal route replay:

Attribution: Routing accounts for $+0.002332$ of $+0.086642$ total NLL increase = 2.7%. Weight error accounts for 97.3%.

Router exemption (quantize everything except 16 routers):

Drift falls 20% but quality gets worse—routing fidelity and output quality move in opposite directions.

Cross-architecture drift (top-k corrected):

The top-k correction reverses the apparent INT4 ordering between DeepSeek and OLMoE.

torch.compile Results

Zero graph breaks is 3× slower than eager. Unbacked symbolic shapes force Inductor to generate size-agnostic code with expensive guards and fallbacks.

---

Theoretical and Practical Implications

Theoretical Implications

1. Amdahl's Law is insufficient: A ceiling of 1.072x with achieved 0.999x at the same shape reveals the model is integration-bound, not Amdahl-bound. The gap is explained by launch overhead, not arithmetic.
2. Correlation ≠ mechanism: Drift correlates with quality at Pearson +0.907, but is 98% collinear with gate KL divergence. The causal intervention is required to separate them—correlation alone cannot.
3. Drift is a symptom, not a cause: Routing changes track the damage without causing it. Experts are substitutable—sending a token to its 9th-best expert instead of 8th-best costs very little.
4. Top-k correction is essential for cross-model comparison: Raw Jaccard drift inflates models with smaller k for the same physical event. Swaps per token ($k \times$ selection shift) is the comparable quantity.

Practical Implications

1. Kernel optimization is misdirected: The binding constraint is the launch structure of expert dispatch (a Python loop over 64 experts × 16 layers ≈ 1000 launches). The optimization that matters is batching dispatch into a single grouped matrix multiplication.
2. Graph-break counts are a traceability diagnostic, not a performance target: Reducing breaks to zero actively harms performance by 3×.
3. Quantization effort should focus on weight error, not routing preservation: 97.3% of INT4's quality loss comes from weight error. Exempting routers from quantization is actively harmful.
4. Deployment scheduling inherits misattributed costs: If single-device costs are misattributed, every schedule built on top inherits the error.

---

Conclusion

Main Takeaways

1. The model is launch-bound: 32× tokens cost 1.55× time; the device waits on ~1000 kernel launches, not arithmetic.
2. The experts are substitutable: INT4 changes 0.53 of 8 experts per token, but replaying those changes through full-precision weights reproduces only 2.7% of the loss.
3. Graph-break elimination is counterproductive: Zero breaks = 3× slower than eager.
4. Routing fidelity and output quality are separable objectives that can trade against each other in the wrong direction.

Limitations

• Causal replay run on OLMoE alone (single-model result)
• Correlation rests on 16 configurations from one checkpoint/prompt set
• Cross-model ordering rests on 3 points with capacity uncontrolled and DeepSeek's gate unquantized
• All models quantized by a single library (bitsandbytes)

Future Work

1. Hand-quantize DeepSeek's nn.Parameter gate to confirm/eliminate the confound behind its apparent advantage
2. Batch the expert dispatch into a single grouped matrix multiplication to attack the launch-bound structure
3. Test quantization objectives penalizing routing flips against the router-exemption result, since lowering drift can raise loss

---

Code Availability

The complete source code, Modal execution harness, and raw per-token route dumps are open source at: https://github.com/GokuHashira/moe-ceilings