Task-Phase-Conditioned Expert Routing Is Not a Reliable Proxy for Quantization Sensitivity in MoE FFN Tensors

Author: Yifan Liu (UIUC MCS Student)

---

Summary (Overview)

• Central Question: Can per-layer, per-task-phase (prompt/reasoning/response) expert-routing statistics in a Mixture-of-Experts (MoE) language model guide layer-wise mixed-precision quantization of routed-expert FFN tensors?
• Key Finding: Budget-neutral reallocation based on routing-behavior statistics fails to outperform the uniform baseline in both quality regimes tested (~4.0 bpw flat and ~2.7 bpw steep).
• Critical Result: In the aggressive low-bit regime, the loss is decisive (z > 4 SE), and a random-selection control performs within ~1 SE of the routing-guided reallocation, demonstrating that the composite routing score carries no selection signal over random.
• Primary Contribution: A comprehensive routing-capture and profiling pipeline for MoE models in llama.cpp, applicable to standard top-k MoE architectures, plus a rigorous negative result challenging the intuitive link between routing behavior and quantization sensitivity.
• Core Conclusion: Routing-behavior statistics are an unvalidated proxy for true per-layer quantization cost; the failure is structural (mandatory downgrades causing KLD tail injection), not a flaw in the heuristic.

---

Introduction and Theoretical Foundation

Background and Motivation

As MoE models grow increasingly large, users frequently turn to aggressive quantization to run these models locally. This creates a need for targeted quantization strategies that achieve better Kullback-Leibler Divergence (KLD) and Perplexity (PPL) values. The paper tests a natural hypothesis: that MoE layers with more distinctive phase-dependent or concentrated routing behavior should receive more quantization precision, while less distinctive layers should receive more aggressive quantization.

Task Phases

The authors characterize three distinct task phases:

• Prompt (prefill region)
• Chain of Thought (CoT) reasoning
• Response (post-reasoning generation)

Prior Work

Key insight from prior work: MoPEQ found that expert activation frequency does not predict sensitivity. The authors build on this by testing whether a phase-conditioned routing-behavior signal can serve as a lightweight proxy for direct measurements (Hessian trace, per-expert ablation).

Theoretical Foundation

The allocation problem is framed as a classical rate-distortion (water-filling) problem [Cover and Thomas, 2006; Shoham and Gersho, 1988]. Optimal bit allocation equalizes marginal distortion-reduction per bit across all dimensions. However, the routing-behavior signals used in the composite score are an indirect proxy for the true distortion-rate curve for a layer, rather than a direct measurement.

---

Methodology

3.1 Model and Setup

• Model: Qwen-3.6-35B-A3B (hybrid linear-attention MoE)
• Architecture: 40 layers + separate MTP head (blk.40); dynamic allocation applies to blk.0–38, with terminal layer blk.39 held at baseline
• Engine: Modified ik llama.cpp with imatrix calibration

3.2 Data Pipeline

Profiling prompts: 30 stratified-sampled coding prompts from:

• HumanEval-X (Python, C++, Java, JS, Go)
• HumanEval-Rust
• LiveCodeBench Lite

Key bug fixes in the pipeline:

1. Added per-request IDs to separate prompts in captured data
2. Disabled graph reuse to prevent silent routing-capture data drops
3. Set batch and ubatch sizes larger than maximum prompt token count

Surrogate routing capture: Model first quantized to ~4 bpw per tensor with Q8_0 precision router and attention/SSM tensors to ensure high-fidelity routing.

3.3 Segmentation

• Prompt: Prefill region (defined by request's prompt token length)
• Reasoning: From end of prompt to first `` token
• Response: Remainder of generation

3.4 Composite Sensitivity Score

The composite score combines four signals per layer:

$$d_l = 1 - H_l, \qquad c_l = \mathrm{CV}_l, \qquad f_l = 1 - \min(J_l^{rr}, J_l^{pr}), \qquad t_l = \tau_l$$ $$z_l^{(s)}(x) = \frac{x_l - \bar{x}^{(s)}}{\sigma^{(s)} + \epsilon} \tag{1}$$ $$S_l^{(s)} = w_d z_l^{(s)}(d_l) + w_c z_l^{(s)}(c_l) + w_p z_l^{(s)}(f_l) + w_t z_l^{(s)}(t_l)$$

Where:

• $H_l$ = normalized routing entropy (decisiveness)
• $\mathrm{CV}_l$ = coefficient of variation of expert usage (concentration)
• $J_l^{rr}$, $J_l^{pr}$ = cross-phase pairwise Jaccard overlaps (phase-fragility)
• $\tau_l$ = top-1 expert fraction (confidence)

Key property: Since the downstream allocation consumes only rank(S), the composite score is invariant to affine rescaling. This allows fixing $w_d \equiv 1$, reducing the search to three free weight ratios $\{w_c, w_p, w_t\}$.

3.5 Allocation Mechanism

Budget regimes:

Budget-fit algorithm:

1. Rank layers by composite-score percentile
2. Split into fragile (low-ranked) and robust (high-ranked) pools
3. Downgrade fragile pool to build byte surplus
4. Fund upgrades of robust pool one rung at a time
5. Enforce budget-neutrality via pre-quantize dry-run and post-quantize byte checks

3.6 Coordinate Descent Search

• Objective: Minimize KLD
• Search space: 3 free weight ratios (with $w_d$ anchored to 1)
• Two rounds: Coarse (multipliers {0.5, 1.5, 3.0}) then adaptive ({0.75, 1.0, 1.25} for interior winners)
• Early stopping: Full pass with no KLD improvement
• Caching: SHA-1 hash of canonicalized tier maps prevents redundant quantizations

3.7 Evaluation Methodology

• Reference: Q8_0 quantization as proxy for BF16 (validated by prior work showing 8-bit tracks BF16 closely)
• Metrics: Mean KLD, PPL, top-1 token agreement, KLD tail (95th/99th percentiles, median, max)
• Evaluation corpus: ~331K tokens from The Stack v3 (21 languages, held out from profiling data)

3.8 Validation-Set Bias Warning

A frozen 12K-token slice proved unreliable: uniform baseline KLD was 0.019325 on the slice but 0.014725 on the full corpus (~24% lower). Relative rankings did not transfer from slice to full corpus, so all final results use the full corpus.

---

Empirical Validation / Results

4.1 Flat Regime (~4 bpw)

All 8 coordinate-descent candidates failed to beat the uniform baseline.

Best candidate is statistically neutral (z ≈ +0.94 SE). At ~4 bpw, every layer is near its per-layer optimum, so reallocation nets to roughly zero change.

4.2 Steep Regime (~2.7 bpw)

Decisive loss across all candidates (z ≈ 4.3 SE combined).

The loss is tail-dominated: 99th-percentile KLD rises from 0.437 to 0.487–0.492, while median KLD stays within 0.0019 of baseline. Mean KLD (0.034) is ~20× the median, confirming heavy-tailed distribution.

4.3 Random-Selection Control

Critical finding: Random layer selection performs within ~1 SE of (and marginally better than) score-guided allocations.

This demonstrates the composite score adds no selection signal over random layer selection.

---

Theoretical and Practical Implications

Why Budget-Neutral Reallocation Loses

The fundamental issue is the zero-sum structure of budget-neutral reallocation:

$$\Delta \mathrm{KLD} = \sum_l g_l - \sum_l c_l$$

Where $g_l$ and $c_l$ are per-layer marginal gains and costs. In the steep regime:

• Upgrading 17 layers by one rung (+1.25 bpw/layer) required downgrading 19 layers to IQ2 floor tier (−1.12 bpw/layer)
• Gains and costs cancel almost exactly

Water-filling theory states optimal allocation equalizes marginal distortion-reduction per bit. The routing-behavior proxy is not a direct measurement of quantization distortion—there was never a theoretical basis for the expectation that phase-distinctive routing correlates with quantization cost.

The Floor-Tier Cliff Mechanism

The loss is driven by tail KLD injection:

• 19 of 39 profiled layers receive mandatory IQ2 downgrades
• 99th-percentile KLD rises from 0.437 to 0.483–0.495 on every reallocated candidate
• Median moves only from 0.0016 to at most 0.0019
• The tail injection is a property of the IQ2 tier itself, not of specific layers

Key implication: Below the elbow of the Pareto curve, reallocation within a fixed budget cannot help. The only mitigation is raising the floor globally (more budget).

Implications for Quantization Research

1. Routing behavior ≠ quantization sensitivity. The intuitive connection presumed to exist between routing behavior and layer-wise quantization sensitivity does not hold empirically.
2. Direct sensitivity measurement is necessary. Hessian-based methods (MoPEQ) or per-layer ablation remain the validated approaches.
3. Budget-neutral reallocation has structural limits. At low bit-widths, mandatory downgrades to floor tiers inject catastrophic tail errors regardless of which layers are selected.

Limitations

• Single model: Qwen-3.6-35B-A3B (hybrid linear-attention MoE); cross-model transfer to GLM-5.3 (753B total) planned but not executed
• Surrogate routing capture: Possible routing perturbation from quantization; no BF16 comparison performed
• No direct-sensitivity ground truth: No single-layer ablation or Hessian correlation values
• Single domain: Code tasks only; no reasoning/roleplay/chat/trivia evaluation
• Small profiling set: 30 prompts; no validation of optimal sample count

---

Conclusion

Task-phase-conditioned routing statistics, despite a carefully instrumented pipeline spanning routing capture, composite scoring, and budget-neutral coordinate descent, do not translate into a useful equal-budget allocation criterion for layer-wise mixed-precision quantization of routed-expert FFN tensors.

Key takeaways:

1. Score-guided allocations are neutral at ~4 bpw and decisively harmful at ~2.7 bpw
2. The failure mechanism is structural: mandatory downgrades cause KLD tail injection
3. The random-selection control proves the heuristic itself is not at fault—routing statistics carry no selection signal over random
4. The intuitive connection between routing behavior and quantization sensitivity does not hold

Future directions:

• Explore allocation at higher granularity (individual experts within a layer, rather than entire layers)
• Validate candidate signals against directly measured layer sensitivity (Hessian trace, per-layer ablation)
• Consider raising the global floor tier rather than budget-neutral reallocation in low-bit regimes