# Task-Phase-Conditioned Expert Routing Is Not a Reliable Proxy for Quantization Sensitivity in MoE FFN Tensors

> Routing-based expert allocation fails to beat uniform quantization and performs no better than random selection, proving routing behavior is not a valid proxy for quantization sensitivity.

- **Source:** [arXiv](https://arxiv.org/abs/2610.20826)
- **Published:** 2026-10-24
- **Permalink:** https://picx.dev/p/4JgQMr
- **Whiteboard:** https://picx.dev/p/4JgQMr/image

## Summary

# Task-Phase-Conditioned Expert Routing Is Not a Reliable Proxy for Quantization Sensitivity in MoE FFN Tensors

**Author:** Yifan Liu (UIUC MCS Student)

---

## Summary (Overview)

- **Central Question:** Can per-layer, per-task-phase (prompt/reasoning/response) expert-routing statistics in a Mixture-of-Experts (MoE) language model guide layer-wise mixed-precision quantization of routed-expert FFN tensors?
- **Key Finding:** Budget-neutral reallocation based on routing-behavior statistics **fails to outperform** the uniform baseline in both quality regimes tested (~4.0 bpw flat and ~2.7 bpw steep).
- **Critical Result:** In the aggressive low-bit regime, the loss is decisive (z > 4 SE), and a **random-selection control** performs within ~1 SE of the routing-guided reallocation, demonstrating that the composite routing score carries **no selection signal over random**.
- **Primary Contribution:** A comprehensive routing-capture and profiling pipeline for MoE models in llama.cpp, applicable to standard top-k MoE architectures, plus a rigorous negative result challenging the intuitive link between routing behavior and quantization sensitivity.
- **Core Conclusion:** Routing-behavior statistics are an **unvalidated proxy** for true per-layer quantization cost; the failure is structural (mandatory downgrades causing KLD tail injection), not a flaw in the heuristic.

---

## Introduction and Theoretical Foundation

### Background and Motivation

As MoE models grow increasingly large, users frequently turn to aggressive quantization to run these models locally. This creates a need for targeted quantization strategies that achieve better Kullback-Leibler Divergence (KLD) and Perplexity (PPL) values. The paper tests a natural hypothesis: that MoE layers with more distinctive phase-dependent or concentrated routing behavior should receive more quantization precision, while less distinctive layers should receive more aggressive quantization.

### Task Phases

The authors characterize three distinct task phases:
- **Prompt** (prefill region)
- **Chain of Thought (CoT) reasoning**
- **Response** (post-reasoning generation)

### Prior Work

| Method | Signal Used | Phase-Aware? |
|--------|------------|--------------|
| MxMoE | Linear-block sensitivity + expert activation frequency | No |
| GEMQ | Global expert-level importance ranking | No |
| MoPEQ | Hessian trace approximation | No |
| EAQuant | Expert-aware activation smoothing | No |
| MoEQuant | Affinity-guided quantization | No |

**Key insight from prior work:** MoPEQ found that *expert activation frequency does not predict sensitivity*. The authors build on this by testing whether a phase-conditioned routing-behavior signal can serve as a lightweight proxy for direct measurements (Hessian trace, per-expert ablation).

### Theoretical Foundation

The allocation problem is framed as a **classical rate-distortion (water-filling)** problem [Cover and Thomas, 2006; Shoham and Gersho, 1988]. Optimal bit allocation equalizes marginal distortion-reduction per bit across all dimensions. However, the routing-behavior signals used in the composite score are an **indirect proxy** for the true distortion-rate curve for a layer, rather than a direct measurement.

---

## Methodology

### 3.1 Model and Setup

- **Model:** Qwen-3.6-35B-A3B (hybrid linear-attention MoE)
- **Architecture:** 40 layers + separate MTP head (blk.40); dynamic allocation applies to blk.0–38, with terminal layer blk.39 held at baseline
- **Engine:** Modified ik llama.cpp with imatrix calibration

### 3.2 Data Pipeline

**Profiling prompts:** 30 stratified-sampled coding prompts from:
- HumanEval-X (Python, C++, Java, JS, Go)
- HumanEval-Rust
- LiveCodeBench Lite

**Key bug fixes in the pipeline:**
1. Added per-request IDs to separate prompts in captured data
2. Disabled graph reuse to prevent silent routing-capture data drops
3. Set batch and ubatch sizes larger than maximum prompt token count

**Surrogate routing capture:** Model first quantized to ~4 bpw per tensor with Q8_0 precision router and attention/SSM tensors to ensure high-fidelity routing.

### 3.3 Segmentation

- **Prompt:** Prefill region (defined by request's prompt token length)
- **Reasoning:** From end of prompt to first `` token
- **Response:** Remainder of generation

### 3.4 Composite Sensitivity Score

The composite score combines four signals per layer:

$$d_l = 1 - H_l, \qquad c_l = \mathrm{CV}_l, \qquad f_l = 1 - \min(J_l^{rr}, J_l^{pr}), \qquad t_l = \tau_l$$

$$z_l^{(s)}(x) = \frac{x_l - \bar{x}^{(s)}}{\sigma^{(s)} + \epsilon} \tag{1}$$

$$S_l^{(s)} = w_d z_l^{(s)}(d_l) + w_c z_l^{(s)}(c_l) + w_p z_l^{(s)}(f_l) + w_t z_l^{(s)}(t_l)$$

Where:
- $H_l$ = normalized routing entropy (decisiveness)
- $\mathrm{CV}_l$ = coefficient of variation of expert usage (concentration)
- $J_l^{rr}$, $J_l^{pr}$ = cross-phase pairwise Jaccard overlaps (phase-fragility)
- $\tau_l$ = top-1 expert fraction (confidence)

**Key property:** Since the downstream allocation consumes only rank(S), the composite score is invariant to affine rescaling. This allows fixing $w_d \equiv 1$, reducing the search to three free weight ratios $\{w_c, w_p, w_t\}$.

### 3.5 Allocation Mechanism

**Budget regimes:**

| Regime | Ladder (rungs) | Down base | GU base | Combined bpw/layer |
|--------|---------------|-----------|---------|-------------------|
| Flat (~4 bpw) | iq3_ks·iq3_k·iq4_kss·iq4_ks·iq4_k | iq4_ks | iq3_k | 11.13 |
| Steep (~2.7 bpw) | iq2_ks·iq2_k·iq2_kl·iq3_ks·iq3_k | iq3_ks | iq2_kl | 8.57 |

**Budget-fit algorithm:**
1. Rank layers by composite-score percentile
2. Split into fragile (low-ranked) and robust (high-ranked) pools
3. Downgrade fragile pool to build byte surplus
4. Fund upgrades of robust pool one rung at a time
5. Enforce budget-neutrality via pre-quantize dry-run and post-quantize byte checks

### 3.6 Coordinate Descent Search

- **Objective:** Minimize KLD
- **Search space:** 3 free weight ratios (with $w_d$ anchored to 1)
- **Two rounds:** Coarse (multipliers {0.5, 1.5, 3.0}) then adaptive ({0.75, 1.0, 1.25} for interior winners)
- **Early stopping:** Full pass with no KLD improvement
- **Caching:** SHA-1 hash of canonicalized tier maps prevents redundant quantizations

### 3.7 Evaluation Methodology

- **Reference:** Q8_0 quantization as proxy for BF16 (validated by prior work showing 8-bit tracks BF16 closely)
- **Metrics:** Mean KLD, PPL, top-1 token agreement, KLD tail (95th/99th percentiles, median, max)
- **Evaluation corpus:** ~331K tokens from The Stack v3 (21 languages, held out from profiling data)

### 3.8 Validation-Set Bias Warning

A frozen 12K-token slice proved unreliable: uniform baseline KLD was 0.019325 on the slice but 0.014725 on the full corpus (~24% lower). **Relative rankings did not transfer** from slice to full corpus, so all final results use the full corpus.

---

## Empirical Validation / Results

### 4.1 Flat Regime (~4 bpw)

**All 8 coordinate-descent candidates failed to beat the uniform baseline.**

| Allocation | KLD (± SE) | Top-1 (%) |
|-----------|------------|-----------|
| Uniform (baseline) | 0.014725 ± 0.000430 | 97.5 |
| C1 (best) | 0.015276 ± 0.000400 | 97.4 |
| C8 (worst) | 0.015851 ± 0.000457 | 97.3 |

Best candidate is **statistically neutral** (z ≈ +0.94 SE). At ~4 bpw, every layer is near its per-layer optimum, so reallocation nets to roughly zero change.

### 4.2 Steep Regime (~2.7 bpw)

**Decisive loss across all candidates (z ≈ 4.3 SE combined).**

| Allocation | KLD (± SE) | KLD p99 | Top-1 (%) |
|-----------|------------|---------|-----------|
| Uniform U (baseline) | 0.034321 ± 0.000608 | 0.437 | 95.8 |
| cv 0.5 | 0.038249 ± 0.000659 | 0.490 | 95.6 |
| cv 1.5 | 0.038142 ± 0.000665 | 0.487 | 95.6 |
| cv 3.0 | 0.038223 ± 0.000662 | 0.492 | 95.6 |

The loss is **tail-dominated**: 99th-percentile KLD rises from 0.437 to 0.487–0.492, while median KLD stays within 0.0019 of baseline. Mean KLD (0.034) is ~20× the median, confirming heavy-tailed distribution.

### 4.3 Random-Selection Control

**Critical finding:** Random layer selection performs within ~1 SE of (and marginally better than) score-guided allocations.

| Allocation | KLD (± SE) | KLD p99 | Top-1 (%) |
|-----------|------------|---------|-----------|
| Uniform U (baseline) | 0.034321 ± 0.000608 | 0.437 | 95.8 |
| random, seed 42 | 0.037728 ± 0.000642 | 0.483 | 95.7 |
| random, seed 1337 | 0.037808 ± 0.000617 | 0.495 | 95.6 |
| ref: score-guided (cv 0.5–3.0) | 0.03814–0.03825 | 0.487–0.492 | 95.6 |

**This demonstrates the composite score adds no selection signal over random layer selection.**

---

## Theoretical and Practical Implications

### Why Budget-Neutral Reallocation Loses

The fundamental issue is the **zero-sum structure** of budget-neutral reallocation:

$$\Delta \mathrm{KLD} = \sum_l g_l - \sum_l c_l$$

Where $g_l$ and $c_l$ are per-layer marginal gains and costs. In the steep regime:
- Upgrading 17 layers by one rung (+1.25 bpw/layer) required downgrading 19 layers to IQ2 floor tier (−1.12 bpw/layer)
- Gains and costs cancel almost exactly

**Water-filling theory** states optimal allocation equalizes marginal distortion-reduction per bit. The routing-behavior proxy is not a direct measurement of quantization distortion—there was never a theoretical basis for the expectation that phase-distinctive routing correlates with quantization cost.

### The Floor-Tier Cliff Mechanism

The loss is driven by **tail KLD injection**:
- 19 of 39 profiled layers receive mandatory IQ2 downgrades
- 99th-percentile KLD rises from 0.437 to 0.483–0.495 on every reallocated candidate
- Median moves only from 0.0016 to at most 0.0019
- The tail injection is a property of the IQ2 tier itself, not of specific layers

**Key implication:** Below the elbow of the Pareto curve, reallocation within a fixed budget cannot help. The only mitigation is raising the floor globally (more budget).

### Implications for Quantization Research

1. **Routing behavior ≠ quantization sensitivity.** The intuitive connection presumed to exist between routing behavior and layer-wise quantization sensitivity does not hold empirically.
2. **Direct sensitivity measurement is necessary.** Hessian-based methods (MoPEQ) or per-layer ablation remain the validated approaches.
3. **Budget-neutral reallocation has structural limits.** At low bit-widths, mandatory downgrades to floor tiers inject catastrophic tail errors regardless of which layers are selected.

### Limitations

- **Single model:** Qwen-3.6-35B-A3B (hybrid linear-attention MoE); cross-model transfer to GLM-5.3 (753B total) planned but not executed
- **Surrogate routing capture:** Possible routing perturbation from quantization; no BF16 comparison performed
- **No direct-sensitivity ground truth:** No single-layer ablation or Hessian correlation values
- **Single domain:** Code tasks only; no reasoning/roleplay/chat/trivia evaluation
- **Small profiling set:** 30 prompts; no validation of optimal sample count

---

## Conclusion

Task-phase-conditioned routing statistics, despite a carefully instrumented pipeline spanning routing capture, composite scoring, and budget-neutral coordinate descent, **do not translate into a useful equal-budget allocation criterion** for layer-wise mixed-precision quantization of routed-expert FFN tensors.

**Key takeaways:**
1. Score-guided allocations are **neutral at ~4 bpw** and **decisively harmful at ~2.7 bpw**
2. The failure mechanism is **structural**: mandatory downgrades cause KLD tail injection
3. The **random-selection control** proves the heuristic itself is not at fault—routing statistics carry no selection signal over random
4. The intuitive connection between routing behavior and quantization sensitivity **does not hold**

**Future directions:**
- Explore allocation at higher granularity (individual experts within a layer, rather than entire layers)
- Validate candidate signals against **directly measured layer sensitivity** (Hessian trace, per-layer ablation)
- Consider raising the global floor tier rather than budget-neutral reallocation in low-bit regimes

---

_Markdown view of https://picx.dev/p/4JgQMr, served by PicX — AI-generated visual whiteboard summaries of research papers._