# ACE: Adaptive Calibration-Free Expert Skipping for MoE-based LLMs

> ACE enables training-free, calibration-free expert skipping in MoE LLMs via dual-view scoring, yielding up to 2.25x prefill and 1.41x decoding speedups with superior quality.

- **Source:** [arXiv](https://arxiv.org/abs/2609.05228)
- **Published:** 2026-09-19
- **Permalink:** https://picx.dev/p/ThlU0A
- **Whiteboard:** https://picx.dev/p/ThlU0A/image

## Summary

# ACE: Adaptive Calibration-Free Expert Skipping for MoE-based LLMs

## Summary (Overview)

- **Problem**: Mixture-of-Experts (MoE) LLMs use fixed top-k routing, executing the same number of expert slots for every token despite significant variation in expert contributions, leading to redundant computation.
- **Proposed Solution**: ACE (Adaptive Calibration-Free Expert Skipping), a training-free, calibration-free, and checkpoint-preserving framework that dynamically skips low-contribution expert slots during inference.
- **Key Innovation**: Two complementary offline estimators—Global Spectral Proxy (GSP) for structural transformation capacity and Router-Conditioned Refinement (RCR) for directional specialization—combined with a conservative max-fusion rule.
- **Results**: Consistent outperformance across three MoE models (Qwen3-30B-A3B, Qwen3.6-35B-A3B, Gemma-4-26B-A4Bit) and eight benchmarks, with up to 2.25× prefill and 1.41× decoding speedups.
- **Key Advantage**: At 50% skipping on Qwen3.6-35B-A3B, ACE reduces WikiText-2 perplexity by 7.96% and improves average downstream accuracy by 4.15 percentage points over the strongest competing method.

## Introduction and Theoretical Foundation

### Background
Mixture-of-Experts architectures scale LLMs efficiently by replacing dense FFN sublayers with multiple experts, routing each token to only a small subset. Models like Qwen3.6-35B-A3B activate only ~3B of 35B parameters, while DeepSeek-V4-Pro uses 1.6T/49B total-to-active configurations.

### Key Challenges Identified

1. **Routing preference ≠ Expert contribution**: Router gates reflect relative assignment preference, not actual output contribution. Experts with similar gate values may produce substantially different responses.

2. **Structural coupling in SwiGLU experts**: Expert output arises from nonlinear interaction among gate, up, and down projections—cannot be characterized by a single matrix norm.

3. **Direction-dependent specialization**: An expert with weak global responses may exhibit strong behavior within local input regions favored by the router.

### Theoretical Foundation

For a SwiGLU expert with input $x = \bar{x}\Gamma_l$ (where $\bar{x}$ is the RMS-normalized input direction and $\Gamma_l$ is the diagonal RMSNorm scaling matrix), the forward transformation is:

$$
f_{l,e}(x) = \left(\mathrm{SiLU}\left(\bar{x}\Gamma_l W_{l,e}^{\mathrm{gate}}\right) \odot \bar{x}\Gamma_l W_{l,e}^{\mathrm{up}}\right) W_{l,e}^{\mathrm{down}}
$$

Since $|\mathrm{SiLU}(z)| \le |z|$, the output admits the upper bound:

$$
\|f_{l,e}(x)\|_2 \leq \|\bar{x}\|_2^2 \|\Gamma_l W_{l,e}^{\text{gate}}\|_2 \|\Gamma_l W_{l,e}^{\text{up}}\|_2 \|W_{l,e}^{\text{down}}\|_2
$$

This multiplicative coupling motivates the GSP design.

## Methodology

### 1. Global Spectral Proxy (GSP)

**Purpose**: Estimate each expert's global transformation capacity from frozen SwiGLU weights without calibration data.

**Score definition**: For the $i$-th routed expert slot in layer $l$ for token $t$:

$$
s_{l,t,i}^{\mathrm{GSP}} = g_{l,t,i} \cdot A_l^{\mathrm{GSP}}(e_i)
$$

where $g_{l,t,i}$ is the router gate and $A_l^{\mathrm{GSP}}(e_i)$ is a static amplification proxy.

**Two complementary factorization paths**:

$$
a_{l,e}^{\mathrm{up}} = \|\Gamma_l W_{l,e}^{\mathrm{up}}\|_F \cdot \|\Gamma_l W_{l,e}^{\mathrm{gate}} W_{l,e}^{\mathrm{down}}\|_F
$$

$$
a_{l,e}^{\mathrm{gate}} = \|\Gamma_l W_{l,e}^{\mathrm{gate}}\|_F \cdot \|\Gamma_l W_{l,e}^{\mathrm{up}} W_{l,e}^{\mathrm{down}}\|_F
$$

**Geometric mean aggregation**:

$$
\tilde{a}_{l,e}^{\mathrm{GSP}} = \sqrt{a_{l,e}^{\mathrm{up}} a_{l,e}^{\mathrm{gate}}}
$$

**Layer-wise normalization**:

$$
A_l^{\mathrm{GSP}}(e) = \frac{\tilde{a}_{l,e}^{\mathrm{GSP}}}{\frac{1}{N_l}\sum_{j=1}^{N_l} \tilde{a}_{l,j}^{\mathrm{GSP}} + \epsilon}
$$

### 2. Router-Conditioned Refinement (RCR)

**Purpose**: Evaluate expert response along router-preferred directions using centered router weights.

**Centered router weights** (invariant to shared translation):

$$
\tilde{w}_{l,e} = w_{l,e} - \frac{1}{N_l}\sum_{j=1}^{N_l} w_{l,j}
$$

**Directional prototype** (RMS-normalized):

$$
q_{l,e} = \frac{\tilde{w}_{l,e}}{\mathrm{RMS}(\tilde{w}_{l,e}) + \epsilon} \cdot \rho_l
$$

**Router-conditioned amplification**:

$$
\tilde{a}_{l,e}^{\mathrm{RCR}} = \frac{\|f_{l,e}(q_{l,e})\|_2}{\|q_{l,e}\|_2 + \epsilon}
$$

**Score combination with gate**:

$$
s_{l,t,i}^{\mathrm{RCR}} = g_{l,t,i} A_l^{\mathrm{RCR}}(e_i)
$$

### 3. Conservative Fusion

ACE combines both perspectives using a **maximum criterion**:

$$
c_{l,t,i}^{\mathrm{ACE}} = \max\left(p_{l,t,i}^{\mathrm{GSP}}, p_{l,t,i}^{\mathrm{RCR}}\right)
$$

A slot is skipped only if **both** views identify it as low-contribution:

$$
\mathcal{S}_{l,t}^{\mathrm{ACE}} = \mathcal{S}_{l,t}^{\mathrm{GSP}} \cap \mathcal{S}_{l,t}^{\mathrm{RCR}}
$$

The top-1 expert (highest original router gate) is always retained.

## Empirical Validation / Results

### Experimental Setup

- **Models**: Qwen3-30B-A3B-Instruct-2507, Qwen3.6-35B-A3B, Gemma-4-26B-A4Bit
- **Benchmarks**: WikiText-2 PPL + 7 downstream tasks (ARC-C, ARC-E, PIQA, MATH-500, GPQA-Diamond, HumanEval, LiveCodeBench)
- **Baselines**: Score, NAEE, MoDES, DiEP, AIMER, Top-P, SERE, XShare, ExpertSparsity

### Key Results on Qwen3-30B-A3B (Table 1 excerpt)

| Method | 40% W2↓ | 40% Acc.↑ | 50% W2↓ | 50% Acc.↑ | 60% W2↓ | 60% Acc.↑ |
|--------|---------|-----------|---------|-----------|---------|-----------|
| Score | 9.63 | 75.39 | 12.03 | 66.69 | 18.76 | 37.64 |
| AIMER | 8.24 | 77.17 | 8.86 | 73.40 | 10.89 | 60.88 |
| Top-P | 8.19 | 77.73 | 8.85 | 74.17 | 10.87 | 61.91 |
| **ACE** | **8.19** | **78.21** | **8.85** | **74.30** | **10.86** | **63.35** |

### Key Results on Qwen3.6-35B-A3B (Table 2 excerpt)

| Method | 50% W2↓ | 50% Acc.↑ | 60% W2↓ | 60% Acc.↑ |
|--------|---------|-----------|---------|-----------|
| MoDES | 9.42 | 71.42 | 10.87 | 69.04 |
| GSP | 8.87 | 75.01 | 10.48 | 70.54 |
| **ACE** | **8.67** | **75.57** | **9.98** | **70.70** |

### Component Ablation (Table 4)

| Skip | Method | PPL↓ | Avg.↑ | Drop |
|------|--------|------|-------|------|
| 50% | GSP | 8.99 | 74.10 | 6.57 |
| 50% | RCR | 11.62 | 67.01 | 13.66 |
| 50% | **ACE** | **8.85** | **74.30** | **6.37** |

### Fusion Strategy Ablation (Table 5)

| Fusion | 30% PPL↓ | 30% Avg.↑ | 50% PPL↓ | 50% Avg.↑ |
|--------|----------|-----------|----------|-----------|
| Min | 8.21 | 77.20 | 9.20 | 73.12 |
| Mean | 7.95 | 78.45 | 8.97 | 74.01 |
| **Max (ACE)** | **7.86** | **79.77** | **8.85** | **74.30** |

### Efficiency Results

**Threshold construction cost** (Table 6): Quantile mapping takes 1.8 minutes vs. 16.5 minutes for binary search (9.2× reduction) and 18.9 minutes for MoDES frontier search (10.5× reduction).

**Latency at 60% skipping** (Table 7, A100 GPU):

| Length | Batch | TTFT BF16 | TTFT ACE | TPOT BF16 | TPOT ACE |
|--------|-------|-----------|----------|-----------|----------|
| 1024 | 1 | 270.9 ms | 120.3 ms | 85.1 ms/tok | 63.0 ms/tok |
| 1024 | 4 | 344.9 ms | 182.6 ms | 127.3 ms/tok | 91.1 ms/tok |

Achieves **1.72–2.25× TTFT** and **1.31–1.41× TPOT** speedups.

## Theoretical and Practical Implications

### Theoretical Contributions

1. **Calibration-free contribution estimation**: GSP provides a principled upper-bound-derived proxy for SwiGLU expert transformation capacity, addressing the multiplicative coupling of gate/up/down projections.

2. **Directional refinement without data**: RCR leverages the mathematical property that softmax routing is invariant to shared translation of router weights, enabling data-free construction of expert-specific direction prototypes via exponential tilting under Gaussian approximation.

3. **Conservative dual-view fusion**: The max-criterion (equivalently, intersection of low-contribution sets) provides a theoretical safeguard against erroneously skipping direction-specialized experts.

### Practical Implications

- **Deployment efficiency**: All expert statistics computed offline; runtime requires only table lookups and lightweight scalar operations.
- **Checkpoint preservation**: No modification to pretrained router or expert parameters.
- **Threshold transferability**: Thresholds for the same realized budget remain closely clustered across workloads and can be transferred across datasets.
- **Versatility**: Effective across different MoE architectures (Qwen3, Qwen3.6, Gemma-4) and model scales.

## Conclusion

ACE introduces a training-free, calibration-free, and checkpoint-preserving framework for token-adaptive expert skipping in MoE-based LLMs. By combining GSP's global structural capacity estimation with RCR's router-conditioned directional refinement under a conservative max-fusion rule, ACE reliably identifies low-contribution expert slots without requiring calibration data or model modification.

**Key takeaways**:
- ACE consistently outperforms static and dynamic baselines, with advantages growing under aggressive skipping (40–60%)
- Achieves state-of-the-art quality-efficiency trade-offs: 2.25× prefill and 1.41× decoding speedups
- Requires only 1.8 minutes for threshold construction (9.2–10.5× faster than alternatives)

**Future directions**:
- Threshold transfer across workloads without matched calibration
- Repeated-run uncertainty analysis (current results are single deterministic runs)
- Distributed expert dispatch optimization
- Extension to broader MoE architectures and multimodal models

---

_Markdown view of https://picx.dev/p/ThlU0A, served by PicX — AI-generated visual whiteboard summaries of research papers._