ACE: Adaptive Calibration-Free Expert Skipping for MoE-based LLMs

Summary (Overview)

• Problem: Mixture-of-Experts (MoE) LLMs use fixed top-k routing, executing the same number of expert slots for every token despite significant variation in expert contributions, leading to redundant computation.
• Proposed Solution: ACE (Adaptive Calibration-Free Expert Skipping), a training-free, calibration-free, and checkpoint-preserving framework that dynamically skips low-contribution expert slots during inference.
• Key Innovation: Two complementary offline estimators—Global Spectral Proxy (GSP) for structural transformation capacity and Router-Conditioned Refinement (RCR) for directional specialization—combined with a conservative max-fusion rule.
• Results: Consistent outperformance across three MoE models (Qwen3-30B-A3B, Qwen3.6-35B-A3B, Gemma-4-26B-A4Bit) and eight benchmarks, with up to 2.25× prefill and 1.41× decoding speedups.
• Key Advantage: At 50% skipping on Qwen3.6-35B-A3B, ACE reduces WikiText-2 perplexity by 7.96% and improves average downstream accuracy by 4.15 percentage points over the strongest competing method.

Introduction and Theoretical Foundation

Background

Mixture-of-Experts architectures scale LLMs efficiently by replacing dense FFN sublayers with multiple experts, routing each token to only a small subset. Models like Qwen3.6-35B-A3B activate only ~3B of 35B parameters, while DeepSeek-V4-Pro uses 1.6T/49B total-to-active configurations.

Key Challenges Identified

1. Routing preference ≠ Expert contribution: Router gates reflect relative assignment preference, not actual output contribution. Experts with similar gate values may produce substantially different responses.

2. Structural coupling in SwiGLU experts: Expert output arises from nonlinear interaction among gate, up, and down projections—cannot be characterized by a single matrix norm.

3. Direction-dependent specialization: An expert with weak global responses may exhibit strong behavior within local input regions favored by the router.

Theoretical Foundation

For a SwiGLU expert with input $x = \bar{x}\Gamma_l$ (where $\bar{x}$ is the RMS-normalized input direction and $\Gamma_l$ is the diagonal RMSNorm scaling matrix), the forward transformation is:

$$f_{l,e}(x) = \left(\mathrm{SiLU}\left(\bar{x}\Gamma_l W_{l,e}^{\mathrm{gate}}\right) \odot \bar{x}\Gamma_l W_{l,e}^{\mathrm{up}}\right) W_{l,e}^{\mathrm{down}}$$

Since $|\mathrm{SiLU}(z)| \le |z|$, the output admits the upper bound:

$$\|f_{l,e}(x)\|_2 \leq \|\bar{x}\|_2^2 \|\Gamma_l W_{l,e}^{\text{gate}}\|_2 \|\Gamma_l W_{l,e}^{\text{up}}\|_2 \|W_{l,e}^{\text{down}}\|_2$$

This multiplicative coupling motivates the GSP design.

Methodology

1. Global Spectral Proxy (GSP)

Purpose: Estimate each expert's global transformation capacity from frozen SwiGLU weights without calibration data.

Score definition: For the $i$-th routed expert slot in layer $l$ for token $t$:

$$s_{l,t,i}^{\mathrm{GSP}} = g_{l,t,i} \cdot A_l^{\mathrm{GSP}}(e_i)$$

where $g_{l,t,i}$ is the router gate and $A_l^{\mathrm{GSP}}(e_i)$ is a static amplification proxy.

Two complementary factorization paths:

$$a_{l,e}^{\mathrm{up}} = \|\Gamma_l W_{l,e}^{\mathrm{up}}\|_F \cdot \|\Gamma_l W_{l,e}^{\mathrm{gate}} W_{l,e}^{\mathrm{down}}\|_F$$ $$a_{l,e}^{\mathrm{gate}} = \|\Gamma_l W_{l,e}^{\mathrm{gate}}\|_F \cdot \|\Gamma_l W_{l,e}^{\mathrm{up}} W_{l,e}^{\mathrm{down}}\|_F$$

Geometric mean aggregation:

$$\tilde{a}_{l,e}^{\mathrm{GSP}} = \sqrt{a_{l,e}^{\mathrm{up}} a_{l,e}^{\mathrm{gate}}}$$

Layer-wise normalization:

$$A_l^{\mathrm{GSP}}(e) = \frac{\tilde{a}_{l,e}^{\mathrm{GSP}}}{\frac{1}{N_l}\sum_{j=1}^{N_l} \tilde{a}_{l,j}^{\mathrm{GSP}} + \epsilon}$$

2. Router-Conditioned Refinement (RCR)

Purpose: Evaluate expert response along router-preferred directions using centered router weights.

Centered router weights (invariant to shared translation):

$$\tilde{w}_{l,e} = w_{l,e} - \frac{1}{N_l}\sum_{j=1}^{N_l} w_{l,j}$$

Directional prototype (RMS-normalized):

$$q_{l,e} = \frac{\tilde{w}_{l,e}}{\mathrm{RMS}(\tilde{w}_{l,e}) + \epsilon} \cdot \rho_l$$

Router-conditioned amplification:

$$\tilde{a}_{l,e}^{\mathrm{RCR}} = \frac{\|f_{l,e}(q_{l,e})\|_2}{\|q_{l,e}\|_2 + \epsilon}$$

Score combination with gate:

$$s_{l,t,i}^{\mathrm{RCR}} = g_{l,t,i} A_l^{\mathrm{RCR}}(e_i)$$

3. Conservative Fusion

ACE combines both perspectives using a maximum criterion:

$$c_{l,t,i}^{\mathrm{ACE}} = \max\left(p_{l,t,i}^{\mathrm{GSP}}, p_{l,t,i}^{\mathrm{RCR}}\right)$$

A slot is skipped only if both views identify it as low-contribution:

$$\mathcal{S}_{l,t}^{\mathrm{ACE}} = \mathcal{S}_{l,t}^{\mathrm{GSP}} \cap \mathcal{S}_{l,t}^{\mathrm{RCR}}$$

The top-1 expert (highest original router gate) is always retained.

Empirical Validation / Results

Experimental Setup

• Models: Qwen3-30B-A3B-Instruct-2507, Qwen3.6-35B-A3B, Gemma-4-26B-A4Bit
• Benchmarks: WikiText-2 PPL + 7 downstream tasks (ARC-C, ARC-E, PIQA, MATH-500, GPQA-Diamond, HumanEval, LiveCodeBench)
• Baselines: Score, NAEE, MoDES, DiEP, AIMER, Top-P, SERE, XShare, ExpertSparsity

Key Results on Qwen3-30B-A3B (Table 1 excerpt)

Key Results on Qwen3.6-35B-A3B (Table 2 excerpt)

Component Ablation (Table 4)

Fusion Strategy Ablation (Table 5)

Efficiency Results

Threshold construction cost (Table 6): Quantile mapping takes 1.8 minutes vs. 16.5 minutes for binary search (9.2× reduction) and 18.9 minutes for MoDES frontier search (10.5× reduction).

Latency at 60% skipping (Table 7, A100 GPU):

Achieves 1.72–2.25× TTFT and 1.31–1.41× TPOT speedups.

Theoretical and Practical Implications

Theoretical Contributions

1. Calibration-free contribution estimation: GSP provides a principled upper-bound-derived proxy for SwiGLU expert transformation capacity, addressing the multiplicative coupling of gate/up/down projections.

2. Directional refinement without data: RCR leverages the mathematical property that softmax routing is invariant to shared translation of router weights, enabling data-free construction of expert-specific direction prototypes via exponential tilting under Gaussian approximation.

3. Conservative dual-view fusion: The max-criterion (equivalently, intersection of low-contribution sets) provides a theoretical safeguard against erroneously skipping direction-specialized experts.

Practical Implications

• Deployment efficiency: All expert statistics computed offline; runtime requires only table lookups and lightweight scalar operations.
• Checkpoint preservation: No modification to pretrained router or expert parameters.
• Threshold transferability: Thresholds for the same realized budget remain closely clustered across workloads and can be transferred across datasets.
• Versatility: Effective across different MoE architectures (Qwen3, Qwen3.6, Gemma-4) and model scales.

Conclusion

ACE introduces a training-free, calibration-free, and checkpoint-preserving framework for token-adaptive expert skipping in MoE-based LLMs. By combining GSP's global structural capacity estimation with RCR's router-conditioned directional refinement under a conservative max-fusion rule, ACE reliably identifies low-contribution expert slots without requiring calibration data or model modification.

Key takeaways:

• ACE consistently outperforms static and dynamic baselines, with advantages growing under aggressive skipping (40–60%)
• Achieves state-of-the-art quality-efficiency trade-offs: 2.25× prefill and 1.41× decoding speedups
• Requires only 1.8 minutes for threshold construction (9.2–10.5× faster than alternatives)

Future directions:

• Threshold transfer across workloads without matched calibration
• Repeated-run uncertainty analysis (current results are single deterministic runs)
• Distributed expert dispatch optimization
• Extension to broader MoE architectures and multimodal models