# Fewer Expert Transfers Without Changing Routing: Expert Eviction for MoE Offloading

> Evicting the expert needed farthest in the future copies 2.1x fewer experts than LRU with zero quality loss, making expert eviction a better optimization target than learned routing.

- **Source:** [arXiv](https://arxiv.org/abs/2611.15842)
- **Published:** 2026-11-14
- **Permalink:** https://picx.dev/p/RMfnOa
- **Whiteboard:** https://picx.dev/p/RMfnOa/image

## Summary

## Summary (Overview)

- **Core finding**: Evicting the expert whose next use lies farthest in the future (Belady's rule) copies **2.1× fewer experts** than LRU with **zero quality loss**, since it changes no routing decisions.
- **Key negative result**: Every tested policy that holds an expert set fixed and decodes faster than LRU increases next-token loss by **at least 0.067 nats** on an L40S GPU.
- **Critical insight**: Fewer routing switches do not imply fewer expert copies—a switch only changes the routing decision, while a copy happens only when requested weights are absent from cache.
- **Lookahead requirement**: Sixteen tokens of future demand recover 96% of the reduction, while past-request-based rules recover only 27%, making future-request prediction an open problem.
- **Learned routing fails to help**: A reimplemented learned controller (option-critic) does not improve the speed-quality trade-off, and a router fine-tuned for simulated cache hits copies as many experts as before fine-tuning.

---

## Introduction and Theoretical Foundation

Mixture-of-experts (MoE) language models activate only a few experts per token. For example, OLMoE-1B-7B uses about 1B of its 7B parameters at a time. When expert weights exceed GPU memory, the system keeps them in CPU memory and copies requested experts to the GPU on demand.

**The two competing approaches:**

1. **Cache management**: Keep the router unchanged and improve the eviction policy (e.g., LRU). This preserves model outputs exactly.
2. **Temporally extended routing**: Train the router to keep the same expert set for several tokens, treating fewer routing switches as a proxy for fewer copies.

**The core theoretical problem**: Fewer switches need not mean fewer copies. Consider a cache of two slots:
- Sequence $A, B, A, B, \ldots$ switches at every token but misses only twice (when A and B first enter the cache).
- Sequence $A, \bar{A}, B, B, \bar{C}, C, \ldots$ switches half as often, yet each switch requests an uncached expert.

**Decoding time model**: When copies block computation, decoding time is

$$
\mathrm{ms/token} = c + \rho \cdot \tau, \tag{1}
$$

where $c$ is compute time, $\rho$ is the number of experts copied per token, and $\tau$ is the time to copy one expert. The hardware sets $\tau$, which varies fivefold across GPUs (from 0.24 ms on H100/H200 to 1.17 ms on RTX 8000).

---

## Methodology

**Model and setup**:
- Frozen OLMoE-1B-7B with a router trained to reuse experts (following Zhu et al., 2026)
- 8 of 16 MoE layers offloaded; each offloaded layer has 64 bf16 experts (12.6 MB each) in pinned CPU memory, with GPU holding $C = 32$ experts
- 16 held-out sequences of at most 384 tokens, teacher-forced with batch size one
- Real, blocking CPU-to-GPU copies in a PyTorch harness; quality, copy counts, and decoding time from the same run

**Policies compared**:

| Policy | Description |
|---|---|
| **LRU** (lossless) | Evicts least recently used expert; preserves every routing decision |
| **Farthest-future eviction** | Belady's rule: evict expert whose next use lies farthest in the future (offline) |
| **Dwell-k** | Holds cache fixed for k tokens, then refills with most-requested experts from a recent window (8k tokens for Dwell-16, k for Dwell-64) |
| **Static** | Fills cache from first 64 tokens, keeps it for the rest of the sequence |
| **Learned** | Reimplementation of Shen and Henderson's option-critic controller with termination head (β) and selection head (µ) |

**Learned controller details**: At each token, a termination head outputs a score $\beta$, and the cache is replaced when $\beta$ exceeds a threshold. Training penalizes switches, changed experts, and misses with hand-set weights rather than the measured $\tau$.

**Evaluation**: Next-token NLL, tokens per second, and steady-state copies per token ($\rho$), excluding the copies that first fill the cache.

---

## Empirical Validation / Results

### 3.1 Quality and throughput across expert-transfer costs

On the L40S (where $\tau = 0.51$ ms):

| Policy | ΔNLL | tok/s | ρ |
|---|---|---|---|
| All 64 experts cached | +0.000 | 49.0 | 0.0 |
| LRU (lossless) | +0.000 | 38.3 | 9.6 |
| **Farthest-future eviction** | **+0.000** | **42.5** | **4.6** |
| Dwell-16 (window 8×) | +0.067 | 42.2 | 4.5 |
| Dwell-64 (window 1×) | +0.085 | 43.5 | 3.9 |
| Static (64-token profile) | +0.110 | 44.0 | 3.4 |
| Learned, thr. 0.5 | +0.076 | 33.8 | 12.3 |
| Learned, thr. 0.9 | +0.350 | 43.6 | 0.9 |

**Key result**: LRU is 22% slower than the all-64 baseline (38.3 vs 49.0 tok/s). Policies that beat LRU's speed all incur at least 0.067 nats of NLL loss. Farthest-future eviction is the only policy that beats LRU's speed with zero loss.

**Cross-platform prediction**: Equation (1) predicts the fastest policy at 10× and 50× copy costs, and the predicted winner matches the measured winner at both costs.

### 3.2 Better eviction without changing routing

- Farthest-future eviction lowers copies from 9.6 to 4.6 per token (2.1× reduction) with every routing decision unchanged
- A lower bound of 14,587 copies vs. the rule's 15,087 places it within **3.5% of optimal** (Belady's proof assumes one request at a time, but MoE requests 8 experts per token, so optimality is not guaranteed)
- **Lookahead requirements**:
  - Next 16 tokens of demand: recovers **96%** of the reduction
  - Exponentially decayed past-request counts: recovers **27%**
  - Learned scorer with router probabilities: recovers **32–33%**

### 3.3 Learned routing does not improve the measured trade-off

- Across six thresholds, the learned controller replaces its cache every 2.9 to 73 tokens
- At thresholds 0.4–0.7, simpler policies give better quality at the same traffic or less traffic at the same quality
- At threshold 0.5: controller and Dwell-16 raise NLL by similar amounts (0.076 vs 0.067), but the controller copies **2.7× more experts** (and this excludes the time spent running its two heads)
- Thresholds 0.8 and 0.9 lower traffic only by raising NLL by 0.238 and 0.350 nats
- **Selection head diagnosis**: When the controller picks a new set every ~3 tokens, NLL rises by only 0.030 nats; when it keeps each set for 73 tokens, NLL rises by 0.350—the selector predicts near-term demand well, but the termination head keeps each prediction too long
- **GRPO fine-tuning failure**: A router fine-tuned with GRPO rewarding simulated LRU cache hits copies 32,050 experts vs. 31,915 before fine-tuning (at the trained layer alone: 10,963 vs 10,954)

---

## Theoretical and Practical Implications

**Theoretical implications**:
- The paper provides a clean decomposition of decoding time into compute and copy components (Equation 1), showing that the optimal policy depends on the hardware-specific ratio of compute to copy time
- It demonstrates that **routing switches are the wrong proxy** for serving cost—cache misses are the correct measure
- The near-optimality of Belady's rule (within 3.5%) despite the multi-expert-per-token setting suggests the single-request optimality proof extends well in practice

**Practical implications**:
- For deployments with CPU-GPU memory hierarchies, **eviction policy is a better target for learning than routing**
- The 16-token lookahead result suggests practical prefetching or demand prediction systems could capture most of the benefit
- The gap between future-request-based (96% recovery) and past-request-based (27% recovery) rules defines a concrete prediction problem for the serving community
- Hardware matters: on slower links (L4, RTX 8000 with $\tau > 1$ ms), the speed gains from better caching are amplified

---

## Conclusion

**Main takeaways**:
1. **Expert eviction, not routing, is the better optimization target**: Evicting the expert needed farthest in the future copies 2.1× fewer experts than LRU, comes within 3.5% of optimal, and changes no routing decision.
2. **Future demand prediction is the open problem**: Sixteen tokens of lookahead recover 96% of the reduction, but past-request rules recover only 27–33%.
3. **Learned routing approaches fail to deliver**: The option-critic controller does not improve the speed-quality trade-off at moderate thresholds, and its low-traffic settings incur large quality losses (0.238–0.350 nats).

**Limitations**:
- Results come from one model (OLMoE-1B-7B), batch-1 teacher forcing, and copies that do not overlap computation
- Batching and asynchronous copies may shrink the gains
- The learned controller is a reimplementation, not the original published checkpoint

**Future directions**: Closing the gap between future-request-based and past-request-based eviction rules is an open prediction problem. The authors suggest that expert eviction—not routing—is where learning-based approaches should focus their efforts.

---

_Markdown view of https://picx.dev/p/RMfnOa, served by PicX — AI-generated visual whiteboard summaries of research papers._