Summary (Overview)

• Core finding: Evicting the expert whose next use lies farthest in the future (Belady's rule) copies 2.1× fewer experts than LRU with zero quality loss, since it changes no routing decisions.
• Key negative result: Every tested policy that holds an expert set fixed and decodes faster than LRU increases next-token loss by at least 0.067 nats on an L40S GPU.
• Critical insight: Fewer routing switches do not imply fewer expert copies—a switch only changes the routing decision, while a copy happens only when requested weights are absent from cache.
• Lookahead requirement: Sixteen tokens of future demand recover 96% of the reduction, while past-request-based rules recover only 27%, making future-request prediction an open problem.
• Learned routing fails to help: A reimplemented learned controller (option-critic) does not improve the speed-quality trade-off, and a router fine-tuned for simulated cache hits copies as many experts as before fine-tuning.

---

Introduction and Theoretical Foundation

Mixture-of-experts (MoE) language models activate only a few experts per token. For example, OLMoE-1B-7B uses about 1B of its 7B parameters at a time. When expert weights exceed GPU memory, the system keeps them in CPU memory and copies requested experts to the GPU on demand.

The two competing approaches:

1. Cache management: Keep the router unchanged and improve the eviction policy (e.g., LRU). This preserves model outputs exactly.
2. Temporally extended routing: Train the router to keep the same expert set for several tokens, treating fewer routing switches as a proxy for fewer copies.

The core theoretical problem: Fewer switches need not mean fewer copies. Consider a cache of two slots:

• Sequence $A, B, A, B, \ldots$ switches at every token but misses only twice (when A and B first enter the cache).
• Sequence $A, \bar{A}, B, B, \bar{C}, C, \ldots$ switches half as often, yet each switch requests an uncached expert.

Decoding time model: When copies block computation, decoding time is

$$\mathrm{ms/token} = c + \rho \cdot \tau, \tag{1}$$

where $c$ is compute time, $\rho$ is the number of experts copied per token, and $\tau$ is the time to copy one expert. The hardware sets $\tau$, which varies fivefold across GPUs (from 0.24 ms on H100/H200 to 1.17 ms on RTX 8000).

---

Methodology

Model and setup:

• Frozen OLMoE-1B-7B with a router trained to reuse experts (following Zhu et al., 2026)
• 8 of 16 MoE layers offloaded; each offloaded layer has 64 bf16 experts (12.6 MB each) in pinned CPU memory, with GPU holding $C = 32$ experts
• 16 held-out sequences of at most 384 tokens, teacher-forced with batch size one
• Real, blocking CPU-to-GPU copies in a PyTorch harness; quality, copy counts, and decoding time from the same run

Policies compared:

Learned controller details: At each token, a termination head outputs a score $\beta$, and the cache is replaced when $\beta$ exceeds a threshold. Training penalizes switches, changed experts, and misses with hand-set weights rather than the measured $\tau$.

Evaluation: Next-token NLL, tokens per second, and steady-state copies per token ($\rho$), excluding the copies that first fill the cache.

---

Empirical Validation / Results

3.1 Quality and throughput across expert-transfer costs

On the L40S (where $\tau = 0.51$ ms):

Key result: LRU is 22% slower than the all-64 baseline (38.3 vs 49.0 tok/s). Policies that beat LRU's speed all incur at least 0.067 nats of NLL loss. Farthest-future eviction is the only policy that beats LRU's speed with zero loss.

Cross-platform prediction: Equation (1) predicts the fastest policy at 10× and 50× copy costs, and the predicted winner matches the measured winner at both costs.

3.2 Better eviction without changing routing

• Farthest-future eviction lowers copies from 9.6 to 4.6 per token (2.1× reduction) with every routing decision unchanged
• A lower bound of 14,587 copies vs. the rule's 15,087 places it within 3.5% of optimal (Belady's proof assumes one request at a time, but MoE requests 8 experts per token, so optimality is not guaranteed)
• Lookahead requirements:
  • Next 16 tokens of demand: recovers 96% of the reduction
  • Exponentially decayed past-request counts: recovers 27%
  • Learned scorer with router probabilities: recovers 32–33%

3.3 Learned routing does not improve the measured trade-off

• Across six thresholds, the learned controller replaces its cache every 2.9 to 73 tokens
• At thresholds 0.4–0.7, simpler policies give better quality at the same traffic or less traffic at the same quality
• At threshold 0.5: controller and Dwell-16 raise NLL by similar amounts (0.076 vs 0.067), but the controller copies 2.7× more experts (and this excludes the time spent running its two heads)
• Thresholds 0.8 and 0.9 lower traffic only by raising NLL by 0.238 and 0.350 nats
• Selection head diagnosis: When the controller picks a new set every ~3 tokens, NLL rises by only 0.030 nats; when it keeps each set for 73 tokens, NLL rises by 0.350—the selector predicts near-term demand well, but the termination head keeps each prediction too long
• GRPO fine-tuning failure: A router fine-tuned with GRPO rewarding simulated LRU cache hits copies 32,050 experts vs. 31,915 before fine-tuning (at the trained layer alone: 10,963 vs 10,954)

---

Theoretical and Practical Implications

Theoretical implications:

• The paper provides a clean decomposition of decoding time into compute and copy components (Equation 1), showing that the optimal policy depends on the hardware-specific ratio of compute to copy time
• It demonstrates that routing switches are the wrong proxy for serving cost—cache misses are the correct measure
• The near-optimality of Belady's rule (within 3.5%) despite the multi-expert-per-token setting suggests the single-request optimality proof extends well in practice

Practical implications:

• For deployments with CPU-GPU memory hierarchies, eviction policy is a better target for learning than routing
• The 16-token lookahead result suggests practical prefetching or demand prediction systems could capture most of the benefit
• The gap between future-request-based (96% recovery) and past-request-based (27% recovery) rules defines a concrete prediction problem for the serving community
• Hardware matters: on slower links (L4, RTX 8000 with $\tau > 1$ ms), the speed gains from better caching are amplified

---

Conclusion

Main takeaways:

1. Expert eviction, not routing, is the better optimization target: Evicting the expert needed farthest in the future copies 2.1× fewer experts than LRU, comes within 3.5% of optimal, and changes no routing decision.
2. Future demand prediction is the open problem: Sixteen tokens of lookahead recover 96% of the reduction, but past-request rules recover only 27–33%.
3. Learned routing approaches fail to deliver: The option-critic controller does not improve the speed-quality trade-off at moderate thresholds, and its low-traffic settings incur large quality losses (0.238–0.350 nats).

Limitations:

• Results come from one model (OLMoE-1B-7B), batch-1 teacher forcing, and copies that do not overlap computation
• Batching and asynchronous copies may shrink the gains
• The learned controller is a reimplementation, not the original published checkpoint

Future directions: Closing the gap between future-request-based and past-request-based eviction rules is an open prediction problem. The authors suggest that expert eviction—not routing—is where learning-based approaches should focus their efforts.