# Accelerating Offloading-Based MoE Inference by Interleaving Concurrent Requests

> Bevy replaces expert prefetching with runahead inference, scheduling ready requests to reveal actual expert needs and achieving 1.26–3.46x throughput over llama.cpp.

- **Source:** [arXiv](https://arxiv.org/abs/2611.04794)
- **Published:** 2026-11-07
- **Permalink:** https://picx.dev/p/Klg804
- **Whiteboard:** https://picx.dev/p/Klg804/image

## Summary

## Summary (Overview)

- **Key Finding**: Expert caching reduces prefetch precision in offloading-based MoE inference, meaning prefetching wastes bandwidth and GPU memory on unnecessary experts as cache size grows.
- **Key Insight**: Proper request scheduling beats prefetching—concurrent requests can use idle link bandwidth to fetch experts *known* to be needed, without prediction.
- **Core Contribution**: **Bevy**, an MoE inference system based on **runahead inference**, where ready requests proceed to later layers and reveal their expert needs early, enabling computation-fetch overlap without prediction.
- **Technical Innovation**: A self-balancing scheduler that dynamically batches ready requests at each layer (opportunistic layerwise grouping) and prioritizes lagging requests (lowest-layer-first), preserving shared-weight reuse while maximizing overlap.
- **Results**: Bevy delivers **1.26–3.46×** the decode throughput of llama.cpp and **1.01–1.55×** that of the fastest prefetching baseline across two local platforms (Apple M6 Mac mini, NVIDIA Jetson AGX Thor), two MoE models, and two workloads.

## Introduction and Theoretical Foundation

### Background

Large language models (LLMs) are increasingly deployed on local systems (desktops, AI workstations). These systems often serve **concurrent requests** from personal agents, chat applications, and parallel subtasks. Mixture-of-experts (MoE) models are well-suited for local inference because they activate only a few experts per token, reducing computation.

### Problem: Expert Offloading

The full set of expert weights can exceed GPU memory. **Expert offloading systems** keep a subset of experts in GPU memory as an *expert cache* (managed by LRU/LFU), fetching missing experts from secondary storage (host memory or SSD) over a link like PCIe. The conventional sequence per layer:

1. Gate network selects experts
2. Fetch absent experts from cache
3. Compute MoE

This leaves the **GPU idle during fetches** and the **link idle during computation**.

### Prefetching and Its Limitation

Prefetching predicts experts needed at later layers and fetches them during computation. However, the authors find that prefetching is **harmful when combined with an expert cache**:

- The cache already holds many needed experts (e.g., a 30% cache holds 54.1% of needed experts on average)
- The cache tends to retain *easily predictable* experts
- Remaining candidates are harder to predict → **precision drops**

The paper decomposes this precision drop (Figure 2) into:
- **Region (1)**: Drop due to fewer needed candidates among nonresident experts (reproduced by a randomized cache)
- **Region (2)**: Additional drop because the actual cache retains predictable experts, leaving harder-to-predict candidates

### Key Insight

> "Concurrent requests can use otherwise idle link bandwidth to fetch experts known to be needed, without prediction. That is, proper request scheduling beats prefetching."

This motivates **runahead inference**: letting a request proceed as soon as *its own* experts are available, reaching the next layer earlier to identify needed experts (including missing ones), then fetching those while the GPU computes other requests.

## Methodology

### Runahead Inference

**Definition**: A scheduling method where ready requests run ahead to later layers independently, without waiting for other requests' fetches. This keeps the link busy fetching only experts *known* to be needed.

**Step time bound**: A step cannot be shorter than $\max(\text{compute time}, \text{fetch time})$, where:
- **Compute time** = total GPU busy time across all requests
- **Fetch time** = total link busy time fetching missing experts

### Two Key Challenges

**C1: Grouping Granularity**
- Running each request separately repeats reads of shared weights → increases compute time (e.g., 6 requests: 126.8 ms separately vs. 56.5 ms as one batch)
- Static grouping (e.g., 2:2:2) reduces compute time but:
  - Restricts runahead opportunities
  - Cannot adapt to step-to-step variation in fetch time (55.7–160.1 ms over 256 steps)

**C2: Scheduling Order**
- Fetch order affects overlap: serving the request with fewest missing experts first enables earlier computation overlap
- But always favoring fewer misses creates **stragglers** (requests with more misses fall behind, accumulating delays)

### Bevy System Design

Bevy consists of three modules (Figure 6):
1. **Compute Scheduler** — selects computation groups
2. **Job Issuer** — checks expert availability, issues or blocks jobs
3. **Expert Fetcher** — fetches missing experts

Jobs are defined as: job $l$ runs MoE of layer $l$ + attention + gate of layer $l+1$ (labeled $l \rightarrow l+1$).

#### Scheduling Policy (Algorithm 1)

**Fetch selection** (lowest-layer-first, then SRTF):

$$
\text{SELECT\_FETCH\_JOB}(J_B) = \arg\min_{j \in J_B} (l(j), m(j))
$$

where $l(j)$ = layer of job $j$, $m(j)$ = number of remaining missing experts.

**Compute group selection**:

$$
\text{SELECT\_COMPUTE\_GROUP}(J_R) = \{j \in J_R \mid l(j) = l_{\min}\}
$$

where $l_{\min} = \min_{j \in J_R} l(j)$ — one batch at the lowest ready layer.

#### Opportunistic Layerwise Grouping

- Groups *all ready jobs at the selected layer* as one batch
- Group lasts only for one layer, not the whole step
- **Self-balancing property**:
  - Few misses → more jobs ready together → larger batches reduce compute time
  - Many misses → fewer jobs ready → smaller batches run ahead, overlapping with fetches

#### Expert Cache Management

- Tracks pending fetches to deduplicate shared expert reads
- **Pinning**: cache entries are pinned while referenced by active jobs; only zero-reference entries can be evicted
- **Deadlock prevention**: runahead enabled only when cache can hold every request's selected experts without relying on sharing (e.g., 128 entries for 16 requests × 8 experts = 1.25% of experts for Qwen3.6-35B-A3B)

## Empirical Validation / Results

### Experimental Setup

| Component | Details |
|-----------|---------|
| **Platforms** | Apple Mac mini (M6, 24 GB), NVIDIA Jetson AGX Thor (128 GB, PCIe 5.0 SSD) |
| **Models** | Qwen3.6-35B-A3B, Gemma4-26B-A4B (both Q4_0 quantized) |
| **Workloads** | Chat (ShareGPT), Agentic Coding (SWE-agent trajectories) |
| **Baselines** | llama.cpp (no prefetch), SpecMD (gate-based prefetch), FineMoE (history-based prefetch) |

### End-to-End Performance

**Varying concurrency (2–16 requests, 30% cache)**:
- Bevy achieves highest throughput in **all 40 configurations**
- **1.01–1.38×** faster than fastest baseline
- **1.29–3.46×** faster than llama.cpp
- Speedup smallest at 2 requests (down to 1.01× for Gemma4 on Thor)

**Varying cache size (10–70% of experts, 8 requests)**:
- Bevy achieves highest throughput at **every cache size**: **1.07–1.55×** vs. fastest baseline
- Prefetching baselines approach llama.cpp as cache grows (prefetch precision drops)
- Bevy's speedup reaches **1.40–1.55×** at 50–60% cache on M6

### Where the Speedup Comes From

**Step time breakdown** (Figure 10a, Gemma4-26B-A4B, 8 Agentic Coding requests, 30% cache):
- llama.cpp + LFU: link and GPU never busy simultaneously; GPU idle 60% of step
- FineMoE/SpecMD: overlap 36–40% of step; GPU still waits 57–59%
- **Bevy**: link and GPU busy together **77%** of step; GPU waits only **10%**

**Expert fetches** (Figure 10b):
- Prefetching baselines fetch **1.4–1.5×** more experts than llama.cpp + LFU
- **51–64%** of prefetched experts go unused
- Bevy fetches ~235 experts/step, nearly identical to llama.cpp + LFU

### Ablation Studies

**Self-balancing grouping vs. static grouping** (Figure 11):
- Which static grouping is fastest depends on the number of missing experts
- Bevy achieves lowest step time at every point: **0.9–4.3 ms below** fastest static grouping

**Lowest-layer-first scheduling** (Figure 12):
- Compute order has larger effect: FIFO makes steps **12.3% longer** (single-job groups rise from 56% to 67%)
- Fetch order: global SRTF makes steps **1.7% longer**
- Both changed: **13.6% longer**
- GPU idle time does not increase — longer steps come from extra computation, not waiting

### Combining Bevy with Prefetching

- Adding Mixtral-Offloading-style prefetching to Bevy does **not** improve throughput
- With 2–4 requests: 4.2–5.4% *lower* throughput (46% and 39% of prefetches wasted)
- With ≥8 requests: link has no idle time for prefetching → no benefit

## Theoretical and Practical Implications

### Theoretical Contributions

1. **Formalizes the interaction between caching and prefetching**: Shows that expert caching *reduces* prefetch precision, not just by shrinking the candidate pool but by selectively retaining predictable experts—a previously unrecognized negative interaction.

2. **Runahead inference as an alternative paradigm**: Demonstrates that for concurrent-request workloads, scheduling can replace prediction for utilizing idle link bandwidth, eliminating prediction error entirely.

3. **Self-balancing scheduling principle**: Shows that dynamic, progress-aware grouping at layer granularity adapts to per-step variation in fetch demand without profiling, outperforming any static grouping.

### Practical Implications

- **Local LLM serving**: Bevy directly improves concurrent request throughput on personal devices (desktops, AI workstations), which is increasingly important for personal agents and multi-task chat applications.
- **No model changes**: Bevy preserves exact model output (unlike approaches like Expert Deferral in KTransformers).
- **Applicability**: Works on both unified memory (Apple silicon, SoCs) and discrete GPUs (by offloading to CPU memory instead of SSD).
- **Orthogonality**: Can be combined with request-batching-for-expert-sharing systems and CPU–GPU hybrid computation (left as future work).

**Limitations noted**:
- Smallest gains at very low concurrency (2 requests)
- Requires sufficient cache capacity to avoid deadlock (though this is only ~1.25% of experts for tested models)

## Conclusion

The paper identifies that expert caching reduces prefetch precision in local MoE inference, wasting link bandwidth on unneeded experts. **Bevy** addresses this via **runahead inference**: letting ready requests run ahead to reveal actual expert demand, fetching only missing experts while computing other requests, without any prediction. Its **self-balancing scheduler** batches ready requests by layer and prioritizes lagging ones, preserving shared-weight reuse while maximizing computation-fetch overlap.

**Key results**: 1.26–3.46× decode throughput over llama.cpp and 1.01–1.55× over the fastest prefetching baseline across two platforms, two models, and two workloads.

**Future directions**:
- Extending Bevy to use CPU computation alongside GPU in hybrid systems
- Combining with expert-sharing-aware request batching
- Exploring runahead inference in other offloading contexts (e.g., dense model offloading)

---

_Markdown view of https://picx.dev/p/Klg804, served by PicX — AI-generated visual whiteboard summaries of research papers._