Summary (Overview)

• Key Finding: Expert caching reduces prefetch precision in offloading-based MoE inference, meaning prefetching wastes bandwidth and GPU memory on unnecessary experts as cache size grows.
• Key Insight: Proper request scheduling beats prefetching—concurrent requests can use idle link bandwidth to fetch experts known to be needed, without prediction.
• Core Contribution: Bevy, an MoE inference system based on runahead inference, where ready requests proceed to later layers and reveal their expert needs early, enabling computation-fetch overlap without prediction.
• Technical Innovation: A self-balancing scheduler that dynamically batches ready requests at each layer (opportunistic layerwise grouping) and prioritizes lagging requests (lowest-layer-first), preserving shared-weight reuse while maximizing overlap.
• Results: Bevy delivers 1.26–3.46× the decode throughput of llama.cpp and 1.01–1.55× that of the fastest prefetching baseline across two local platforms (Apple M6 Mac mini, NVIDIA Jetson AGX Thor), two MoE models, and two workloads.

Introduction and Theoretical Foundation

Background

Large language models (LLMs) are increasingly deployed on local systems (desktops, AI workstations). These systems often serve concurrent requests from personal agents, chat applications, and parallel subtasks. Mixture-of-experts (MoE) models are well-suited for local inference because they activate only a few experts per token, reducing computation.

Problem: Expert Offloading

The full set of expert weights can exceed GPU memory. Expert offloading systems keep a subset of experts in GPU memory as an expert cache (managed by LRU/LFU), fetching missing experts from secondary storage (host memory or SSD) over a link like PCIe. The conventional sequence per layer:

1. Gate network selects experts
2. Fetch absent experts from cache
3. Compute MoE

This leaves the GPU idle during fetches and the link idle during computation.

Prefetching and Its Limitation

Prefetching predicts experts needed at later layers and fetches them during computation. However, the authors find that prefetching is harmful when combined with an expert cache:

• The cache already holds many needed experts (e.g., a 30% cache holds 54.1% of needed experts on average)
• The cache tends to retain easily predictable experts
• Remaining candidates are harder to predict → precision drops

The paper decomposes this precision drop (Figure 2) into:

• Region (1): Drop due to fewer needed candidates among nonresident experts (reproduced by a randomized cache)
• Region (2): Additional drop because the actual cache retains predictable experts, leaving harder-to-predict candidates

Key Insight

"Concurrent requests can use otherwise idle link bandwidth to fetch experts known to be needed, without prediction. That is, proper request scheduling beats prefetching."

This motivates runahead inference: letting a request proceed as soon as its own experts are available, reaching the next layer earlier to identify needed experts (including missing ones), then fetching those while the GPU computes other requests.

Methodology

Runahead Inference

Definition: A scheduling method where ready requests run ahead to later layers independently, without waiting for other requests' fetches. This keeps the link busy fetching only experts known to be needed.

Step time bound: A step cannot be shorter than $\max(\text{compute time}, \text{fetch time})$, where:

• Compute time = total GPU busy time across all requests
• Fetch time = total link busy time fetching missing experts

Two Key Challenges

C1: Grouping Granularity

• Running each request separately repeats reads of shared weights → increases compute time (e.g., 6 requests: 126.8 ms separately vs. 56.5 ms as one batch)
• Static grouping (e.g., 2:2:2) reduces compute time but:
  • Restricts runahead opportunities
  • Cannot adapt to step-to-step variation in fetch time (55.7–160.1 ms over 256 steps)

C2: Scheduling Order

• Fetch order affects overlap: serving the request with fewest missing experts first enables earlier computation overlap
• But always favoring fewer misses creates stragglers (requests with more misses fall behind, accumulating delays)

Bevy System Design

Bevy consists of three modules (Figure 6):

1. Compute Scheduler — selects computation groups
2. Job Issuer — checks expert availability, issues or blocks jobs
3. Expert Fetcher — fetches missing experts

Jobs are defined as: job $l$ runs MoE of layer $l$ + attention + gate of layer $l+1$ (labeled $l \rightarrow l+1$).

Scheduling Policy (Algorithm 1)

Fetch selection (lowest-layer-first, then SRTF):

$$\mathrm{SELECT\_FETCH\_JOB}(J_B) = \arg\min_{j \in J_B} (l(j), m(j))$$

where $l(j)$ = layer of job $j$, $m(j)$ = number of remaining missing experts.

Compute group selection:

$$\mathrm{SELECT\_COMPUTE\_GROUP}(J_R) = \{j \in J_R \mid l(j) = l_{\min}\}$$

where $l_{\min} = \min_{j \in J_R} l(j)$ — one batch at the lowest ready layer.

Opportunistic Layerwise Grouping

• Groups all ready jobs at the selected layer as one batch
• Group lasts only for one layer, not the whole step
• Self-balancing property:
  • Few misses → more jobs ready together → larger batches reduce compute time
  • Many misses → fewer jobs ready → smaller batches run ahead, overlapping with fetches

Expert Cache Management

• Tracks pending fetches to deduplicate shared expert reads
• Pinning: cache entries are pinned while referenced by active jobs; only zero-reference entries can be evicted
• Deadlock prevention: runahead enabled only when cache can hold every request's selected experts without relying on sharing (e.g., 128 entries for 16 requests × 8 experts = 1.25% of experts for Qwen3.6-35B-A3B)

Empirical Validation / Results

Experimental Setup

End-to-End Performance

Varying concurrency (2–16 requests, 30% cache):

• Bevy achieves highest throughput in all 40 configurations
• 1.01–1.38× faster than fastest baseline
• 1.29–3.46× faster than llama.cpp
• Speedup smallest at 2 requests (down to 1.01× for Gemma4 on Thor)

Varying cache size (10–70% of experts, 8 requests):

• Bevy achieves highest throughput at every cache size: 1.07–1.55× vs. fastest baseline
• Prefetching baselines approach llama.cpp as cache grows (prefetch precision drops)
• Bevy's speedup reaches 1.40–1.55× at 50–60% cache on M6

Where the Speedup Comes From

Step time breakdown (Figure 10a, Gemma4-26B-A4B, 8 Agentic Coding requests, 30% cache):

• llama.cpp + LFU: link and GPU never busy simultaneously; GPU idle 60% of step
• FineMoE/SpecMD: overlap 36–40% of step; GPU still waits 57–59%
• Bevy: link and GPU busy together 77% of step; GPU waits only 10%

Expert fetches (Figure 10b):

• Prefetching baselines fetch 1.4–1.5× more experts than llama.cpp + LFU
• 51–64% of prefetched experts go unused
• Bevy fetches ~235 experts/step, nearly identical to llama.cpp + LFU

Ablation Studies

Self-balancing grouping vs. static grouping (Figure 11):

• Which static grouping is fastest depends on the number of missing experts
• Bevy achieves lowest step time at every point: 0.9–4.3 ms below fastest static grouping

Lowest-layer-first scheduling (Figure 12):

• Compute order has larger effect: FIFO makes steps 12.3% longer (single-job groups rise from 56% to 67%)
• Fetch order: global SRTF makes steps 1.7% longer
• Both changed: 13.6% longer
• GPU idle time does not increase — longer steps come from extra computation, not waiting

Combining Bevy with Prefetching

• Adding Mixtral-Offloading-style prefetching to Bevy does not improve throughput
• With 2–4 requests: 4.2–5.4% lower throughput (46% and 39% of prefetches wasted)
• With ≥8 requests: link has no idle time for prefetching → no benefit

Theoretical and Practical Implications

Theoretical Contributions

1. Formalizes the interaction between caching and prefetching: Shows that expert caching reduces prefetch precision, not just by shrinking the candidate pool but by selectively retaining predictable experts—a previously unrecognized negative interaction.

2. Runahead inference as an alternative paradigm: Demonstrates that for concurrent-request workloads, scheduling can replace prediction for utilizing idle link bandwidth, eliminating prediction error entirely.

3. Self-balancing scheduling principle: Shows that dynamic, progress-aware grouping at layer granularity adapts to per-step variation in fetch demand without profiling, outperforming any static grouping.

Practical Implications

• Local LLM serving: Bevy directly improves concurrent request throughput on personal devices (desktops, AI workstations), which is increasingly important for personal agents and multi-task chat applications.
• No model changes: Bevy preserves exact model output (unlike approaches like Expert Deferral in KTransformers).
• Applicability: Works on both unified memory (Apple silicon, SoCs) and discrete GPUs (by offloading to CPU memory instead of SSD).
• Orthogonality: Can be combined with request-batching-for-expert-sharing systems and CPU–GPU hybrid computation (left as future work).

Limitations noted:

• Smallest gains at very low concurrency (2 requests)
• Requires sufficient cache capacity to avoid deadlock (though this is only ~1.25% of experts for tested models)

Conclusion

The paper identifies that expert caching reduces prefetch precision in local MoE inference, wasting link bandwidth on unneeded experts. Bevy addresses this via runahead inference: letting ready requests run ahead to reveal actual expert demand, fetching only missing experts while computing other requests, without any prediction. Its self-balancing scheduler batches ready requests by layer and prioritizes lagging ones, preserving shared-weight reuse while maximizing computation-fetch overlap.

Key results: 1.26–3.46× decode throughput over llama.cpp and 1.01–1.55× over the fastest prefetching baseline across two platforms, two models, and two workloads.

Future directions:

• Extending Bevy to use CPU computation alongside GPU in hybrid systems
• Combining with expert-sharing-aware request batching
• Exploring runahead inference in other offloading contexts (e.g., dense model offloading)