Summary (Overview)

• Core Problem: Single-GPU long-context inference with Mixture-of-Experts (MoE) models requires spilling the Key-Value Cache (KV-Cache) to CPU memory. The spilled KV serves two opposing purposes—loading to GPU for attention vs. computing in-place on CPU—which demand conflicting physical states (pinned vs. unpinned, contiguous vs. strided).
• Key Insight: The write-time principle—each byte's physical residency is fixed at write time—makes load balancing a property of the data format rather than a runtime mechanism, eliminating post-hoc data reorganization costs.
• Main Contribution: InplaceKVCache, the first KV-Cache abstraction with a four-region layout along two orthogonal dimensions (device affinity × access pattern), enabling dynamic CPU–GPU load balancing without moving data after placement.
• Results: On three MoE models with 32 GB VRAM, WriteScope achieves geometric-mean speedups of 1.5×–2.5× (A100) and 1.4×–1.7× (V100) over four baselines at ≥8K context, supporting 1M-token aggregate inference where vLLM, SGLang, and KTransformers fail.
• Scheduling: A portable roofline performance model determines the optimal CPU share as sequence length evolves, with online feedback tracking CPU cost drift.

Introduction and Theoretical Foundation

Background and Motivation

MoE architectures are natural fits for budget-constrained, privacy-first single-GPU deployment due to sparse activation. However, long-context inference demands far exceed single-GPU capacity: Qwen3-30B-A3B requires 78 GB for 128K-token context, well beyond 32 GB VRAM. The KV-Cache must spill to CPU memory.

The Core Tension

Existing CPU–GPU hybrid systems adopt one of two extremes:

• GPU-centric (FlexGen, SolidAttention): Load all KV back to GPU—bottlenecked by PCIe bandwidth
• CPU-centric (NEO, ScoutAttention): Compute all KV in-place on CPU—bottlenecked by CPU compute

Both rely on AF overlap (asynchronous pre-execution of next layer's attention during FFN phase), which works for dense models where FFN runs entirely on GPU. However, MoE invalidates this: spilled expert parameters reside on CPU, saturating both CPU and PCIe during FFN, turning overlap into contention. Measured impact: attention latency rises from 33.7 ms to 74.2 ms, KV transfer bandwidth drops from 25.4 GB/s to 14.9 GB/s.

Why Existing Abstractions Fail

KV-Cache abstractions offer only storage semantics over a monolithic object of a single physical state. The two uses impose opposing requirements:

• GPU loading: requires pinned memory and logical contiguity (for full DMA bandwidth)
• CPU computation: requires unpinned memory and per-head organization

Preparing buffers for both uses costs 3.5 ms → 185.9 ms per layer per step as context grows from 2K to 64K, negating load-balancing gains.

Methodology

The Write-Time Principle

"A byte has exactly one final physical residency, and that residency is determined at write time."

This fits KV-Cache because its consumption is deterministic (unlike stochastic expert activation, which requires schedule-time arbitration).

InplaceKVCache Four-Region Layout

The cache partitions space along two orthogonal dimensions:

• Unpinned compute region: CPU computes attention in-place
• Pinned load region: GPU streams via full-bandwidth DMA
• D-window: Decode KV staged on GPU (zero H2D cost), evicted in batches of $D = 256$ entries
• Ping-pong region: Overwrite-reuse buffer for streamed KV

Batched Write-Time Routing

Prefill KV routes to final destination in one shot. Decode tokens are staged in the D-window, then evicted whole into unpinned or pinned regions based on the roofline policy. The D2H write-back uses idle PCIe full-duplex bandwidth, hiding within the expert phase.

Roofline Performance Model

The CPU is bandwidth-bound (attention's arithmetic intensity of 0.5 MAC/byte sits two orders below the compute–bandwidth balance point). With per-token costs $A$ (CPU read) and $B$ (GPU transfer), and fixed overheads $C_{CPU}$, $C_{GPU}$:

$$T_{CPU}(r) = r \cdot A \cdot L + C_{CPU}, \quad T_{GPU}(r) = (1 - r) \cdot B \cdot L + C_{GPU} \tag{1}$$

Load balancing requires $T_{CPU} = T_{GPU}$, yielding:

$$r(L) = \min\left(1, \max\left(0, \frac{B}{A+B} + \frac{\Delta}{(A+B) \cdot L}\right)\right) \tag{2}$$

where $\Delta = C_{GPU} - C_{CPU}$, $r$ is the CPU's share of the sequence, and $L$ is the placed sequence length.

Sequence-Dimension Splitting

The split is along the sequence axis (not head dimension) because:

• Sequence axis makes the split ratio continuous
• Head-wise split is quantized by KV-head count (coarse under GQA, ill-defined under MLA's shared latent)

Both engines compute over all heads on disjoint sequence ranges, merging via log-sum-exp (LSE) rescaling.

Empirical Validation / Results

Experimental Setup

• Hardware: A100 PCIe (Gen4 ×16, 32 GB VRAM budget), V100-32GB (Gen3 ×16), Intel Xeon Gold 5318Y (24 cores)
• Models: DeepSeek-V2-Lite (MLA), Qwen3-30B-A3B (GQA), Mixtral-8×7B (GQA), DeepSeek-V4-Flash-Int8 (CSA/HCA sparsity)
• Baselines: Four reproduced attention baselines spanning the placement spectrum (SolidAttention, ScoutAttention, HybridGen, FastDecode) plus three mainstream frameworks (vLLM, SGLang, KTransformers)

Prefill Results

• Write-time routing is free: Route vs. Ideal differ by ≤1 tok/s
• WriteScope completes all configurations; frameworks OOM at bs≥16
• Mixtral: WriteScope 435–538 tok/s vs. vLLM 78–80 tok/s (5.4–6.9× gap)

Decode Throughput (≥8K regime)

Speedup amplifies with sequence length: DS2 bs=32, 1.40× at 2K → 2.49× at 32K (over SolidAttention), 1.28× → 4.42× (over ScoutAttention).

Ablation Study (DS2, normalized throughput)

Memory Footprint (seq=32K, bs=32)

GPU residency is <1% of full KV at deployed $D=256$.

Multi-User Concurrency

Runtime adaptation handles contention: fixed-r inflates latency 2.6× over solo at bs4/8K, while adaptive reduces it by 57% (1010→436 ms).

Theoretical and Practical Implications

Theoretical Contributions

1. Root-cause attribution: The absence of dynamic load balancing in KV-Cache systems is traced to monolithic storage semantics—physical ownership must be in the data format, not a runtime mechanism
2. Write-time principle: Generalizes beyond KV-Cache—any deterministic-consumption data structure can benefit from format-embedded physical partitioning
3. Roofline-guided scheduling: The optimal CPU share drifts with workload; a portable model with online feedback tracks this drift without statistical convergence delays

Practical Implications

• Single-GPU MoE inference becomes practical: Long-context MoE inference no longer requires multi-GPU servers—it becomes viable on privacy-favored single-GPU machines
• Composition with sparsity: Block-aligned format supports native sparse attention (DeepSeek-V4 CSA) via pipelined sparse-selection loading with no extra overhead
• Capacity decoupling: GPU VRAM is decoupled from sequence length—GPU residency grows with $D \times bs$, making CPU memory the sole capacity bound
• Multi-user robustness: The same roofline loop handles contention from concurrent requests, adapting to congestion on shared CPU and PCIe

Conclusion

WriteScope demonstrates that dynamic CPU–GPU load balancing for KV-Cache is achievable when physical ownership is embedded in the data format. Key takeaways:

1. Physical partitioning at write time eliminates the reorganization costs that made dynamic balancing infeasible
2. Roofline-guided scheduling with online feedback tracks the drifting balance point as workloads evolve
3. Practical impact: 1.5×–2.5× speedups (A100) and 1.4×–1.7× (V100) over baselines, 1M-token aggregate inference on 32 GB VRAM, and successful composition with model-native sparsity

Future directions: Extending to joint scheduling of expert prefetching and attention placement under a unified roofline policy; validating on Gen5 platforms (RTX 5090) with higher PCIe bandwidth; exploring multi-user concurrency as a first-class deployment mode.