# Physically Partitioned KVCache Format for CPU-GPU Load Balancing in MoE Inference (InplaceKVCache/WriteScope)

> WriteScope embeds CPU-GPU KV-Cache placement into the data format at write time, achieving 1.5-2.5x speedups for single-GPU long-context MoE inference.

- **Source:** [arXiv](https://arxiv.org/abs/2609.14507)
- **Published:** 2026-09-19
- **Permalink:** https://picx.dev/p/dgY7Vv
- **Whiteboard:** https://picx.dev/p/dgY7Vv/image

## Summary

## Summary (Overview)

- **Core Problem**: Single-GPU long-context inference with Mixture-of-Experts (MoE) models requires spilling the Key-Value Cache (KV-Cache) to CPU memory. The spilled KV serves two opposing purposes—loading to GPU for attention vs. computing in-place on CPU—which demand conflicting physical states (pinned vs. unpinned, contiguous vs. strided).
- **Key Insight**: The **write-time principle**—each byte's physical residency is fixed at write time—makes load balancing a property of the data format rather than a runtime mechanism, eliminating post-hoc data reorganization costs.
- **Main Contribution**: **InplaceKVCache**, the first KV-Cache abstraction with a four-region layout along two orthogonal dimensions (device affinity × access pattern), enabling dynamic CPU–GPU load balancing without moving data after placement.
- **Results**: On three MoE models with 32 GB VRAM, WriteScope achieves geometric-mean speedups of 1.5×–2.5× (A100) and 1.4×–1.7× (V100) over four baselines at ≥8K context, supporting 1M-token aggregate inference where vLLM, SGLang, and KTransformers fail.
- **Scheduling**: A portable roofline performance model determines the optimal CPU share as sequence length evolves, with online feedback tracking CPU cost drift.

## Introduction and Theoretical Foundation

### Background and Motivation

MoE architectures are natural fits for budget-constrained, privacy-first single-GPU deployment due to sparse activation. However, long-context inference demands far exceed single-GPU capacity: Qwen3-30B-A3B requires 78 GB for 128K-token context, well beyond 32 GB VRAM. The KV-Cache must spill to CPU memory.

### The Core Tension

Existing CPU–GPU hybrid systems adopt one of two extremes:
- **GPU-centric** (FlexGen, SolidAttention): Load all KV back to GPU—bottlenecked by PCIe bandwidth
- **CPU-centric** (NEO, ScoutAttention): Compute all KV in-place on CPU—bottlenecked by CPU compute

Both rely on **AF overlap** (asynchronous pre-execution of next layer's attention during FFN phase), which works for dense models where FFN runs entirely on GPU. However, MoE invalidates this: spilled expert parameters reside on CPU, saturating both CPU and PCIe during FFN, turning overlap into contention. Measured impact: attention latency rises from 33.7 ms to 74.2 ms, KV transfer bandwidth drops from 25.4 GB/s to 14.9 GB/s.

### Why Existing Abstractions Fail

KV-Cache abstractions offer only storage semantics over a monolithic object of a single physical state. The two uses impose opposing requirements:

- **GPU loading**: requires pinned memory and logical contiguity (for full DMA bandwidth)
- **CPU computation**: requires unpinned memory and per-head organization

Preparing buffers for both uses costs 3.5 ms → 185.9 ms per layer per step as context grows from 2K to 64K, negating load-balancing gains.

## Methodology

### The Write-Time Principle

> "A byte has exactly one final physical residency, and that residency is determined at write time."

This fits KV-Cache because its consumption is deterministic (unlike stochastic expert activation, which requires schedule-time arbitration).

### InplaceKVCache Four-Region Layout

The cache partitions space along two orthogonal dimensions:

| Dimension | CPU | GPU |
|-----------|-----|-----|
| **Compute** | Unpinned compute region | Ping-pong compute region |
| **Transfer** | Pinned load region | D-window offload region |

- **Unpinned compute region**: CPU computes attention in-place
- **Pinned load region**: GPU streams via full-bandwidth DMA
- **D-window**: Decode KV staged on GPU (zero H2D cost), evicted in batches of $D = 256$ entries
- **Ping-pong region**: Overwrite-reuse buffer for streamed KV

### Batched Write-Time Routing

Prefill KV routes to final destination in one shot. Decode tokens are staged in the D-window, then evicted whole into unpinned or pinned regions based on the roofline policy. The D2H write-back uses idle PCIe full-duplex bandwidth, hiding within the expert phase.

### Roofline Performance Model

The CPU is bandwidth-bound (attention's arithmetic intensity of 0.5 MAC/byte sits two orders below the compute–bandwidth balance point). With per-token costs $A$ (CPU read) and $B$ (GPU transfer), and fixed overheads $C_{CPU}$, $C_{GPU}$:

$$T_{CPU}(r) = r \cdot A \cdot L + C_{CPU}, \quad T_{GPU}(r) = (1 - r) \cdot B \cdot L + C_{GPU} \tag{1}$$

Load balancing requires $T_{CPU} = T_{GPU}$, yielding:

$$r(L) = \min\left(1, \max\left(0, \frac{B}{A+B} + \frac{\Delta}{(A+B) \cdot L}\right)\right) \tag{2}$$

where $\Delta = C_{GPU} - C_{CPU}$, $r$ is the CPU's share of the sequence, and $L$ is the placed sequence length.

### Sequence-Dimension Splitting

The split is along the sequence axis (not head dimension) because:
- Sequence axis makes the split ratio continuous
- Head-wise split is quantized by KV-head count (coarse under GQA, ill-defined under MLA's shared latent)

Both engines compute over all heads on disjoint sequence ranges, merging via log-sum-exp (LSE) rescaling.

## Empirical Validation / Results

### Experimental Setup

- **Hardware**: A100 PCIe (Gen4 ×16, 32 GB VRAM budget), V100-32GB (Gen3 ×16), Intel Xeon Gold 5318Y (24 cores)
- **Models**: DeepSeek-V2-Lite (MLA), Qwen3-30B-A3B (GQA), Mixtral-8×7B (GQA), DeepSeek-V4-Flash-Int8 (CSA/HCA sparsity)
- **Baselines**: Four reproduced attention baselines spanning the placement spectrum (SolidAttention, ScoutAttention, HybridGen, FastDecode) plus three mainstream frameworks (vLLM, SGLang, KTransformers)

### Prefill Results

- Write-time routing is free: Route vs. Ideal differ by ≤1 tok/s
- WriteScope completes all configurations; frameworks OOM at bs≥16
- Mixtral: WriteScope 435–538 tok/s vs. vLLM 78–80 tok/s (5.4–6.9× gap)

### Decode Throughput (≥8K regime)

| Platform | SolidAttention | ScoutAttention | HybridGen | FastDecode |
|----------|---------------|----------------|-----------|------------|
| **A100** | 1.52× | 2.22× | 2.48× | 2.51× |
| **V100** | 1.55× | 1.43× | 1.67× | 1.71× |

Speedup amplifies with sequence length: DS2 bs=32, 1.40× at 2K → 2.49× at 32K (over SolidAttention), 1.28× → 4.42× (over ScoutAttention).

### Ablation Study (DS2, normalized throughput)

| Replaced Component | Normalized Throughput |
|-------------------|----------------------|
| Custom AVX-512 kernel → IPEX oneDNN | 0.84–1.00 |
| Dynamic ratio → fixed $r=0.50$ | 0.89–0.96 |
| Contiguous DMA → strided DMA | 0.47–0.90 |
| Ping-pong reuse → per-step rebuild | 0.53–0.87 |
| Pinned buffer → unpinned | 0.62–0.94 |
| InplaceKVCache → DynamicCache | **0.26–0.79** |

### Memory Footprint (seq=32K, bs=32)

| Model | Ping-pong | D-window | Unpinned | Pinned |
|-------|-----------|----------|----------|--------|
| DS2 | 37.8 MB | 255 MB | 15.95 GB | 16.66 GB |
| Qwen3 | 67.1 MB | 805 MB | 61.95 GB | 41.13 GB |
| Mixtral | 134.2 MB | 1074 MB | 103.08 GB | 34.36 GB |

GPU residency is <1% of full KV at deployed $D=256$.

### Multi-User Concurrency

Runtime adaptation handles contention: fixed-r inflates latency 2.6× over solo at bs4/8K, while adaptive reduces it by 57% (1010→436 ms).

## Theoretical and Practical Implications

### Theoretical Contributions

1. **Root-cause attribution**: The absence of dynamic load balancing in KV-Cache systems is traced to monolithic storage semantics—physical ownership must be in the data format, not a runtime mechanism
2. **Write-time principle**: Generalizes beyond KV-Cache—any deterministic-consumption data structure can benefit from format-embedded physical partitioning
3. **Roofline-guided scheduling**: The optimal CPU share drifts with workload; a portable model with online feedback tracks this drift without statistical convergence delays

### Practical Implications

- **Single-GPU MoE inference becomes practical**: Long-context MoE inference no longer requires multi-GPU servers—it becomes viable on privacy-favored single-GPU machines
- **Composition with sparsity**: Block-aligned format supports native sparse attention (DeepSeek-V4 CSA) via pipelined sparse-selection loading with no extra overhead
- **Capacity decoupling**: GPU VRAM is decoupled from sequence length—GPU residency grows with $D \times bs$, making CPU memory the sole capacity bound
- **Multi-user robustness**: The same roofline loop handles contention from concurrent requests, adapting to congestion on shared CPU and PCIe

## Conclusion

WriteScope demonstrates that dynamic CPU–GPU load balancing for KV-Cache is achievable when physical ownership is embedded in the data format. Key takeaways:

1. **Physical partitioning at write time** eliminates the reorganization costs that made dynamic balancing infeasible
2. **Roofline-guided scheduling** with online feedback tracks the drifting balance point as workloads evolve
3. **Practical impact**: 1.5×–2.5× speedups (A100) and 1.4×–1.7× (V100) over baselines, 1M-token aggregate inference on 32 GB VRAM, and successful composition with model-native sparsity

**Future directions**: Extending to joint scheduling of expert prefetching and attention placement under a unified roofline policy; validating on Gen5 platforms (RTX 5090) with higher PCIe bandwidth; exploring multi-user concurrency as a first-class deployment mode.

---

_Markdown view of https://picx.dev/p/dgY7Vv, served by PicX — AI-generated visual whiteboard summaries of research papers._