Summary (Overview)

• OasisKV is a memory-centric LLM inference system that decouples full KV-cache storage from HBM during decoding, keeping only the most relevant KV entries in HBM for attention computation.
• The system leverages lookahead tokens from speculative decoding (SD) to accurately predict future important KV blocks one step ahead, achieving >98.2% per-layer agreement with true next-token attention patterns.
• OasisKV implements a fully asynchronous prefetch pipeline with three stages (top-K prediction, KV selection, KV transfer) running on separate CUDA streams, hiding of-GPU memory transfer behind decode compute.
• The system achieves 1.69× throughput improvement over dense vLLM on real reasoning workloads with only 0.1 points of accuracy loss, and up to 2.1× on multi-GPU long-context serving.
• Under prefill-decode (PD) disaggregation, OasisKV's remote partial fetching reduces per-request KV admission by 6.5–9.7× and decode-node host memory by 2.2–2.6× compared to full KV transfer.

Introduction and Theoretical Foundation

The Memory Wall Problem

Large language model (LLM) inference is increasingly constrained by memory rather than compute. As agentic workloads grow (web-use, computer-use, coding agents), context lengths have increased over 10× compared to chatbot-era workloads. The key-value (KV) cache dominates both memory footprint and memory traffic during decode.

The KV footprint for a batch of $B$ requests with context length $L$ is:

$$M_{\mathrm{KV}}(B, L) = B \cdot L \cdot N_{\text{layer}} \cdot 2N_{\mathrm{kv}}d_h \cdot s \tag{1}$$

The maximum decode batch size allowed by a GPU KV-cache budget $M_{\mathrm{HBM}}$ is:

$$B_{\mathrm{max}} = \left\lfloor \frac{M_{\mathrm{HBM}}}{L \cdot N_{\mathrm{layer}} \cdot 2N_{\mathrm{kv}}d_h \cdot s} \right\rfloor \tag{2}$$

The linear dependence on $L$ makes long-context decoding fundamentally capacity-constrained. For a 32B-class GQA model with FP16/BF16 KV cache, per-token KV footprint is 256 KiB. At 32.7K average context tokens (production coding-agent trace), a single request requires ~8.6 GB of KV cache, limiting a nominal 80 GB HBM to only 9 concurrent requests.

Existing Approaches and Their Limitations

Three key KV-sparsity-based approaches exist:

1. Sparse attention — reduces KV reads but does not reduce HBM capacity required
2. KV retrieval — stages only active subset in HBM but places of-GPU transfers on the decode critical path
3. KV prefetch — overlaps memory fetching with decode compute but is pinned against CPU-GPU IO bandwidth roofs

The fundamental constraint is the per-step byte budget:

$$C_{\mathrm{token}} = B_{\mathrm{link}} \cdot T_{\mathrm{decode}} \tag{3}$$

where $B_{\mathrm{link}}$ is effective inter-tier bandwidth and $T_{\mathrm{decode}}$ is available compute time. For Qwen3-8B on H100 with 2K active context, a decode step takes ≈17 ms, so a ≈64 GB/s PCIe link admits only ≈118 tokens of newly active KV per request per step.

Methodology

Look-ahead Attention

OasisKV leverages draft tokens from speculative decoding (EAGLE-3) to predict future KV access patterns. The key insight is that adjacent decoding steps exhibit strong temporal locality in block importance. By propagating the draft token over the normal token's sparse KV, the predicted top-K set agrees with the true next-token query's set above 98.2% in every layer (98.74% average).

The architecture consists of:

• Foreground: Processes normal and draft queries together over one resident KV working set
• Background: Draft query scans compressed key summaries (Quest-style min/max pooling) to rank all logical blocks independently for every KV head
• Compressed-key cache: Stores two summary vectors per block (coordinate-wise min/max), costing only 1/16 of full KV-cache size

Asynchronous Prefetch Pipeline

For a model with $N_{\mathrm{layer}}$ layers, the average interval between consecutive prefetch tasks is $\Delta_{\mathrm{prefetch}} = T_{\mathrm{step}} / N_{\mathrm{layer}}$. The pipeline's steady-state completion interval is:

$$T_{\mathrm{pipe}} = \max\{T_{\mathrm{pred}}^l, T_{\mathrm{select}}^l, T_{\mathrm{transfer}}^l\} \leq \Delta_{\mathrm{prefetch}} \tag{4}$$

Three persistent background workers run on separate CUDA streams:

• Top-K prediction: Computes next step's top-K blocks
• KV selection: Compares predicted set with resident blocks, builds transfer plan
• KV transfer: Moves planned blocks from CPU to GPU

CUDA events preserve dependencies within each layer, allowing layer-local synchronization.

Delta Selection and Capped Eviction

To bound PCIe traffic, OasisKV uses a capped eviction policy:

1. Intersect predicted and resident sets (resident blocks generate no traffic)
2. Rank resident blocks outside predicted set by last-selected step
3. Pair each admitted nonresident block with one eviction target (LRU-style)
4. Admit at most $C$ pairs per decoding step per KV head

This bounds transfer to $C \cdot H$ block-head entries across $H$ KV heads regardless of prediction drift.

Head-wise Memory Mapping

OasisKV adds a head-wise logical-to-logical mapping layer above existing GPU and CPU page tables. Each KV head maintains different sparse block sets, and the GPU pool holds a bounded working set while the CPU pool retains the full KV cache. Under tensor parallelism, each GPU updates mappings for its local KV heads independently without cross-GPU synchronization.

Remote Partial Fetching (PD Disaggregation)

For disaggregated serving:

• Partial transfer at admission: Sends only first-step selected KV blocks, compressed-key cache, and draft KV state — not the full KV cache
• Network prefetch at decode: Handles cache misses by fetching blocks from prefill node's staging pool, aggregated by layer across requests

Empirical Validation / Results

System Performance

Single-GPU (Qwen3-8B, H100):

• Up to 2.1× dense throughput (1,398 vs. 676 tok/s at 16K context)
• At moderate concurrency (16), improves both TPOT and throughput simultaneously
• Bounded 2,048-token working set supports 90–95 concurrent requests vs. 22 under dense

Multi-GPU (Qwen3-235B-A22B, TP8):

• Up to 1.9× throughput (1,102 tok/s)
• Overtakes dense at batch size 32 (16K) and 16 (32K)

PD Disaggregation:

• 2.1–2.3× throughput over dense (1,204 vs. 550 tok/s at 24K)
• RPF reduces per-request host memory to 1.54 GiB (vs. 3.38 GiB full transfer), 2.2× less

Accuracy Results

Table 1: Accuracy under 2,048-token KV budget (Δ vs. own stack's full attention)

Fetch-Cap Ablation

Table 2: Fetch-cap ablation on Qwen3-8B (AIME24, LRU eviction). Dense attention scores 76.04 / 90.00.

Key finding: Throughput is bounded by PCIe bandwidth, not attention compute. As the cap loosens, throughput collapses 2.6× while accuracy rises only modestly.

Remote Partial Fetching Benefits

• Admission traffic: 0.52 GiB (24K) and 0.46 GiB (32K) per request vs. 3.37 and 4.50 GiB full transfer — 6.5× and 9.7× reductions
• End-to-end traffic: 1.33× and 1.50× reduction when including decode-time drift
• Bandwidth distribution: RPF distributes traffic evenly (1–2% link idle time) vs. full transfer's bursty pattern (60% idle, 5.3 GB/s bursts)
• TTFT benefit: With 90% prefix-cache hit rate, RPF reduces TTFT by 2.0× at 24K and 2.2× at 32K over 100 Gbps link

Theoretical and Practical Implications

Design Requirements Established

1. Accurate lookahead, prefetched off the critical path: Prediction must be training-free, cheap, and accurate. Previous-token proxies recover true top-20 blocks unreliably with high layer variance; draft-token lookahead achieves >98% agreement.

2. Efficient sparse serving in production engines: Requires bounded, head-wise working sets on the paged KV pool, dense and sparse requests batched together, and per-step data movement within the decode-time budget $C_{\mathrm{token}}$.

3. Sparse staging across network and host tiers: Avoid materializing full KV cache in decode-node DRAM; apply sparsity to blocks transferred over network and materialized in host memory.

Key Insights

• Speculative decoding integration: OasisKV is inherently compatible with SD, reusing draft tokens that would otherwise be discarded. This opens a new design space for model-system co-design.
• Sparsity-accuracy tradeoff is manageable: Throughput swings 2.6× across fetch-cap settings while accuracy moves only 3 points, within run-to-run variance of dense.
• Memory-centric design: The system treats of-GPU prefetch overlap, lookahead-driven sparsity prediction, and sparsity-accuracy tradeoff as a joint systems optimization problem.

Conclusion

OasisKV demonstrates that KV-cache sparsity can be turned into throughput gain in production-grade serving systems. By reusing speculative decoding's draft tokens as a lookahead signal, the system achieves accurate prediction of future KV access patterns without additional training or model-specific modifications.

Key takeaways:

• Lookahead prediction maintains accuracy within 0.7 points of full attention under a 2,048-token KV budget
• 1.69× throughput gain over dense vLLM on reasoning workloads with only 0.1 points accuracy loss
• Up to 2.1× on multi-GPU long-context serving
• ~2× dense throughput under PD disaggregation with 6.5–9.7× less KV admission and 2.2–2.6× less host memory

Future directions: Enabling speculative decoding jointly with draft-based prefetching (allowing accepted draft tokens to amortize the draft overhead), extending the prefill node's host DRAM to remote memory servers or SSD-based storage tiers, and exploring new model-system co-design opportunities spanning KV-cache sparsity, long-context capability, memory hardware scaling, and system performance.