# OasisKV: Scaling In-Decode KV Cache Beyond HBM with Lookahead Sparse Prefetching

> OasisKV uses speculative decoding's draft tokens to predict future KV-cache access, enabling asynchronous prefetching that boosts LLM inference throughput up to 2.1x with negligible accuracy loss.

- **Source:** [arXiv](https://arxiv.org/abs/2608.08097)
- **Published:** 2026-08-22
- **Permalink:** https://picx.dev/p/Kt71Qi
- **Whiteboard:** https://picx.dev/p/Kt71Qi/image

## Summary

## Summary (Overview)

- **OasisKV** is a memory-centric LLM inference system that decouples full KV-cache storage from HBM during decoding, keeping only the most relevant KV entries in HBM for attention computation.
- The system leverages **lookahead tokens from speculative decoding (SD)** to accurately predict future important KV blocks one step ahead, achieving >98.2% per-layer agreement with true next-token attention patterns.
- OasisKV implements a **fully asynchronous prefetch pipeline** with three stages (top-K prediction, KV selection, KV transfer) running on separate CUDA streams, hiding of-GPU memory transfer behind decode compute.
- The system achieves **1.69× throughput improvement** over dense vLLM on real reasoning workloads with only 0.1 points of accuracy loss, and up to **2.1× on multi-GPU long-context serving**.
- Under prefill-decode (PD) disaggregation, OasisKV's **remote partial fetching** reduces per-request KV admission by 6.5–9.7× and decode-node host memory by 2.2–2.6× compared to full KV transfer.

## Introduction and Theoretical Foundation

### The Memory Wall Problem

Large language model (LLM) inference is increasingly constrained by memory rather than compute. As agentic workloads grow (web-use, computer-use, coding agents), context lengths have increased over 10× compared to chatbot-era workloads. The key-value (KV) cache dominates both memory footprint and memory traffic during decode.

The KV footprint for a batch of $B$ requests with context length $L$ is:

$$M_{\mathrm{KV}}(B, L) = B \cdot L \cdot N_{\text{layer}} \cdot 2N_{\mathrm{kv}}d_h \cdot s \tag{1}$$

The maximum decode batch size allowed by a GPU KV-cache budget $M_{\mathrm{HBM}}$ is:

$$B_{\mathrm{max}} = \left\lfloor \frac{M_{\mathrm{HBM}}}{L \cdot N_{\mathrm{layer}} \cdot 2N_{\mathrm{kv}}d_h \cdot s} \right\rfloor \tag{2}$$

The linear dependence on $L$ makes long-context decoding fundamentally capacity-constrained. For a 32B-class GQA model with FP16/BF16 KV cache, per-token KV footprint is 256 KiB. At 32.7K average context tokens (production coding-agent trace), a single request requires ~8.6 GB of KV cache, limiting a nominal 80 GB HBM to only 9 concurrent requests.

### Existing Approaches and Their Limitations

Three key KV-sparsity-based approaches exist:

1. **Sparse attention** — reduces KV reads but does not reduce HBM capacity required
2. **KV retrieval** — stages only active subset in HBM but places of-GPU transfers on the decode critical path
3. **KV prefetch** — overlaps memory fetching with decode compute but is pinned against CPU-GPU IO bandwidth roofs

The fundamental constraint is the per-step byte budget:

$$C_{\mathrm{token}} = B_{\mathrm{link}} \cdot T_{\mathrm{decode}} \tag{3}$$

where $B_{\mathrm{link}}$ is effective inter-tier bandwidth and $T_{\mathrm{decode}}$ is available compute time. For Qwen3-8B on H100 with 2K active context, a decode step takes ≈17 ms, so a ≈64 GB/s PCIe link admits only ≈118 tokens of newly active KV per request per step.

## Methodology

### Look-ahead Attention

OasisKV leverages draft tokens from speculative decoding (EAGLE-3) to predict future KV access patterns. The key insight is that **adjacent decoding steps exhibit strong temporal locality** in block importance. By propagating the draft token over the normal token's sparse KV, the predicted top-K set agrees with the true next-token query's set above 98.2% in every layer (98.74% average).

The architecture consists of:
- **Foreground**: Processes normal and draft queries together over one resident KV working set
- **Background**: Draft query scans compressed key summaries (Quest-style min/max pooling) to rank all logical blocks independently for every KV head
- **Compressed-key cache**: Stores two summary vectors per block (coordinate-wise min/max), costing only 1/16 of full KV-cache size

### Asynchronous Prefetch Pipeline

For a model with $N_{\mathrm{layer}}$ layers, the average interval between consecutive prefetch tasks is $\Delta_{\mathrm{prefetch}} = T_{\mathrm{step}} / N_{\mathrm{layer}}$. The pipeline's steady-state completion interval is:

$$T_{\mathrm{pipe}} = \max\{T_{\mathrm{pred}}^l, T_{\mathrm{select}}^l, T_{\mathrm{transfer}}^l\} \leq \Delta_{\mathrm{prefetch}} \tag{4}$$

Three persistent background workers run on separate CUDA streams:
- **Top-K prediction**: Computes next step's top-K blocks
- **KV selection**: Compares predicted set with resident blocks, builds transfer plan
- **KV transfer**: Moves planned blocks from CPU to GPU

CUDA events preserve dependencies within each layer, allowing layer-local synchronization.

### Delta Selection and Capped Eviction

To bound PCIe traffic, OasisKV uses a **capped eviction policy**:
1. Intersect predicted and resident sets (resident blocks generate no traffic)
2. Rank resident blocks outside predicted set by last-selected step
3. Pair each admitted nonresident block with one eviction target (LRU-style)
4. Admit at most $C$ pairs per decoding step per KV head

This bounds transfer to $C \cdot H$ block-head entries across $H$ KV heads regardless of prediction drift.

### Head-wise Memory Mapping

OasisKV adds a **head-wise logical-to-logical mapping layer** above existing GPU and CPU page tables. Each KV head maintains different sparse block sets, and the GPU pool holds a bounded working set while the CPU pool retains the full KV cache. Under tensor parallelism, each GPU updates mappings for its local KV heads independently without cross-GPU synchronization.

### Remote Partial Fetching (PD Disaggregation)

For disaggregated serving:
- **Partial transfer at admission**: Sends only first-step selected KV blocks, compressed-key cache, and draft KV state — not the full KV cache
- **Network prefetch at decode**: Handles cache misses by fetching blocks from prefill node's staging pool, aggregated by layer across requests

## Empirical Validation / Results

### System Performance

**Single-GPU (Qwen3-8B, H100):**
- Up to 2.1× dense throughput (1,398 vs. 676 tok/s at 16K context)
- At moderate concurrency (16), improves both TPOT and throughput simultaneously
- Bounded 2,048-token working set supports 90–95 concurrent requests vs. 22 under dense

**Multi-GPU (Qwen3-235B-A22B, TP8):**
- Up to 1.9× throughput (1,102 tok/s)
- Overtakes dense at batch size 32 (16K) and 16 (32K)

**PD Disaggregation:**
- 2.1–2.3× throughput over dense (1,204 vs. 550 tok/s at 24K)
- RPF reduces per-request host memory to 1.54 GiB (vs. 3.38 GiB full transfer), 2.2× less

### Accuracy Results

Table 1: Accuracy under 2,048-token KV budget (Δ vs. own stack's full attention)

| Dataset / Subset | Metric | Full (HF) | Quest | Δ | FreeKV | Δ | Full (vLLM) | Ours | Δ |
|---|---|---|---|---|---|---|---|---|---|
| **Long input — Llama-3.1-8B, LongBench v2** | | | | | | | | | |
| Overall | | 29.62 | 29.42 | -0.20 | 29.03 | -0.59 | 30.23 | 29.62 | -0.61 |
| **Long input — Qwen3-8B, LongBench v2** | | | | | | | | | |
| Overall | | 32.21 | 31.61 | -0.60 | 31.01 | -1.20 | 33.60 | 33.20 | -0.40 |
| **Long output — Qwen3-8B, reasoning** | | | | | | | | | |
| Overall | pass@k | 81.41 | 78.86 | -2.56 | 77.91 | -3.50 | 78.84 | 78.18 | -0.66 |
| | avg@k | 69.48 | 66.65 | -2.83 | 66.85 | -2.63 | 67.63 | 67.28 | -0.35 |
| AIME24 | pass@8 | 86.67 | 86.67 | 0.00 | 80.00 | -6.67 | 83.33 | 83.33 | 0.00 |
| AIME25 | avg@8 | 70.83 | 64.17 | -6.66 | 68.75 | -2.08 | 65.42 | 67.97 | **+2.55** |
| GPQA-Diamond | avg@4 | 60.10 | 60.36 | +0.26 | 59.72 | -0.38 | 58.30 | 57.20 | -1.10 |

### Fetch-Cap Ablation

Table 2: Fetch-cap ablation on Qwen3-8B (AIME24, LRU eviction). Dense attention scores 76.04 / 90.00.

| Fetch Ratio | Fetch (GB/step) | BW (GB/s) | TPS (tok/s) | AIME24 avg@32 | AIME24 pass@32 |
|---|---|---|---|---|---|
| 0.01 | 0.30 | 5.0 | 2,178 | 74.90 | 90.00 |
| 0.02 | 0.60 | 9.8 | 2,066 | 75.10 | 90.00 |
| **0.05** | **1.49** | **23.8** | **2,083** | **75.94** | **86.67** |
| 0.10 | 2.87 | 31.4 | 1,421 | 76.77 | 86.67 |
| 0.20 | 4.34 | 34.0 | 1,035 | 77.40 | 93.33 |
| Fetch all | 5.05 | 33.5 | 824 | 76.46 | 86.67 |

Key finding: **Throughput is bounded by PCIe bandwidth, not attention compute**. As the cap loosens, throughput collapses 2.6× while accuracy rises only modestly.

### Remote Partial Fetching Benefits

- **Admission traffic**: 0.52 GiB (24K) and 0.46 GiB (32K) per request vs. 3.37 and 4.50 GiB full transfer — **6.5× and 9.7× reductions**
- **End-to-end traffic**: 1.33× and 1.50× reduction when including decode-time drift
- **Bandwidth distribution**: RPF distributes traffic evenly (1–2% link idle time) vs. full transfer's bursty pattern (60% idle, 5.3 GB/s bursts)
- **TTFT benefit**: With 90% prefix-cache hit rate, RPF reduces TTFT by 2.0× at 24K and 2.2× at 32K over 100 Gbps link

## Theoretical and Practical Implications

### Design Requirements Established

1. **Accurate lookahead, prefetched off the critical path**: Prediction must be training-free, cheap, and accurate. Previous-token proxies recover true top-20 blocks unreliably with high layer variance; draft-token lookahead achieves >98% agreement.

2. **Efficient sparse serving in production engines**: Requires bounded, head-wise working sets on the paged KV pool, dense and sparse requests batched together, and per-step data movement within the decode-time budget $C_{\mathrm{token}}$.

3. **Sparse staging across network and host tiers**: Avoid materializing full KV cache in decode-node DRAM; apply sparsity to blocks transferred over network and materialized in host memory.

### Key Insights

- **Speculative decoding integration**: OasisKV is inherently compatible with SD, reusing draft tokens that would otherwise be discarded. This opens a new design space for model-system co-design.
- **Sparsity-accuracy tradeoff is manageable**: Throughput swings 2.6× across fetch-cap settings while accuracy moves only 3 points, within run-to-run variance of dense.
- **Memory-centric design**: The system treats of-GPU prefetch overlap, lookahead-driven sparsity prediction, and sparsity-accuracy tradeoff as a joint systems optimization problem.

## Conclusion

OasisKV demonstrates that **KV-cache sparsity can be turned into throughput gain** in production-grade serving systems. By reusing speculative decoding's draft tokens as a lookahead signal, the system achieves accurate prediction of future KV access patterns without additional training or model-specific modifications.

Key takeaways:
- Lookahead prediction maintains accuracy within 0.7 points of full attention under a 2,048-token KV budget
- 1.69× throughput gain over dense vLLM on reasoning workloads with only 0.1 points accuracy loss
- Up to 2.1× on multi-GPU long-context serving
- ~2× dense throughput under PD disaggregation with 6.5–9.7× less KV admission and 2.2–2.6× less host memory

**Future directions**: Enabling speculative decoding jointly with draft-based prefetching (allowing accepted draft tokens to amortize the draft overhead), extending the prefill node's host DRAM to remote memory servers or SSD-based storage tiers, and exploring new model-system co-design opportunities spanning KV-cache sparsity, long-context capability, memory hardware scaling, and system performance.

---

_Markdown view of https://picx.dev/p/Kt71Qi, served by PicX — AI-generated visual whiteboard summaries of research papers._