Summary (Overview)

• Problem: MoE models serving multi-turn agentic workloads face a "prefix-cache cliff" – reusable KV-cache prefixes grow across turns, but static HBM partitioning between expert weights and KV cache prevents reallocating memory, causing TTFT to spike sharply when cache capacity is exhausted.
• Key insight: Expert weights occupy most HBM (72 GiB per GPU for Qwen3-Next-80B on H100) despite only ~3B of 80B parameters being active per token; this memory can be dynamically repurposed for KV cache when needed.
• Solution: VAMP (Value-Aware Memory Partitioning) uses a three-way cost model comparing expert offloading, KV-cache eviction, and request preemption penalties, then uses CUDA Virtual Memory Management (VMM) to remap physical HBM pages between expert weights and KV cache at runtime.
• Results: On a 2,103-turn recorded SWE-bench replay, VAMP-15 reduces TTFT p90 from 26.1s to 1.10s (23.6×), increases request throughput by 20.7%, and raises prefix-cache hit rate from 0.19% to 89.8%, at a cost of +31.1% TPOT.
• Impact: VAMP expands the per-GPU KV-cache pool by up to 2.7× on H100, moving the capacity boundary (onset of the prefix-cache cliff) to significantly higher concurrency across both TP2 and TP4 configurations.

Introduction and Theoretical Foundation

Background

Large language model serving systems face a fundamental tension: model weights must remain resident in GPU HBM while KV-cache footprints grow linearly with batch size and context length. Traditional serving systems (e.g., vLLM) treat model placement and KV-cache management as separate decisions, relying on two implicit assumptions:

1. Model weights occupy only a modest fraction of HBM, leaving ample room for KV cache
2. KV-cache requirements remain within the preallocated pool bounds

Why These Assumptions Fail

Assumption 1 fails for MoE architectures: Mixture-of-Experts models structurally decouple model capacity from per-token compute cost. The total parameter count grows aggressively while active parameters stay constant:

Assumption 2 fails for multi-turn agentic workloads: Agentic trajectories average 157 turns and 32.7k tokens of context, with some runs exceeding 1M tokens. KV-cache utilization stays at 80–100%, causing repeated evictions and re-prefills.

The Prefix-Cache Cliff

When the reusable prefix-cache footprint exceeds KV-cache capacity:

• Cache eviction reduces prefix-cache hit rate
• Re-prefills inflate TTFT and consume serving capacity
• Sustained memory pressure leads to further preemptions
• This creates a threshold effect – TTFT remains stable until pool saturation, then rises sharply

This is not gradual degradation but an abrupt, threshold-driven collapse.

Methodology

VAMP Architecture

VAMP consists of three components:

1. Value-Aware Action Selection – A three-way cost model comparing future penalties
2. Dynamic Resource Orchestrator – Repartitions HBM between expert weights and KV cache
3. Batching-Aware Expert Offloading – Overlaps weight transfers with computation

Cost Model

VAMP estimates the future cost of three actions when KV-cache allocation cannot be satisfied:

Cached-block eviction cost ($C_c$): Uses a sliding-window prefix-cache hit rate $r_{hit}$ and candidate age:

$$p_{eff} = \frac{\alpha \cdot r_{hit}}{1 + age/\tau}$$ $$C_c = p_{eff} \cdot (n_{tok} \cdot t_{prefill} + t_{overhead})$$

where $n_{tok}$ is the number of tokens in reclaimed blocks, $t_{prefill}$ is the per-token re-prefill cost, and $t_{overhead} = t_{sched} + t_{queue}$ covers scheduling overhead.

Expert-offload cost ($C_e$): Estimates per-step transfer cost over a decode-based accounting horizon $w$:

$$C_e(n_{cand}) = \min(n_{cand}, E) \cdot c_{eff} \cdot w$$

where $n_{cand} = g \cdot n_{groups}$ is the shard-equivalent size of the candidate expansion, $E$ is the number of local expert shards per layer per TP rank, and $c_{eff}$ is a configured comparison coefficient.

Request preemption cost ($C_p$): Based on the victim's last partially-filled block:

$$C_p^{(1)} = t_{sched} + t_{queue} + n_{partial} \cdot t_{prefill}$$ $$C_p = C_p^{(1)} \cdot G$$

where $G$ is the number of additional uncached KV-cache blocks needed.

Combined action cost ($C_b$): When expansion is not an exact multiple of the increment:

$$C_b = C_e^{floor} + C_c^{remainder}$$

Dynamic HBM Repartitioning

• Uses CUDA VMM to unmap physical pages from expert groups and remap them into the KV-cache pool
• Expert tensors' virtual address ranges remain reserved; only physical backing is reassigned
• Expert groups are the minimum page-aligned remap unit (6 MiB per group for Qwen3-Next: two 3-MiB shards under TP2, four 1.5-MiB shards under TP4)
• Offloading bound: at most $E_{buffer}$ expert shards offloaded per layer, keeping at least $E - E_{buffer}$ resident

Batching-Aware Expert Offloading

• Each batched MoE-layer execution may require 262–473 of 512 routed experts (mean across 1,012 profiled steps), not just top-k=10
• VAMP loads the entire contiguous offloaded region into a staging buffer ahead of MoE execution
• Double buffering alternates between two staging buffers across consecutive layers, overlapping transfers with computation

Empirical Validation / Results

Experimental Setup

• Model: Qwen3-Next-80B-A3B-Instruct (512 routed experts/layer, top-k=10, 48 layers)
• Platforms: 2×H100 PCIe (TP2, memory-constrained) and 4×H200 NVLink (TP4)
• Baseline: vLLM 0.15.1 with prefix caching and chunked prefill
• Workloads: SWE-agent closed-loop, recorded trace replays (2,103 turns), LMCache traces, controlled in-house multi-turn

Measured KV-Cache Expansion (H100-PCIe)

Recorded SWE-Agent Replay Results (n=5, mean±SD)

Capacity Boundary Results

H100-PCIe: The baseline is already beyond capacity at c=16 (TTFT p99 = 22.1s), while VAMP-25 stays at 3.0–3.9s through c=28. The boundary moves with offloading ratio: VAMP-15 crosses at c=24, VAMP-20 at c=28, VAMP-25 doesn't cross within the range.

H200-NVL: All configurations stay within 9–11s through c=28. At c=32, baseline TTFT p99 rises to 92.8s while VAMP-20 remains at 9.1s (10.2× reduction).

Tradeoff Analysis

• VAMP-15 achieves the lowest TTFT p90 with throughput comparable to VAMP-10
• VAMP-20 provides comparable TTFT but higher TPOT and lower throughput
• The optimal offloading ratio depends on workload: at 128 LMCache sessions, VAMP-15 is better; at 64 sessions, VAMP-20 achieves lower TTFT p99 (1.30s vs 1.57s)

Theoretical and Practical Implications

Design Implications

1. Static memory partitioning is fundamentally inadequate for MoE models serving multi-turn workloads. The two-pool approach (weights vs. KV cache) creates artificial scarcity that triggers cascading evictions precisely when prefix reuse is highest.

2. Value-aware decisions require forward-looking cost estimates. The paper demonstrates that comparing the estimated future penalties of different actions (rather than just reacting to immediate pressure) is essential for optimal memory management.

3. CUDA VMM enables efficient cross-pool reallocation without data copying. Page remapping at runtime provides fine-grained control that was previously unavailable in serving frameworks.

4. Per-token sparsity ≠ small batched working set. Even with top-k=10, batched execution requires 262–473 distinct experts per layer, making LRU-style expert caching impractical.

Practical Implications

• VAMP demonstrates that MoE serving can operate far beyond static KV-cache capacity boundaries, extending the practical serving range for long-running agentic workloads
• The framework provides a tunable knob (maximum offloading ratio) that can be matched to workload characteristics
• The approach is complementary to other techniques: host-side KV-cache offloading, prefix-cache scheduling, and weight quantization

Conclusion

VAMP breaks the static boundary between expert-weight memory and the KV-cache pool in MoE serving, using CUDA VMM page remapping guided by a three-way cost model. Key results:

• 23.6× reduction in TTFT p90 (26.1s → 1.10s) on a 2,103-turn recorded trace
• 20.7% increase in request throughput with near-zero preemptions
• Up to 2.7× KV-cache pool expansion per GPU
• Prefix-cache hit rate improvement from 0.19% to 89.8%

The cost is increased TPOT (+31.1%) due to expert weight staging transfers.

Future directions identified by the authors:

1. Restoring expert residency when KV-cache pressure subsides (currently one-way reallocation)
2. Combining with host-side KV-cache offloading to extend coverage beyond the expanded capacity
3. Matching offloading ratio to workload characteristics dynamically
4. Exploring interconnect-aware scheduling to reduce contention between weight transfers and tensor-parallel traffic