# Periodic Weak Spots: Phase Sensitivity from Chunked KV-Cache Compression

> Chunked KV-cache compression creates periodic retrieval weak spots tied to token position mod the compression stride, which average benchmarks hide.

- **Source:** [arXiv](https://arxiv.org/abs/2609.36322)
- **Published:** 2026-10-24
- **Permalink:** https://picx.dev/p/5xx5Hj
- **Whiteboard:** https://picx.dev/p/5xx5Hj/image

## Summary

# Summary of "Periodic Weak Spots: Phase Sensitivity from Chunked KV-Cache Compression"

## Summary (Overview)

- **Discovery of phase sensitivity**: The paper identifies a systematic failure mode in transformer models using chunked KV-cache compression, where retrieval accuracy varies periodically (by up to **40 percentage points** in DeepSeek-V4 at 128K context) depending on a token's *phase*—its position modulo the compression stride $S$.

- **Empirical reproduction**: The authors pretrain a family of Qwen3-derived transformers from scratch with various KV-compression kernels, showing that *every* compressed model exhibits phase sensitivity with period matching the compression stride, while full-attention baselines do not.

- **Mechanistic analysis via knockouts**: Causal interventions (KV-head knockouts) reveal **phase specialization**—different attention heads contribute asymmetrically to retrieval at different phases, with some heads developing persistent static preferences for specific compression-window offsets during pretraining.

- **Theoretical foundation**: An idealized induction model proves that optimal compression is maximally concentrated (Theorem 1), and gradient flow drives gates toward **static concentration** (Theorem 2), explaining how phase specialization emerges even when future queries are unknown.

- **Key implication**: Average benchmark scores conceal systematic positional failures; evaluation of compressed models must measure performance across all compression phases.

---

## Introduction and Theoretical Foundation

### Background and Motivation

Long-context inference in transformers is bottlenecked by the KV cache, whose memory and attention costs grow with context length. **Chunked KV-cache compression** (used in models like DeepSeek-V4) groups consecutive tokens into fixed-size windows, compresses each window into fewer cache entries, and uses these summaries for long-context retrieval. Crucially, compression happens *before* the query is known, making each summary **query-agnostic**.

### The Phase Coordinate

The compression design imposes a periodic structure on the context. A new compression window begins every $S$ tokens (the compression stride). Each token has a **phase** defined as its position modulo $S$:

$$\text{phase}(t) = t \bmod S$$

Shifting input by a few tokens changes which tokens are compressed together without changing content or relative positions—so the same information may be retained with different fidelity depending on its phase.

### Theoretical Setup for Analysis

The paper analyzes an idealized induction task where a context $\pmb{X} = (\pmb{x}_{\ell,r})_{\ell \leq L, r \leq S}$ consists of $LS$ distinct token embeddings sampled uniformly without replacement from a vocabulary $\mathcal{V} \subset \mathbb{R}^d$. Each of $1 \leq H < S$ heads compresses chunk $\ell$ into:

$$
\boldsymbol{k}_{h,\ell} = \sum_{r=1}^{S} p_{h,\ell}^{K}(r) \boldsymbol{x}_{\ell,r} \in \mathbb{R}^{d}, \quad \boldsymbol{v}_{h,\ell} = \sum_{r=1}^{S} p_{h,\ell}^{V}(r) \boldsymbol{x}_{\ell,r} \in \mathbb{R}^{d} \tag{3}
$$

where $p_{h,\ell}^{K}, p_{h,\ell}^{V}$ are probability vectors chosen *before* the query is revealed. An idealized embedding geometry assumes:

$$
\left| \boldsymbol{x}^{\top}\boldsymbol{x}^{\prime} - \kappa - \mathbf{1}\{\boldsymbol{x} = \boldsymbol{x}^{\prime}\} \right| \leq \varepsilon \quad \forall \boldsymbol{x}, \boldsymbol{x}^{\prime} \in \mathcal{V} \tag{4}
$$

with common component $\kappa > 0$ and error $\varepsilon \geq 0$.

---

## Methodology

### KV Compression Kernel Family

Each kernel compresses a window into one key and one value per KV head. For one head, let $h_{j,i} \in \mathbb{R}^{d}$ be the hidden state at offset $0 \leq i < W$ in window $j$. For branch $B \in \{K, V\}$:

**Gate logits**:
$$
\pmb{s}_{j,i}^{K} = \pmb{Z}_{i}^{K}\pmb{h}_{j,i} + \pmb{b}_{i}^{K} \tag{1}
$$

**Softmax gate scores and compressed output**:
$$
\boldsymbol{\alpha}_{j,i}^{K} = \frac{\exp(\boldsymbol{s}_{j,i}^{K})}{\sum_{u=0}^{W-1}\exp(\boldsymbol{s}_{j,u}^{K})}, \quad \widehat{K}_{j} = \sum_{i=0}^{W-1} \boldsymbol{\alpha}_{j,i}^{K} \odot (\boldsymbol{C}_{i}^{K}\boldsymbol{h}_{j,i}) \tag{2}
$$

Key design variations tested:
- **Window size $W$ vs. stride $S$**: varied independently (W8/S4, W8/S6, W8/S8, W12/S12, W4/S4) to determine whether period follows window or stride
- **Overlapping windows** ($S < W$): tests whether tokens appearing at multiple offsets remove the effect
- **Uniform averaging**: replaces learned gates with fixed $1/W$ scores
- **Vector gates** ($d_g = d_h$) vs. scalar gates ($d_g = 1$)
- **DeepSeek-V4's CSA kernel**: tests the actual production kernel on the Qwen backbone
- **Ablations**: with/without RoPE, tied/untied K/V gates

### Evaluation Protocols

Two padding protocols control for confounds:
- **Prefix Padding**: shifts all records together with outer fillers
- **In-sequence Padding**: varies gaps between records so distractor phases don't move in lockstep with target

Both protocols fix query position and match mean/variance of target-position sampling distributions across residue groups of source-key position modulo 24.

### Knockout Interventions

**Mean replacement** [35] replaces a KV head's output at intervened positions with its average over many other prompts, so the output no longer carries prompt-specific information. Accuracy drops at a phase indicate the head's causal contribution to retrieval at that phase.

---

## Empirical Validation / Results

### Phase Sensitivity in DeepSeek-V4 Family

**Code-completion example**: DeepSeek-V4-Flash-Base's preferred completion of a code snippet flips periodically as a decorative docstring length varies. The model predicts "8" when $L \bmod 4 \in \{2, 3\}$ and "32" otherwise, with $P(8) - P(32)$ ranging from −0.79 to 0.95. The period matches the compression stride $S=4$.

**NIAH retrieval results** (128K context, 16k key–value pairs):

| Model | Max phase gap (percentage points) |
|-------|-----------------------------------|
| DeepSeek-V4-Flash-Base | up to 40.2 |
| DeepSeek-V4-Pro (post-trained) | large gaps remain |
| DeepSeek-V4.1-Flash | smallest gap; all even residue groups above all odd ones |

### Controlled Pretraining Results

Every compressed model shows periodicity matching the compression stride; full-attention baselines stay flat. Key findings:

- **Prefix Padding gaps**: up to **78 percentage points** for compressed models vs. at most **6.1** for baselines
- **Period follows stride**: $S \in \{4, 6, 8\}$ give periods of about 4, 6, and 8 tokens
- **Phase sensitivity persists** without RoPE and with uniform averaging (no learned gates)
- **Window boundary is insufficient**: In W8/S8, accuracy varies by more than 50 percentage points among phases 0–6 whose key–value pairs share a window

### Knockout Maps

- **Full attention baseline**: uniform horizontal stripes—each layer contributes similarly at every source position
- **W8/S8 compressed model**: heterogeneous patterns; some heads have losses concentrated on a few phases
- **DeepSeek-V4-Flash-Base**: pattern repeats with period 4 (CSA stride); odd layers using heavily compressed attention (HCA, compression rate 128) contribute little to retrieval

### Gate Preference Case Study

Using the effective number of offsets $\exp(H(\cdot))$ (Shannon entropy), the paper distinguishes:
- **Static concentration**: same offsets favored in every window ($\exp(H(\mathbb{E}[\boldsymbol{\alpha}]))$ small)
- **Input-dependent concentration**: offsets vary with content ($\mathbb{E}[\exp(H(\boldsymbol{\alpha}))]$ small)

During pretraining, some gates (layers 9, 10, 14) drift along the diagonal toward static concentration. Each key gate peaks at a few adjacent offsets (early for layer 14, middle for layer 10, late for layer 9), with value gates peaking one offset later—matching a bigram key–value structure.

### Theoretical Results

**Theorem 1 (Informal: optimal compression is concentrated)**: Every global minimizer over the oracle class sets $p_{h,\ell}^{K} = e_r$ and $p_{h,\ell}^{V} = e_{r+1}$ in every context, chunk, and head, where $r \in \{1, \ldots, S-1\}$. When $\varepsilon = 0$, selected positions are distinct across heads within each context and chunk.

**Theorem 2 (Informal: gradient flow yields static concentration)**: Under exact geometry ($\varepsilon = 0$) and small random initialization, gradient flow yields for each head $h$ a position $r_h$ such that $p_{h,\ell}^{K}(r_h), p_{h,\ell}^{V}(r_h + 1) \to 1$ as training time tends to infinity, uniformly over contexts and chunks. Different heads may select the same $r_h$.

---

## Theoretical and Practical Implications

### Theoretical Implications

1. **Optimal compression is concentrated even without query knowledge**: The loss landscape inherently favors one-hot compression (Theorem 1), explaining why models learn selective retention.

2. **Gradient flow explains static specialization**: Training dynamics fix each head's preferred phase permanently (Theorem 2), providing an idealized account of the adjacent key-value preferences observed in layers 9, 10, and 14.

3. **Phase specialization is a distinct failure mode**: Different from "Lost in the Middle" (context depth effects) or segment-boundary effects in gist-based compression; it recurs every compression stride.

### Practical Implications

1. **Evaluation methodology must change**: Average accuracy is insufficient for compressed models. Evaluation should shift input by a few tokens while keeping queried information fixed to expose phase sensitivity.

2. **Deployment risk**: A model with high average benchmark scores can systematically fail on information at specific phases—a critical concern for production systems using chunked KV compression.

3. **Architectural guidance**: The finding that gate cycling shifts accuracy patterns (Appendix H) suggests explicit coordination across heads and layers might make retrieval more uniform across phases.

4. **Robustness concerns**: Periodic errors in code completion motivate investigating whether meaning-preserving shifts within a context alter refusal/compliance behavior in safety-critical applications.

---

## Conclusion

This paper identifies **phase sensitivity** as a systematic failure mode in models with chunk-compressed KV caches: retrieval accuracy depends on the source information's phase relative to compression windows. The phenomenon appears in DeepSeek-V4, V4.1, and all pretrained compressed models, with period following the compression stride.

**Key takeaways**:
- Chunked KV-cache compression introduces periodic weak spots that average benchmarks conceal
- Phase specialization emerges through static gate concentration during training
- Both boundary effects (key–value pairs straddling windows) and within-window asymmetries contribute

**Open questions**:
- Gate cycling does not restore the boundary phase, leaving a unified explanation of boundary and within-window asymmetries unresolved
- Whether explicit coordination across heads/layers can make retrieval uniform without sacrificing average performance or compression efficiency
- Whether phase sensitivity affects robustness-relevant behaviors (e.g., refusal/compliance changes under meaning-preserving shifts) in real applications

The authors' mechanistic conclusions are strongest in controlled settings and do not establish a universal cause of phase sensitivity in large, heterogeneous models—but the phenomenon's consistency across architectures and training regimes makes it an essential consideration for any deployment of chunked KV-cache compression.

---

_Markdown view of https://picx.dev/p/5xx5Hj, served by PicX — AI-generated visual whiteboard summaries of research papers._