# DeepSeek-V4.1-Flash: Pushing the Limits of KV Cache Compression

> DeepSeek-V4.1-Flash compresses KV cache 437x via causal encoder-decoder and FP4 sparse attention, enabling a 552B model to outperform larger frontier models on agentic benchmarks.

- **Source:** [arXiv](https://arxiv.org/abs/2609.19969)
- **Published:** 2026-09-19
- **Permalink:** https://picx.dev/p/G5v4Vn
- **Whiteboard:** https://picx.dev/p/G5v4Vn/image

## Summary

## Summary (Overview)

- **DeepSeek-V4.1-Flash** is a multimodal Mixture-of-Experts (MoE) model with 552B backbone parameters supporting contexts up to 1M tokens, designed to push the limits of KV cache compression for input-heavy agentic workloads.
- The model introduces a **Causal Encoder-Decoder (CED)** architecture that activates only 8B parameters per token during prefill (vs. 16B during decode), nearly halving prefill computation.
- **Compressed Sparse Attention 2 (CSA2)** combines cross-layer KV cache reuse with FP4 KV caching, reducing the global KV cache footprint to **890 bytes per token** — roughly 1/4 of DeepSeek-V4-Flash and 1/437 of DeepSeek-V1.
- **SWA Bounded Replay** reduces the persistent KV cache footprint to roughly 1/8 of DeepSeek-V4-Flash by approximately reconstructing SWA states from only the most recent $n_{win}$ tokens.
- Despite its compact activation footprint, the model achieves state-of-the-art agentic performance, **surpassing closed-source frontier models** on benchmarks like DeepSWE v1.1 (74.2%) and Terminal-Bench 2.1 (90.6%).

## Introduction and Theoretical Foundation

The paper addresses the growing challenge of **long-horizon agent workloads** that are increasingly input-heavy. While prior work (DeepSeek-V4) reduced long-sequence computation costs via sparse attention, the **KV cache footprint** remains a critical bottleneck across three dimensions:

1. **Compute**: Prefill computation scales with input length
2. **Storage**: Global KV caches strain HBM capacity; persistent KV caches strain SSD/host memory
3. **Bandwidth**: I/O and interconnect bandwidth limit cache migration and loading

The theoretical foundation builds on several key insights:

- DeepSeek-V4's architecture combines a global attention branch with local Sliding-Window Attention (SWA)
- KV cache costs can be reduced along **three multiplicative dimensions**: entry size (GQA/MLA), sequence dimension (compression), and layer dimension (cross-layer reuse)
- Prior work (IndexCache, YOIO, HySparse) addressed individual dimensions but none covered all three jointly

The core observation motivating CED comes from YOCO: the upper half of layers can share KV caches generated by the lower half. CED extends this by treating the bottom $L/2$ layers as a causal encoder whose final hidden states project decoder global KV.

## Methodology

### Architecture Components

**1. Causal Encoder-Decoder (CED)**
For global attention, decoder KV entries are projected from the $(L/2)$-th layer hidden state:

$$C_l = H_{L/2} W_l^{KV}, \quad Z_l = H_{L/2} W_l^{Z}, \quad l > \frac{L}{2} \tag{1}$$

This reduces prefill complexity from $O(NL)$ to $O(NL/2 + n_{win} \times L/2) \approx O(NL/2)$.

**2. Compressed Sparse Attention 2 (CSA2)**
Three statically assigned modes:
- **Full Mode**: Computes own main KV, indexer K, and Top-K indices
- **Reindex Mode**: Reuses main KV from preceding layer, computes fresh Top-K indices
- **Reuse Mode**: Reuses both global KV and Top-K indices from preceding layer

**3. Hierarchical Sparse Indexer**
The first Full Mode layer constructs a candidate pool (up to 16,384 positions from 2,048 blocks × 8 positions) that constrains search for subsequent Reindex layers, making per-query cost constant in context length.

**4. Single-Pass mHC**
Shifts input-mixing coefficients by one block to eliminate data dependencies:

$$X_{l+1} = B_l X_l + C_l \mathcal{F}_l(A_{l-1} X_l), \quad (A_l, B_l, C_l) = \mathcal{H}(X_l) \tag{6}$$

This achieves the theoretical lower bound of $(2n+2)d$ activation memory traffic (halving the original).

**5. FP4 Main KV Cache**
Uses OCP-standard MXFP4 format (E2M1 with one E4M3 scale per 16 channels). The maximum KV magnitude is bounded by $\sqrt{512} \approx 22.6$, well within the format's range of 2688.

**6. Sinkhorn-Balanced Updates**
For Engram embedding tables and prediction heads, replacing Adam:

$$\Delta_t = \sqrt{n} U^{(K)} = \sqrt{n} D_r \widehat{G}_t D_c \tag{7}$$

### Training Setup
- 45T tokens multimodal corpus, 7:1 text-to-multimodal ratio
- Batch size: 100.6M tokens; peak LR $2.6 \times 10^{-4}$ with cosine decay
- Sequence length: 64K from scratch, extended to 1M at 34T tokens
- Optimizers: Muon (linear weights), AdamW (norms), Sinkhorn-balanced (embeddings)

## Empirical Validation / Results

### Base Model Performance (Table 1)

| Benchmark | DS-V4-Flash Base | DS-V4-Pro Base | DS-V4.1-Flash Base |
|---|---|---|---|
| MMLU-Pro | 68.3 | 73.5 | **74.1** |
| BigCodeBench | 56.8 | 59.2 | **60.6** |
| HumanEval | 69.5 | 76.8 | **79.4** |
| GSM8K | 90.8 | 92.6 | **93.0** |
| SuperGPQA | 46.5 | 53.9 | **53.1** |

The base model achieves performance comparable to DeepSeek-V4-Pro-Base (1.6T params) using only 552B params and 8B/16B activated parameters.

### Post-Trained Agentic Performance (Table 3)

| Benchmark | Opus-5 | GPT-5.6 Sol | Kimi-K3 | DS-V4-Pro | **DS-V4.1-Flash** |
|---|---|---|---|---|---|
| DeepSWE v1.1 | 74.0 | 73.0 | 66.9 | 62.7 | **74.2** |
| Terminal-Bench 2.1 | 89.1 | 88.8 | 88.2 | 87.9 | **90.6** |
| Automation-Bench | 50.3 | 45.8 | 48.8 | 43.2 | **54.8** |
| Agents' Last Exam | 28.6 | 26.7 | 28.5 | 25.7 | **31.8** |
| CyberGym | - | 84.5 | 84.5 | 83.3 | **88.1** |
| Codeforces (Rating) | - | - | - | 3348 | **3471** |

### KV Cache Compression Results

- **Global KV cache**: 890 bytes/token (vs. ~3,560 for V4-Flash, ~389,000 for V1)
- **Persistent KV cache**: 1/8 of V4-Flash (via SWA Bounded Replay)
- **Decode FLOPs**: Nearly constant across context lengths — extending from 4K to 1M increases FLOPs by only 1/4

### Reasoning Effort Control
Raising effort from 25 to 100 improves:
- Average Pass@1 on 8 reasoning benchmarks: 67.1% → 76.3%
- DeepSWE v1.1: 66.0% → 74.2%
- Terminal-Bench 2.1: 82.4% → 90.6%
- At the cost of ~2.5× more output tokens

### Multi-Agent Performance
Multi-agent configurations outperform single-agent at every deadline:
- ProgramBench: 30.04% vs 20.39% (at 8 hours)
- FrontierSWE v2: 32.90% vs 28.20% (at 20 hours)

## Theoretical and Practical Implications

### Theoretical Contributions
1. **Joint optimization across three dimensions** of KV cache reduction (entry size, sequence, layer), demonstrating that multiplicative compression is achievable without performance degradation
2. **SWA Bounded Replay** establishes a new storage–computation trade-off: approximate reconstruction from $n_{win}$ tokens (vs. $L \times n_{win}$) incurs negligible quality loss
3. **Sinkhorn-balanced optimization** extends matrix-level normalization to embedding tables, showing empirical superiority over Adam with only a momentum buffer

### Practical Implications
- **Deployment cost reduction**: 4× reduction in HBM pressure and 8× in SSD/host memory enables large-scale long-horizon agent deployment
- **Parameter efficiency**: 552B backbone achieving parity with 1.6T models suggests diminishing returns from raw scale alone
- **Reasoning effort control**: A single checkpoint can serve diverse cost–quality operating points via scalar effort levels (50/75/100)
- **Scaffold robustness**: Consistent performance across 8 configurations from 6 scaffold families indicates the model learns general agentic behavior, not harness-specific conventions

## Conclusion

DeepSeek-V4.1-Flash demonstrates that aggressive KV cache compression (437× vs. V1) can be achieved **without sacrificing—and indeed improving—model capability**. Key takeaways:

1. **Architecture matters more than scale**: CED + CSA2 + FP4 caching enable a 552B model to outperform much larger counterparts
2. **Data pipeline is the primary lever in post-training**: All gains came from data synthesis and environment construction, not algorithmic novelty
3. **Storage–computation trade-offs can be renegotiated**: SWA Bounded Replay shows approximate reconstruction is viable when exact recovery is prohibitive

**Limitations and Future Directions**:
- Robustness boundaries of CSA2 selection errors and SWA reconstruction remain uncharacterized
- A performance gap persists on expert-level scientific tasks (e.g., Terminal-Bench 4.0)
- Evaluation infrastructure is increasingly susceptible to agent gaming, requiring community attention
- Future work will pursue joint scaling of architecture, pre-training, and post-training, with model–harness co-design

---

_Markdown view of https://picx.dev/p/G5v4Vn, served by PicX — AI-generated visual whiteboard summaries of research papers._