Summary (Overview)

• DeepSeek-V4.1-Flash is a multimodal Mixture-of-Experts (MoE) model with 552B backbone parameters supporting contexts up to 1M tokens, designed to push the limits of KV cache compression for input-heavy agentic workloads.
• The model introduces a Causal Encoder-Decoder (CED) architecture that activates only 8B parameters per token during prefill (vs. 16B during decode), nearly halving prefill computation.
• Compressed Sparse Attention 2 (CSA2) combines cross-layer KV cache reuse with FP4 KV caching, reducing the global KV cache footprint to 890 bytes per token — roughly 1/4 of DeepSeek-V4-Flash and 1/437 of DeepSeek-V1.
• SWA Bounded Replay reduces the persistent KV cache footprint to roughly 1/8 of DeepSeek-V4-Flash by approximately reconstructing SWA states from only the most recent $n_{win}$ tokens.
• Despite its compact activation footprint, the model achieves state-of-the-art agentic performance, surpassing closed-source frontier models on benchmarks like DeepSWE v1.1 (74.2%) and Terminal-Bench 2.1 (90.6%).

Introduction and Theoretical Foundation

The paper addresses the growing challenge of long-horizon agent workloads that are increasingly input-heavy. While prior work (DeepSeek-V4) reduced long-sequence computation costs via sparse attention, the KV cache footprint remains a critical bottleneck across three dimensions:

1. Compute: Prefill computation scales with input length
2. Storage: Global KV caches strain HBM capacity; persistent KV caches strain SSD/host memory
3. Bandwidth: I/O and interconnect bandwidth limit cache migration and loading

The theoretical foundation builds on several key insights:

• DeepSeek-V4's architecture combines a global attention branch with local Sliding-Window Attention (SWA)
• KV cache costs can be reduced along three multiplicative dimensions: entry size (GQA/MLA), sequence dimension (compression), and layer dimension (cross-layer reuse)
• Prior work (IndexCache, YOIO, HySparse) addressed individual dimensions but none covered all three jointly

The core observation motivating CED comes from YOCO: the upper half of layers can share KV caches generated by the lower half. CED extends this by treating the bottom $L/2$ layers as a causal encoder whose final hidden states project decoder global KV.

Methodology

Architecture Components

1. Causal Encoder-Decoder (CED) For global attention, decoder KV entries are projected from the $(L/2)$-th layer hidden state:

$$C_l = H_{L/2} W_l^{KV}, \quad Z_l = H_{L/2} W_l^{Z}, \quad l > \frac{L}{2} \tag{1}$$

This reduces prefill complexity from $O(NL)$ to $O(NL/2 + n_{win} \times L/2) \approx O(NL/2)$.

2. Compressed Sparse Attention 2 (CSA2) Three statically assigned modes:

• Full Mode: Computes own main KV, indexer K, and Top-K indices
• Reindex Mode: Reuses main KV from preceding layer, computes fresh Top-K indices
• Reuse Mode: Reuses both global KV and Top-K indices from preceding layer

3. Hierarchical Sparse Indexer The first Full Mode layer constructs a candidate pool (up to 16,384 positions from 2,048 blocks × 8 positions) that constrains search for subsequent Reindex layers, making per-query cost constant in context length.

4. Single-Pass mHC Shifts input-mixing coefficients by one block to eliminate data dependencies:

$$X_{l+1} = B_l X_l + C_l \mathcal{F}_l(A_{l-1} X_l), \quad (A_l, B_l, C_l) = \mathcal{H}(X_l) \tag{6}$$

This achieves the theoretical lower bound of $(2n+2)d$ activation memory traffic (halving the original).

5. FP4 Main KV Cache Uses OCP-standard MXFP4 format (E2M1 with one E4M3 scale per 16 channels). The maximum KV magnitude is bounded by $\sqrt{512} \approx 22.6$, well within the format's range of 2688.

6. Sinkhorn-Balanced Updates For Engram embedding tables and prediction heads, replacing Adam:

$$\Delta_t = \sqrt{n} U^{(K)} = \sqrt{n} D_r \widehat{G}_t D_c \tag{7}$$

Training Setup

• 45T tokens multimodal corpus, 7:1 text-to-multimodal ratio
• Batch size: 100.6M tokens; peak LR $2.6 \times 10^{-4}$ with cosine decay
• Sequence length: 64K from scratch, extended to 1M at 34T tokens
• Optimizers: Muon (linear weights), AdamW (norms), Sinkhorn-balanced (embeddings)

Empirical Validation / Results

Base Model Performance (Table 1)

The base model achieves performance comparable to DeepSeek-V4-Pro-Base (1.6T params) using only 552B params and 8B/16B activated parameters.

Post-Trained Agentic Performance (Table 3)

KV Cache Compression Results

• Global KV cache: 890 bytes/token (vs. ~3,560 for V4-Flash, ~389,000 for V1)
• Persistent KV cache: 1/8 of V4-Flash (via SWA Bounded Replay)
• Decode FLOPs: Nearly constant across context lengths — extending from 4K to 1M increases FLOPs by only 1/4

Reasoning Effort Control

Raising effort from 25 to 100 improves:

• Average Pass@1 on 8 reasoning benchmarks: 67.1% → 76.3%
• DeepSWE v1.1: 66.0% → 74.2%
• Terminal-Bench 2.1: 82.4% → 90.6%
• At the cost of ~2.5× more output tokens

Multi-Agent Performance

Multi-agent configurations outperform single-agent at every deadline:

• ProgramBench: 30.04% vs 20.39% (at 8 hours)
• FrontierSWE v2: 32.90% vs 28.20% (at 20 hours)

Theoretical and Practical Implications

Theoretical Contributions

1. Joint optimization across three dimensions of KV cache reduction (entry size, sequence, layer), demonstrating that multiplicative compression is achievable without performance degradation
2. SWA Bounded Replay establishes a new storage–computation trade-off: approximate reconstruction from $n_{win}$ tokens (vs. $L \times n_{win}$) incurs negligible quality loss
3. Sinkhorn-balanced optimization extends matrix-level normalization to embedding tables, showing empirical superiority over Adam with only a momentum buffer

Practical Implications

• Deployment cost reduction: 4× reduction in HBM pressure and 8× in SSD/host memory enables large-scale long-horizon agent deployment
• Parameter efficiency: 552B backbone achieving parity with 1.6T models suggests diminishing returns from raw scale alone
• Reasoning effort control: A single checkpoint can serve diverse cost–quality operating points via scalar effort levels (50/75/100)
• Scaffold robustness: Consistent performance across 8 configurations from 6 scaffold families indicates the model learns general agentic behavior, not harness-specific conventions

Conclusion

DeepSeek-V4.1-Flash demonstrates that aggressive KV cache compression (437× vs. V1) can be achieved without sacrificing—and indeed improving—model capability. Key takeaways:

1. Architecture matters more than scale: CED + CSA2 + FP4 caching enable a 552B model to outperform much larger counterparts
2. Data pipeline is the primary lever in post-training: All gains came from data synthesis and environment construction, not algorithmic novelty
3. Storage–computation trade-offs can be renegotiated: SWA Bounded Replay shows approximate reconstruction is viable when exact recovery is prohibitive

Limitations and Future Directions:

• Robustness boundaries of CSA2 selection errors and SWA reconstruction remain uncharacterized
• A performance gap persists on expert-level scientific tasks (e.g., Terminal-Bench 4.0)
• Evaluation infrastructure is increasingly susceptible to agent gaming, requiring community attention
• Future work will pursue joint scaling of architecture, pre-training, and post-training, with model–harness co-design