# Lost in a Few Heads: Diagnosing and Repairing Block Selection in Sparse Attention

> Sparse attention's long-context failures stem from selectors under-ranking the answer's block, not budget limits, and training-free fixes on a few flagged heads recover most accuracy.

- **Source:** [arXiv](https://arxiv.org/abs/2611.21917)
- **Published:** 2026-11-21
- **Permalink:** https://picx.dev/p/0GZn7A
- **Whiteboard:** https://picx.dev/p/0GZn7A/image

## Summary

# Summary of "Lost in a Few Heads: Diagnosing and Repairing Block Selection in Sparse Attention"

## Summary (Overview)

- **Core Finding**: Sparse attention models lose long-context retrieval accuracy not because their block-selection budget is too small, but because their selectors **under-rank the needle block** (the block containing the answer key) — a ranking failure rather than a budget shortfall.

- **Diagnostic Method**: The authors develop a **calibration probe** with two readouts: (1) the needle attention that the selector forfeits (explains *why* a head fails), and (2) the fix gain — the integrated effect of repairing a head's selection on the answer's log-probability (identifies *which* heads to fix).

- **Key Result**: In the MiniCPM4-8B lineage (InfLLM-V2, NOSA), the multi-key retrieval loss concentrates in just **three "blind" attention heads** (layers 9, 16, 22, head 0). Fixing only these heads recovers most of the loss without any training.

- **Training-Free Fixes**: Full attention in flagged heads, a **4-bit scorer** at the shipped budget, or a larger per-layer budget each recover most of the multi-key loss. On MiniCPM4.1-8B at 64K, these fixes lift accuracy from 54.1% to 97.5%, matching full attention.

- **Generalization**: A unified threshold rule identifies a small set of blind heads across multiple selector designs (SeerAttention-R, NSA, DashAttention), while MoBA spreads the loss over many heads. Token-level selectors (DSA, LongCat) show no such loss.

---

## Introduction and Theoretical Foundation

### Background

Large language models (LLMs) increasingly handle long contexts (32K–128K+ tokens), where full attention's quadratic cost becomes prohibitive. Sparse attention addresses this by selecting a fixed budget of key blocks per query. However, **the attention after selection is exact** — so any accuracy loss comes purely from the *selection mechanism*.

### The Problem

A fixed budget hides a subtle failure: a budget adequate on average can still be spent on the wrong blocks for a particular query. For example, InfLLM-V2-8B's sparse mode performs far below its full-attention mode on RULER's multi-key retrieval task.

### Theoretical Foundation

The paper builds on several key concepts:

1. **Exact mass selection** as an oracle: The selection criterion that keeps the most attention mass for query $t$ at budget $K$ is:
$$c_b^{\mathrm{exact}} = M_b = \sum_{g=1}^{G} \sum_{j \in \mathcal{B}_b} a_{t,j}^{(g)} \tag{2}$$

2. **Pooled-key selection** (InfLLM-V2 style), which scores blocks via mean-pooled keys:
$$\bar{\boldsymbol{k}}_m = \frac{1}{w} \sum_{j \in W_m} \boldsymbol{k}_j, \qquad c_b^{\mathrm{pool}} = \max_{m \in \mathcal{W}(b)} \sum_{g=1}^{G} \operatorname{softmax}_m \left(\frac{\boldsymbol{q}_t^{(g)\top} \bar{\boldsymbol{k}}_m}{\sqrt{d}}\right) \tag{1}$$

3. **Integrated gradients** (Sundararajan et al., 2017) for attributing the answer's log-probability across heads, along the path from shipped selector to full attention.

### Key Theoretical Insight (Proposition 1)

A two-block model shows when pooling loses the needle. Let a block hold one needle key whose logit exceeds its neighbors' by $\Delta$, and a competitor block have every key's logit elevated by $\delta$. Then:

- **Exact mass** ranks the needle block first whenever $\Delta > \delta + \log L$
- **Any score increasing with the mean logit** over a window of $w$ keys ranks the competitor first whenever $\Delta < w\delta$

So every needle with $\delta + \log L < \Delta < w\delta$ is **lost to pooling** — a "blind window" where mean-pooling erases the needle's signal.

---

## Methodology

### 1. Diagnostic Instrument

The instrument applies exact mass selection at specific tokens $\mathcal{T}$ and heads $\mathcal{H}$:

$$\mathcal{S}_t = \begin{cases} \operatorname{TopK}_{b \notin \mathcal{F}}(M_b, K) & \text{if } t \in \mathcal{T} \text{ and the head is in } \mathcal{H}, \\ \mathcal{S}_t^{\mathrm{ship}} & \text{otherwise} \end{cases} \tag{3}$$

This separates ranking failures from budget shortfalls by comparing accuracy against the shipped selector.

### 2. Calibration Probe

The probe **teacher-forces** the answer's needle tokens and measures two quantities per head:

**Forfeited needle attention**: $F_u = (r_u(\mathcal{E}) - r_u(\mathcal{S})) \cdot m_u$, where $r_u(\mathcal{E}) - r_u(\mathcal{S})$ is the recall loss against exact mass and $m_u$ is the attention on the needle block.

**Fix gain** (integrated gradient):
$$\Gamma_u = \frac{1}{|\mathcal{X}_c|} \sum_{x \in \mathcal{X}_c} \left| \int_0^1 \frac{\partial \mathcal{F}_x}{\partial \alpha_u} \Big|_{\boldsymbol{\alpha} = \beta \mathbf{1}} d\beta \right| \tag{5}$$

The gate $\alpha_u$ blends shipped and full attention per head:
$$\boldsymbol{o}_i^{(g)}(\alpha) = \boldsymbol{o}_i^{(g),\mathrm{ship}} + \alpha_u \left(\boldsymbol{o}_i^{(g),\mathrm{full}} - \boldsymbol{o}_i^{(g),\mathrm{ship}}\right) \tag{4}$$

### 3. Threshold Rule

Flagged heads are those above a threshold swept per checkpoint:
$$\mathcal{H}(\tau) = \{u : \Gamma_u \geq \tau\}$$

The sweep starts at the largest score, lowers $\tau$ by a factor of $\sqrt{2}$ per step, and keeps the first threshold whose heads close a share $\rho = 0.9$ of the gap between shipped and full attention.

### 4. Training-Free Fixes

Three fixes, all acting only in flagged heads:

1. **Full attention** in flagged heads (reads every block)
2. **Larger budget** in flagged layers (4× budget, no new code)
3. **4-bit scorer** at shipped budget, using quantized keys:
$$\hat{\boldsymbol{k}}_{j,c} = \mu_c + \sigma_c \Delta_b z_{j,c}, \qquad z_{j,c} = \mathrm{clamp}\left(\left\lfloor \frac{k_{j,c} - \mu_c}{\sigma_c \Delta_b} \right\rfloor + \frac{1}{2}, -2^{b-1} + \frac{1}{2}, 2^{b-1} - \frac{1}{2}\right) \tag{6}$$

The scorer approximates exact mass with 4-bit keys:
$$\hat{M}_b = \sum_{g=1}^{G} \sum_{j \in \mathcal{B}_b} \mathrm{softmax}_j \left(\frac{\boldsymbol{q}_t^{(g)\top} \hat{\boldsymbol{k}}_j}{\sqrt{d}}\right) \tag{7}$$

---

## Empirical Validation / Results

### Where the Loss Is Made (Table 1)

| Selection | InfLLM-V2-8B, 32K | MiniCPM4.1-8B, 32K | MiniCPM4.1-8B, 64K |
|---|---|---|---|
| Sparse | 63.1 | 82.7 | 54.1 |
| Exact mass at decode | 85.4 | 89.2 | 80.0 |
| Exact mass at decode and window | 95.3 | 100.0 | 97.5 |
| Dense at every token of flagged heads | 96.0 | 100.0 | 97.5 |
| Dense | 97.8 | 100.0 | 97.1 |

**Key finding**: Exact mass selection at the *shipped budget* recovers most of the loss — proving the selector under-ranks the needle rather than running out of budget.

### Fix Results (Table 2)

| Model | Context | Dense | Sparse | Full Attn, Flagged | 4-bit Scorer (Decode) | 4-bit (+Question) | 4-bit (Every Token) |
|---|---|---|---|---|---|---|---|
| InfLLM-V2-8B | 32K | 97.8 | 63.1 | 96.0 | 85.6 | 95.1 | 94.7 |
| MiniCPM4.1-8B | 32K | 100.0 | 82.7 | 100.0 | 89.1 | 99.9 | 100.0 |
| MiniCPM4.1-8B | 64K | 97.1 | 54.1 | 97.5 | 79.7 | 97.2 | 97.5 |

### The Three Blind Heads

Both probe readouts consistently identify **key-value head 0 of layers 9, 16, and 22** across all InfLLM-V2 checkpoints. These heads:
- Attend to the needle whenever its block is selected
- Fail because their selector (mean-pooled keys) dilutes the needle's single decisive token
- Are flagged from 5 calibration seeds, at 16K, and from held-out JSON data

### Generalization Across Selectors (Table 6)

| Model | Selector | Heads | Dense | Sparse | Flagged + Full Attn | Flagged + 4-bit | Echo | Random |
|---|---|---|---|---|---|---|---|---|
| Qwen3-8B | SeerAttention-R | 18 of 288 | 98.7 | 29.8 | 97.7 | 97.3 | 69.4 | 31.4 |
| Qwen3-4B | SeerAttention-R | 9 of 288 | 97.5 | 16.6 | 87.8 | 88.9 | 56.3 | 17.5 |
| Qwen3-14B | SeerAttention-R | 5 of 320 | 99.7 | 54.5 | 96.9 | 98.1 | 64.1 | 54.6 |
| Dash-8B | DashAttention | 13 of 64 | 99.4 | 66.6 | 98.0 | 92.0 | 80.9 | 91.0 |
| NSA-8B | NSA | 4 of 64 | 98.1 | 14.1 | 95.2 | 94.5 | 68.2 | 13.5 |
| MoBA, Qwen3-8B | MoBA | 513 of 1152 | 94.4 | 21.1 | 89.6 | 75.8 | 85.9 | 50.5 |

### Cost Analysis (Table 3, InfLLM-V2-8B at 32K)

| Fix | MK3 | CWE | PopQA | Bytes per Cached Token |
|---|---|---|---|---|
| Shipped selector | 63.1 | 45.5 | 41.2 | 1,024 |
| Full attention, flagged heads | 96.0 | 33.9 | 38.0 | 2,512 |
| 4× budget, flagged layers | 97.2 | 36.6 | 37.7 | 2,140 |
| Scorer, every token | 94.7 | 40.0 | 37.8 | 1,168 |
| Scorer, decode | 85.6 | 45.6 | 41.0 | 1,168 |
| Dense | 97.8 | 51.6 | 39.5 | 32,768 |

### Latency (Figure 7)

Full attention in flagged heads keeps **84–89% of the TTFT savings** that sparse attention provides at 32K–128K context lengths in production serving engines.

---

## Theoretical and Practical Implications

### Theoretical Implications

1. **Pooling dilutes decisive tokens**: The two-block model (Proposition 1) formally characterizes when mean-pooled scoring loses needles. The blind window $\delta + \log L < \Delta < w\delta$ widens with more competitors and with longer contexts.

2. **Token resolution > precision**: At the shipped budget, what matters is **token-level resolution in the right heads**, not higher precision. Quantized keys (4-bit) preserve the needle's rank, while mean-pooling destroys it.

3. **Localization depends on selector design**: The concentration of loss in a few heads is a *measurable property* of each selector, not a universal constant:
   - SeerAttention-R gates and NSA: few heads (5–22)
   - DashAttention: moderate set (13), led by the same three blind heads
   - MoBA: many heads (433–513 of 1152)
   - Token-level selectors: no such loss at all

4. **Integrated gradients > attribution patching**: The first-order (attribution patching) estimate misjudges heads that matter only after others are fixed; integrating along the path corrects this.

### Practical Implications

1. **Training-free repair**: No retraining needed — fixes act only on the top-k path of flagged heads, keeping the shipped kernel and cache.

2. **Safe default for unknown workloads**: The 4-bit scorer at decode tokens never loses more than a point on any of 17 task types and beats the shipped selector on 100% of 10,000 random task mixtures.

3. **Production viability**: Full attention in flagged heads preserves most of sparse attention's TTFT savings in production engines (CPM.cu).

4. **Cross-selector transferability**: The three blind heads transfer across different selectors on the same backbone (InfLLM-V2, NOSA, DashAttention, NSA), suggesting they are a property of the pretrained model, not the selector.

---

## Conclusion

### Main Takeaways

1. **Diagnosis precedes repair**: Exact mass selection at the shipped budget cleanly separates ranking failures from budget shortfalls — the needle is under-ranked, not lost.

2. **Three blind heads carry the loss** in the MiniCPM4-8B lineage, where mean-pooled key summaries dilute a single decisive token.

3. **One unified rule across selectors**: A threshold on the fix gain, swept per checkpoint to close 90% of the gap to full attention, identifies the responsible heads on every selector design.

4. **Training-free fixes are practical**: Full attention in flagged heads, a 4-bit scorer at shipped budget, or a larger per-layer budget each recover most of the multi-key loss, with the decode-token scorer being the safe default for unknown workloads.

### Future Directions

- **Independent backbones with InfLLM-V2 selector**: The three blind heads belong to one lineage sharing a backbone; validation on independently pretrained checkpoints is needed.
- **Other task types**: The fixes are tested on retrieval-heavy tasks; broader evaluation on aggregation-heavy tasks (where fixes can hurt) is warranted.
- **Larger contexts**: Beyond native ranges (e.g., 64K for a 32K-native model), no fix reaches full attention, and budget itself becomes limiting.
- **Real-data benchmarks**: The flagged heads recover losses on synthetic prompts and partially on MRCR; broader real-text evaluation is needed.

---

_Markdown view of https://picx.dev/p/0GZn7A, served by PicX — AI-generated visual whiteboard summaries of research papers._
