Summary (Overview)

• Core Question: The paper asks whether new attention mechanisms (gated attention, linear attention, attention residuals) that fix attention sinks at 128K tokens still work at the million-token context scale, and whether fixing the sink also fixes uneven context reading (recency bias).
• Key Contribution: The authors build SinkProbe, a four-metric diagnostic suite measuring sink mass, massive activations, position-resolved recall, and the recency gap, and validate it on four controlled ~1M parameter models.
• Main Findings: (1) The training objective—not the architecture—creates the attention sink; (2) gated attention did not reproduce its published sink-reducing effect at small scale because the gate never learned to be sparse; (3) sink mass, activations, and position bias move independently, meaning they are separate problems requiring separate measurements.
• Practical Result: The Kimi K3 layer mix (69 of 93 layers with fixed-size recurrent states) needs ~24 GiB cache at 1M tokens versus ~2.5 TiB for a dense stack—a ~107× reduction.
• Protocol: The paper registers thresholds and measurement points in advance for evaluating released Kimi K3 weights, providing a commitment to avoid post-hoc interpretation.

---

Introduction and Theoretical Foundation

The Two Habits Limiting Long-Context Models

Attention Sinks: Softmax attention forces every head to distribute exactly one unit of attention budget across visible tokens. When a head finds nothing worth reading, it must still spend its budget—typically on the first position (a beginning-of-sequence marker). This is called the attention sink. Qiu et al. measured a 15B model sending 46.7% of attention to the first token (one layer reaching 83%).

Uneven Context Reading: Where a fact sits in the context changes whether the model finds it. Liu et al. demonstrated a U-shaped accuracy curve (lost in the middle), with both ends of the context answered well and the center poorly. The authors formalize this as the recency gap:

$$\Delta_r = \mathbb{E}_{d \in (0.75, 1]}[p(d)] - \mathbb{E}_{d \in [0, 0.25]}[p(d)]$$

where $p(d)$ is recall probability when evidence sits at depth $d \in [0,1]$ (0 = opening, 1 = just before the question).

Why Existing Evidence Stops Short

• Gated attention (Qiu et al., NeurIPS 2025): A small input-dependent gate on attention output cut first-token attention from 46.7% to 4.8% and massive activations from ~1053 to ~94—but only tested up to 128K tokens.
• Kimi K3: Uses Kimi Delta Attention (linear layers with fixed recurrent state), gated global layers, and Attention Residuals, advertising a 1M token window—but reports only benchmark scores, not sink diagnostics.

The gap: Nobody has run sink diagnostics at million-token scale on the architectures designed to fix them.

Research Question

When a model replaces softmax attention with gated and linear mechanisms and extends its window to one million tokens, does the attention sink disappear, does the context start being read evenly, and are those two outcomes the same event or two different ones?

---

Methodology

SinkProbe Diagnostic Suite

Four metrics form the core of the suite:

1. Sink Mass (Equation 1): The share of attention landing on the first position:

$$\sigma = \frac{1}{|\mathcal{G}| H (T-1)} \sum_{\ell \in \mathcal{G}} \sum_{h=1}^{H} \sum_{t=2}^{T} A_{t,1}^{(\ell,h)}$$

where $\mathcal{G}$ is the set of softmax layers, $H$ is heads per layer, $T$ is context length, and $A_{t,1}^{(\ell,h)}$ is attention from query $t$ to key position 1.

2. Massive Activation (Equation 2): The largest absolute hidden state value:

$$\mu = \frac{1}{L+1} \sum_{\ell=0}^{L} \max_{t,j} |h_\ell[t,j]|$$

3. Attention Entropy (Equation 3): Measures whether attention is spread or concentrated:

$$\mathcal{H} = -\frac{1}{|\mathcal{G}| H (T-1)} \sum_{\ell,h,t} \sum_{s \leq t} A_{t,s}^{(\ell,h)} \log A_{t,s}^{(\ell,h)}$$

4. Recency Gap (Equation 4): As defined above, with Wilson score intervals.

The Model Ladder

Four models (~1M parameters each) differing only in token/depth mixing:

Linear Layer Design (Equation 5)

$$S_t = \text{Diag}(a_t) S_{t-1} + k_t v_t^\top, \qquad o_t = (S_t^\top q_t) \odot g_t$$

where $a_t \in (0,1]^{d_k}$ is a channel-wise decay, $g_t$ is an output gate, and $S_t \in \mathbb{R}^{d_k \times d_v}$ is the fixed-size recurrent state. The paper notes this is a simplification—the real Kimi Delta Attention includes a delta-rule correction term that was omitted.

Attention Residuals (Equation 6)

$$\alpha_i^\ell = \frac{\exp(w_\ell^\top \text{RMSNorm}(b_i))}{\sum_j \exp(w_\ell^\top \text{RMSNorm}(b_j))}, \qquad h_\ell = \sum_i \alpha_i^\ell b_i$$

where $b_i$ are block-level representations (sums of layer outputs within blocks) and $w_\ell$ is a learned pseudo-query.

Experimental Design

• Training length: 96 tokens; evaluation: 96, 192, 384, 768 tokens (8× extrapolation)
• Synthetic retrieval task: haystack with planted key-value pairs, distractors, and a query
• 3 seeds per model; Wilson score intervals for all proportions
• Thresholds registered in advance (Table IV): e.g., sink mass "fixed" if below 0.05, "not fixed" if above 0.20

---

Empirical Validation / Results

The Sink is Made by the Objective

Table VII shows the pivotal control experiment—same architecture, same data, only the loss changes:

A factor of 6× reduction in sink mass from changing only the training objective, with recall unchanged (the sink was not paying for retrieval).

Every Diagnostic, Every Model, Every Length

Table VIII (key rows at training length 96 tokens):

Critical caveats:

• Seed spread is enormous: baseline sink masses ranged from 0.089 to 0.455 across seeds
• Hybrid models did not learn the task—their recall (6.0%, 5.7%) is below blind chance (8.3%), and far below the 25% floor for returning any value in context

Why Gating Did Not Reproduce Published Results

The gate mechanism failed to reduce sink mass at this scale. The suite explains why:

• Published gate mean score (15B model): 0.116 (gate closed most of the time)
• Our gate mean score: 0.686 (gate open most of the time)

The mechanism was present in the architecture but absent from the trained weights—the gate never learned to be sparse.

Massive Activations: Clear Separation

• Gate halved activations: 74.9 → 36.2 (−52%) while sink mass fell only 17% (within seed spread)
• Attention Residuals cut activations by 60%: 97.9 → 38.9 while sink mass did not fall at all (0.183 → 0.202)

This confirms the dissociation: sink mass and activations are independent quantities.

Position Bias: Large, Flips Direction, Independent of Sink

At the training length (96 tokens), the opening is the strong end (negative recency gap: −10.9 for baseline). At 768 tokens, the picture reverses:

The gated stack has the lower sink mass but the steeper position preference at 768 tokens—the separation this paper set out to test.

Cache Growth Model

With 69 of 93 layers holding fixed-size states, the hybrid design needs ~107× less cache at 1M tokens. The layer fraction (74.2%) requires no assumptions; byte counts assume latent rank 512 and 128×128 state per head across 96 heads.

---

Theoretical and Practical Implications

Two Problems, Not One

The paper provides three independent instances of dissociation:

1. Sink mass vs. recency gap: The model with lower sink mass had the steeper position preference
2. Sink mass vs. activations (twice): Gate and Attention Residuals each cut activations sharply while leaving sink mass unchanged
3. Published ablation agreement: Qiu et al.'s gate-after-value-projection cut activations 1053→125 while leaving sink at 0.297

"A model card that reports low first token attention has said something true and something narrow. It has not said that the middle of the context is being read."

What a Sink Is a Fact About

The sink is a property of a specific checkpoint and its training objective, not an architecture family. Gating removes the sink only once it learns to be sparse—which requires sufficient training budget. The presence or absence of a sink cannot be read off a design.

Practical Consequences

• For model selection: Long-document users need all three numbers (sink mass, activation magnitude, position profile)—none are currently published for models advertising the longest windows
• For architecture claims: A mechanism that works at 15B parameters and trillions of tokens may not transfer to small models trained briefly; small proxies cannot speak for frontier models
• For cost modeling: The cache arithmetic is robust—even doubling/halving the two assumed dimensions moves the total by less than 2×

---

Conclusion

Main Takeaways

1. The objective makes the sink: Changing only the training objective (full next-token prediction vs. single-answer) moved sink mass from 31.6% to 4.9% with recall unchanged—a 6× effect far exceeding seed spread.

2. Gating did not reproduce its published effect at small scale: The gate never learned sparsity (mean score 0.686 vs. published 0.116), demonstrating that small models cannot proxy for frontier-scale behavior.

3. Sink mass, activations, and position bias are three measurements, not one: Two mechanisms cut activations by 52–60% while sink mass stayed within seed spread; position bias flipped direction with length and did not follow sink mass ordering.

4. The question in the title remains open: It cannot be closed with laptop-scale models; the protocol and thresholds are ready to be executed on released Kimi K3 weights.

Future Directions

• Add the delta-rule correction to the linear layer so the pilot matches the real Kimi mechanism
• Extend beyond synthetic retrieval to realistic long-context tasks (summarization, multi-hop reasoning, code navigation)
• Run the suite in languages other than English, where gaps are already known to be larger

Closing Reflection

"The sink is treatable and is being treated. But it was never the whole problem. A model that no longer wastes attention on its first token can still fail to read the middle of a document it was given, and nobody currently reports that second number. Until they do, a context window is a claim about capacity rather than about memory."