Diversification of Selection Signal Sources and the Dispute over Sink Origins
Highlights of This Issue
The strongest signals this issue are the simultaneous advancement of cross-layer KV sharing and the diversification of selection signal sources. HySparse2 unifies YOCO-style cross-layer KV sharing (KV Bridging) with intra-block KV reuse (KV Reuse) into a two-level sharing scheme, and for the first time places token-level sparse selection and forced local windows into the same KV cache, allowing prefill to fully early-exit after the self-decoder—providing compute-matched comparisons and degradation curves on RULER-v2/MRCR-v2/AgentPPL/LongPPL for 80B-A3B and 290B-A8B. This directly responds to the cross-layer KV and index reuse approach of DeepSeek-V4.1-Flash from Issue 6, but advances from "decoupling cache sharing and index reuse" to a complete design where "inter-layer sharing and token-level selection are jointly covered."
The sources of selection signals have seen a notable diversification. Language Models Can Control Their Own Attention lets the model explicitly declare <global>/<focus>/<local> tags in the CoT, and the inference engine constructs block-level masks accordingly, completely eliminating the per-query O(N) selection cost—this is a new axis orthogonal to the trainable Forecast indexer of SparDA from Issue 1 and the query-group shared index of PIVOT from Issue 6. DeltaS uses the state drift of the recurrent state in mixed linear attention itself as a training-free retention signal for KV eviction in full-attention layers, allowing the two memories of hybrid architectures to cooperate. CompKV is the first to jointly optimize selection and tail compensation, proposing the p·σ² selection criterion.
A mechanistic dispute over the origins of sinks needs reconciliation. Elastic Threshold Attention uses multiplicative gates to establish a uniform attention floor, claiming that local attention sinks completely disappear during pretraining; whereas SinkProbe provides controlled small-model experiments with the opposite negative result—training objectives rather than architecture produce sinks, and gating did not reproduce its published effects. The scale difference is vast, but both point to sinks not being a single problem. Complex KDA advances delta-rule linear attention from the expressivity side, proving that a single-layer CKDA can track all finite subgroups of SO(3).
Community and Developments
On the engineering practice side of hybrid architectures, two signals are noteworthy. Zebra-HyLo provides a complete recipe for upcycling a pretrained Transformer into an MLA+Gated DeltaNet/Mamba-2 hybrid, explicitly reporting that "an 8K upcycled model drops from 55.1 to 0.8 on RULER at 64K"—long-context capability must be a training objective, not a free property of the architecture—and includes configurable layer ratios and real degradation curves. Optimizing Native Sparse Attention with Latent Attention and Local-Global Alternating Strategies alternates local/global layers on top of NSA and introduces latent attention, matching or exceeding full attention at 340M-1.3B with 50% KV reduction, a direct increment on the inter-layer ratio design axis.
On the evaluation validity side, The Pitfalls of KV Cache Compression provides an important negative result: under multi-instruction prompts, eviction causes non-uniform "selective amnesia" and system prompt leakage, while aggregate benchmarks mask these failures—it directly warns about the fragility of "negligible loss" conclusions, consistent with this issue's caution about weak needle-in-a-haystack evidence.
Open Questions
- HySparse2's KV Bridging only bridges full-attention layers, and prefill fully early-exits after the self-decoder—how does this "candidate domain limited by the self-decoder" mechanism compare with the Hierarchical Sparse Indexer candidate pool of DeepSeek-V4.1-Flash from Issue 6? Does early exit introduce new alignment biases?
- The explicit text declaration of Language Models Can Control Their Own Attention, the implicit threshold prediction of Elastic Threshold Attention, and the log-space gate injection of SAS from Issue 6—how do these three "let the model decide sparsity" mechanisms trade off in training stability and selection fidelity? Can explicit declaration complement trainable indexers?
- DeltaS uses recurrent state drift to guide KV eviction, and CompKV uses compensation residuals to guide selection—can "state new information content" and "compensation error" as retention signals be unified under the 'Trust the Mass' selection ceiling framework from Issue 3?
- SinkProbe and Elastic Threshold Attention give opposite conclusions on sink origins—is this a scale effect (1M vs 1.45B) or a mechanism difference (multiplicative floor vs gating)? Can the developmental panel of 'When Do Attention Circuits Form?' from Issue 3 arbitrate this dispute at larger scales?
Papers in this issue
HySparse2's two-level KV sharing cuts prefill FLOPs by 2.92x and KV cache to 2.69 GB at 1M tokens while boosting long-context accuracy by up to 19.81 points.
Editor's noteUnifies YOCO-style cross-layer KV sharing (KV Bridging) with intra-block KV reuse (KV Reuse) into a two-level sharing scheme, and for the first time places token-level sparse selection and forced local windows into the same KV cache, allowing prefill to fully early-exit after the self-decoder. Relative to the cross-layer KV and index reuse of DeepSeek-V4.1-Flash from Issue 6, the increment is jointly covering both inter-layer sharing and token-level selection dimensions, providing compute-matched comparisons and degradation curves on RULER-v2/MRCR-v2/AgentPPL/LongPPL for 80B-A3B/290B-A8B; the comparison between KV Bridging and KV Mirror also reveals the advantage of full-attention layers as KV sources.
CKDA extends KDA with signed gates and beta up to 2, enabling complex eigenvalues and rotations that match DeltaProduct expressivity with a single delta-rule update per token.
Editor's noteBy relaxing the KDA gate to [-1,1] and the delta rule coefficient to [0,2], a single diagonal-plus-rank-one transition can implement 2D rotations and complex eigenvalues, proving that every orthogonal DPR1 matrix is exactly a CKDA transition. Relative to the scalar gates of DeltaNet/GDN, the increment is showing that channel-wise gates themselves can provide reflections, avoiding extra rank-one updates, and a single layer can track all finite subgroups of SO(3) (including A5) with one fewer layer than GDN. Provides compute-matched comparisons and scaling analysis for state-tracking and language modeling, but long-context evaluation is limited to periodic waveform continuation.
Declarative Attention lets off-the-shelf LLMs declare their own sparse attention scope via text tags, cutting attended tokens by up to 52% with minimal accuracy loss.
Editor's noteFor the first time advances the selection signal for sparse attention from internal activations/proxy scores to an explicit text protocol declared by the model: the model outputs <global>/<focus>/<local> tags in the CoT, and the inference engine dynamically constructs block-level masks accordingly, completely eliminating the per-query O(N) selection cost. Relative to the trainable Forecast indexer of SparDA from Issue 1 and the query-group shared index of PIVOT from Issue 6, the increment is making selection itself part of the model's output (system-2 sparse attention), with zero training, zero auxiliary scorers, and providing compute-matched comparisons and degradation curves across 15 tasks and multiple model scales. The limitation is that the zero-shot protocol adds about 1/3 decoding steps and efficiency claims are based on roofline.
Elastic Threshold Attention learns query-conditioned sparse masks via multiplicative logit suppression, matching dense quality at 38% active density while achieving 2.5x faster decoding.
Editor's noteUses multiplicative sigmoid gates to shrink unselected logits toward zero rather than -inf, establishing a uniform attention floor, thereby eliminating local attention sinks during pretraining and making the model robust to block over-inclusion at inference. Relative to the log-space gate injection of SAS from Issue 6 and the differentiable selection of HiLS from Issue 5, the increment is providing the mechanistic claim that a "uniform floor eliminates sinks," along with a query-conditioned per-head threshold predictor and an O(1) block filtering kernel. Provides comparisons with full attention at 1.45B, but long-context evidence is limited to needle-in-a-haystack scores, so quality claims should be treated cautiously.
DeltaS uses the normalized change in gated-delta linear attention's recurrent state as a query-agnostic retention signal, outperforming all bounded-memory baselines on six long-video benchmarks with minimal overhead.
Editor's noteFor the first time uses the state drift of the recurrent state in mixed linear attention (Gated DeltaNet) itself as a training-free, query-agnostic retention signal for KV cache eviction in full-attention layers, and under fixed budgets and retention policies, compares against position/attention/key-value signals in a controlled manner. Relative to the static retention horizon of DASC from Issue 6 and the confidence states of Kalman Delta Networks from Issue 5, the increment is directly using the "new information content" of the recurrent state to guide retention of another memory (KV cache), providing real degradation curves on long-video benchmarks. The limitation is that it uses only a single hybrid backbone and evaluation is in the video domain.
CompKV is the first sparse attention framework that selects KV blocks based on downstream compensation error, achieving near-full-attention accuracy with up to 6.85x speedup.
Editor's noteFor the first time jointly optimizes selection and tail compensation (mean compensation) in block sparse attention: theoretical analysis proves that the post-compensation residual is determined by both block attention quality and intra-block logit variance, leading to the p·σ² selection criterion. Relative to the top-mass near-optimality proof of 'Trust the Mass' from Issue 3, the increment is explicitly modeling the interaction between selection and compensation, providing closed-form KL derivations and being training-free. Provides comparisons with Quest/InfLLM on RULER/LongBench-Pro, but efficiency claims depend on a CPU-offload system implementation and no real long-context degradation curves are reported.
The training objective, not architecture, creates attention sinks, and sink mass, activations, and position bias are independent problems requiring separate diagnostics at million-token scale.
Editor's noteProposes the SinkProbe diagnostic suite (sink mass, massive activation, positional parse recall, recency gap), testing on four small models differing only in mixing method whether gating and mixed attention truly fix sinks at million-token contexts. The key negative result is that training objectives rather than architecture produce sinks, gating did not reproduce its published effects at our scale, and sink mass, activations, and positional biases move independently—directly challenging the assumption since 'When Do Attention Circuits Form?' from Issue 3 that sinks are a single problem. The limitation is the small scale of 1M parameters and 768 tokens.
