Efficient Sequence ModelingIssue 15Nov 14 – 21, 2026

Who Decides What to Keep: Decision Attribution in Sparse Attention

Highlights of This Issue

The most prominent signal this issue: the "decision attribution" of sparse attention and KV eviction has advanced from the selection criterion itself to "who decides"—which head, which layer, which layer ordering. Lost in a Few Heads for the first time localizes block-sparse selection failures to specific attention heads (blind heads), using teacher-forcing calibrated probes to simultaneously read the "needle attention abandoned by the selector" and the "share of answer log-probability recovered by fixing that head's selection" (integrated gradients along the path from the shipped selector to full attention), thereby proposing a training-free head-level repair (full attention or 4-bit scorer), and providing a unified threshold rule across InfLLM-V2/SeerAttention-R/NSA/DashAttention/MoBA—this converges with the no-op gate diagnosis of The Gating Mechanism of KV Cache Eviction (Issue 13) and the closed-form scoring of Sparse Attention Is Matrix Approximation (Issue 12), advancing failure attribution from selection criteria to per-head mechanisms. Complementarily, Who Should Score the Cache for the first time establishes "which layer scores" as an independent design axis for KV eviction: shallow layers collapse, middle layers are optimal, and deep layers degrade, and proposes a parameter-free survival model to predict the depth curve from prefill attention, validated across 21 models (0.5B–109B), and provides a mechanistic explanation for the writer-depth penalty in shared-cache architectures (Gemma-4).

Hybrid architectures' "layer ordering" and "layer selection" each have a substantive advance. The Effect of Layer Ordering in Hybrid Architectures for the first time provides a task-specific formal characterization of layer ordering in hybrid architectures: for any layer configuration, it constructs a family of tasks that strictly outperform any other configuration under a fixed resource budget, and distinguishes "wrong ordering" (requiring exponential resources) from "insufficient layers but correct ordering" (only constant-factor disadvantage), proposing "included sub-architecture" as a deep design criterion—this converges with the dynamics study of Competing Learning (Issue 12) and the empirical comparison of Mechanics of Long-Context Hybrid Models (Issue 9), advancing layer ordering from empirical to provable resource separation. On-Policy Morphing into Hybrid Attention Models advances Transformer→hybrid conversion layer selection from off-policy (continuous gate interpolation + teacher-forced hidden states + frozen weights) to on-policy: sampling discrete hybrid architectures that satisfy the budget for end-to-end forward passes, and jointly optimizing layer selection scores and model weights with a recovery objective, eliminating two types of off-policy mismatch, with an average RULER improvement of 2.42 points.

Indexers and linear memory each have a systematic increment worth reading. Multi-layer Indexer advances the serial dependency of DSA indexers from "reducing the number of indexers" (IndexCache's cross-layer reuse) to "hiding the latency of remaining indexers": letting each layer's attention consume the selection of the previous full layer, with indexer and attention executing concurrently, requiring no retraining and no new weights, and combinable with IndexCache (up to 14.5% higher decode throughput at the same retention). ADIndex advances cross-query sharing to layer-adaptive calibration + distillation-driven group indexers, achieving 6.22× speedup on 256K prefill indexing on DeepSeek-V3.2/GLM-5.1. Toaster for the first time reconstructs the update of "deep memory" (TTT-MLP/Titans-like) as a cascade of two linear memories, using target propagation to provide target values for hidden layers, enabling each layer to be exactly advanced with unmodified WY kernels—turning "nonlinear memory" and "WY parallelism" from opposition to compatibility, with training speed close to KDA and quality better than Titans.

Community and Updates

  • StateBench proposes a benchmark for long-document state tracking, categorized into four tiers by algebraic complexity (r-trivial/aperiodic/soluble/finite), directly testing the abilities that recurrent/hybrid architectures (e.g., Gated DeltaNet) claim to excel at: from-scratch trained GDN and hybrid models extrapolate almost perfectly on r-trivial/aperiodic/soluble, while Transformers cannot learn general finite problems even with direct training—this is an evaluation axis not isolated by RULER/LongBench, directly relevant to this direction's concern about long-context evaluation validity.
  • Context Rot Quantified quantifies long-context degradation on agentic coding tasks, distinguishing degradation differences between task-relevant and irrelevant contexts, showing that some models degrade regardless of relevance while others degrade only on relevant contexts—this is an evaluation-side complement to the competitive attribution of Length Is Cheap (Issue 13).
  • How Fast Do Agents Rot? characterizes agentic long-range degradation as a geometric law (per-step reliability parameter saturates below 1, collapsing around 16 steps), providing priors for designing architectures that maintain long-range reliability.
  • Sparse Attention · Part II — Detach and Distill compares two indexer training schemes, DeepSeek DSA and MiniMax MSA, highlighting MSA's input-decoupling ablation and dense warmup/KL alignment recipe—useful as a tour of the trainable block-sparse indexer design space, but with no new experiments.

Open Questions

  1. The blind heads in Lost in a Few Heads come only from a single backbone lineage—are these blind heads consistent across different model families? Can their training-free head-level repair be combined with trainable indexers (e.g., AttentionIndex's causal IVF from Issue 14) into a "diagnose-then-repair" closed loop?
  2. Who Should Score the Cache establishes scoring layer depth as a design axis, but its gains disappear above 32B—is there an interaction between scoring layer depth and the layer-ordering resource separation of The Effect of Layer Ordering? Is "middle-layer optimal" just a small-model phenomenon?
  3. Toaster's target propagation restores WY parallelism for nonlinear memory, but efficiency evidence only goes to 1.6B—will target propagation encounter gated rank allocation issues at larger scales, like Muir's sensitivity allocation in Issue 13?
  4. On-Policy Morphing's Stage 2 weights are not used for final recovery (only guiding selection)—does on-policy selection still outperform off-policy at larger scales (>30B), and does its long-term benefit depend on weight adaptation?

Papers in this issue

  1. Sparse attention's long-context failures stem from selectors under-ranking the answer's block, not budget limits, and training-free fixes on a few flagged heads recover most accuracy.

    Editor's note

    For the first time localizes block-sparse selection failures to specific attention heads (blind heads), using teacher-forcing calibrated probes to simultaneously quantify the "needle attention abandoned by the selector" and the "share of answer log-probability recovered by fixing that head's selection" (integrated gradients along the path from the shipped selector to full attention), thereby proposing a training-free head-level repair (full attention or 4-bit scorer). Relative to the no-op gate diagnosis of The Gating Mechanism of KV Cache Eviction (Issue 13) and the closed-form scoring of Sparse Attention Is Matrix Approximation (Issue 12), the increment is advancing failure attribution from selection criteria to per-head mechanisms, and providing a unified threshold rule across InfLLM-V2/SeerAttention-R/NSA/DashAttention/MoBA with degradation curves by needle depth. Must-read for those building trainable block-sparse indexers or retrieval heads; limitation: blind heads come only from a single backbone lineage, no agentic long-context benchmarks.

  2. The optimal hybrid Transformer-SSM layer ordering is task-specific, with no universally best architecture, and containing the dominant sub-architecture predicts performance.

    Editor's note

    For the first time provides a task-specific formal characterization of layer ordering in hybrid architectures: for any layer configuration, it constructs a family of tasks that strictly outperform any other configuration under a fixed resource budget, and distinguishes "wrong ordering" (requiring exponential resources) from "insufficient layers but correct ordering" (only constant-factor disadvantage), proposing "included sub-architecture" as a deep design criterion. Relative to the dynamics study of Competing Learning (Issue 12) and the empirical comparison of Mechanics of Long-Context Hybrid Models (Issue 9), the increment is advancing layer ordering selection from empirical/dynamics to provable resource separation theorems. Must-read for those doing layer allocation in hybrid architectures; limitation: theoretical tasks are constructive, experiments only up to 8-layer small models, no long-context degradation curves.

  3. TOASTER achieves nonlinear-memory quality at near-linear-memory speed by converting a two-layer recurrent memory into cascaded delta-rule kernels via target propagation, outperforming KDA across all scales.

    Editor's note

    For the first time reconstructs the update of "deep memory" (TTT-MLP/Titans-like) as a cascade of two linear memories, using target propagation to provide target values for hidden layers, enabling each layer to be exactly advanced with unmodified WY kernels—restoring chunkwise parallelism while maintaining nonlinear expressiveness. Relative to Muir's sensitivity gating (Issue 13) and ParaTTT's iterative fixed-point solving (Issue 12), the increment is turning "nonlinear memory" and "WY form" from opposition to compatibility, with training speed close to KDA and quality better than Titans, with compute-matched scaling ladder and 3:1 hybrid comparison. Should-read for those doing hybrid linear attention or TTT-like deep memory; limitation: efficiency evidence only to 1.6B, single seed, no specialized kernels.

  4. Multi-layer Indexer overlaps indexer scan with attention in DeepSeek Sparse Attention, yielding up to 14.5% faster decode with no retraining by exploiting high overlap between adjacent layers' token selections.

    Editor's note

    For the first time advances the serial dependency of DSA indexers from "reducing the number of indexers" (IndexCache's cross-layer reuse) to "hiding the latency of remaining indexers": letting each layer's attention consume the selection of the previous full layer, with indexer and attention executing concurrently, requiring no retraining and no new weights, and combinable with IndexCache. Relative to AttentionIndex's indexer acceleration (Issue 14) and LatentIndex's cross-layer sharing (Issue 9), the increment is providing a drop-in scheduling-level overlap scheme, with up to 14.5% higher decode throughput at the same retention, with LongBench/RULER degradation curves. Should-read for those doing trainable block-sparse indexers or hybrid architecture inference; limitation: gains decrease with concurrent requests, fused kernels only effective for DeepSeek-V3.2, no >195K degradation curves.

  5. ADIndex accelerates DeepSeek Sparse Attention up to 6.22x by sharing token banks across neighboring queries via layer-adaptive calibration and distilled group indexers, with near-lossless accuracy.

    Editor's note

    For the first time advances DSA indexer acceleration from "cross-query sharing" to "layer-adaptive sharing + distillation-driven group indexers": using offline calibration to select query group size and shared token bank budget per layer, and training lightweight group indexers (only 1/8 heads) with group-level distillation to improve Top-K coverage. Relative to AttentionIndex's causal IVF (Issue 14) and PIVOT's training-free query aggregation, the increment is explicitly modeling inter-layer overlap heterogeneity and providing distillation-driven coverage gain evidence, achieving 6.22× speedup on 256K prefill indexing and 1.54× end-to-end on DeepSeek-V3.2/GLM-5.1. Should-read for those accelerating trainable block-sparse indexers; limitation: no compute-matched comparison with full attention, long-context evaluation mainly on RULER.

  6. FlashMorph2 fixes two off-policy mismatches in FlashMorph's layer selection, improving hybrid attention model conversion by up to +2.42 RULER points with under 4 GPU hours of search.

    Editor's note

    For the first time advances layer selection in Transformer→hybrid attention conversion from off-policy (continuous gate interpolation + teacher-forced hidden states + frozen weights) to on-policy: sampling discrete hybrid architectures that satisfy the budget for end-to-end forward passes, and jointly optimizing layer selection scores and model weights with a recovery objective. Relative to FlashMorph's continuous gates + frozen weights (Issue 14), the increment is explicitly eliminating two types of off-policy mismatch and providing ablation evidence that both are complementary (removing on-policy computation drops RULER by 29.55), with an average RULER improvement of 2.42 points. Should-read for those doing Transformer→hybrid conversion layer selection or hybrid architecture layer allocation; limitation: evaluation only up to 8B/30B-A3B, no agentic long-context benchmarks.

  7. The depth of the scoring layer in KV-cache eviction is a critical ignored hyperparameter, with mid-depth scoring beating default methods by up to 43 points on models up to 14B.

    Editor's note

    For the first time systematically studies the overlooked hyperparameter of "which layer scores" in KV cache eviction: scoring layer depth determines eviction quality, shallow layers collapse, middle layers are optimal, and deep layers degrade, and proposes a parameter-free survival model to predict the depth curve from prefill attention and select the scoring layer. Relative to existing per-layer self-scoring eviction methods and cross-layer shared caches that default to writer scoring, the increment is establishing "scoring layer depth" as an independent design axis, and providing mechanistic evidence for the penalty caused by writer depth in shared caches, validated across 21 models (0.5B–109B, including hybrid attention Qwen3.5-9B). Should-read for those doing KV eviction selection criteria; limitation: gains disappear above 32B, no gains on LongBench, no compute-matched comparison with full attention.

  8. HiSA sparsifies the hidden dimension of RoPE attention, computing only ~30% of dot products while achieving lower validation loss than dense attention.

    Editor's note

    For the first time advances the sparsity axis of sparse attention from token dimension to hidden (feature) dimension: based on RoPE frequency-pair structure, statically and parameter-free truncating high-frequency pairs in query-key dot products, giving each frequency pair an adaptive window that grows with distance, with a FlashAttention-3 kernel and KV cache chunked ring buffer. Relative to token-level block-sparse indexers like AttentionIndex (Issue 14) and Recursive Sparse Attention (Issue 13), the increment is providing a fully learning-free, routing-free hidden-dimension sparsity rule, and proving that dense attention actively suppresses the discarded high-frequency pairs during training (E_real/E_null ratio evidence). Should-read for those doing sparse attention architectures; limitation: only 135M/360M scale, long-context evaluation only up to 16K and no retrieval-type benchmarks.