Sliding-window beats linear attention: the premise of post-training linearization is shaken
Highlights of this issue
The strongest signal this issue comes from a counterintuitive negative result: Sliding-window beats linear attention is the first systematic comparison of post-training linearized models against training-free sliding-window attention (with sink), finding that SWA recovers 99% of baseline average performance on short-context tasks, is 2-10x higher on long-context reasoning, and requires zero post-training and zero specialized kernels. This corroborates the "lazy large window" from Issue 1: efficient attention is more about optimization priors than long-range information carriers, and the premise of post-training linearization itself is challenged. Modern Transformers Are Implicit Hybrids offers a constructive way out from the mechanism side—using interventional metrics RFIS/RPD to dichotomize retrieval heads and positional heads on the RoPE frequency axis, discovering a GPBand functional boundary that shifts with training length scale, and proposing head-granularity, layer-specific hybrid design principles (NoPE FA for global retrieval, LA for local positional modeling), advancing "efficient attention as optimization prior" from observation to actionable architecture design.
On the KV compression side, Still is the first to advance compression from per-context synthesis to amortized synthesis—using a per-layer Perceiver to synthesize compact key/value in a single forward pass, breaking the subset ceiling of selection methods and supporting iterative chunked compression for long-range scenarios; Why Gated DeltaNet Survives 4-Bit Quantization proves that the recurrent half of hybrid LLMs can be losslessly quantized under NVFP4 W4A4, providing a four-part mechanistic explanation (block scaling, gate nonlinearity compression, delta rule erasure, context flushing), and identifies the global scale mismatch between fused-GEMM and per-module calibration as a serving pitfall. On the inference side, Faster Than Flash uses top-δ local pseudo-maxima to replace global softmax synchronization for distribution-adaptive sparsity, while CRISP proves that cumulative coverage thresholds accumulate O(n) background noise under long contexts and provides a sink-aware threshold fix.
Evaluation validity continues to tighten. Benchmarking the Residual proposes the "horizon residual" Γ_H—the log ratio of a product baseline matching short-stage success probabilities to actual end-to-end success rates—decomposing "longer is harder" into work exposure, dependency depth, and local difficulty, advancing from degradation curves to a falsifiable diagnostic framework. On the community side, Is Context Rot Real pre-registered cross-provider null results show no measurable length-driven degradation for simple probes within 150k tokens, creating tension with the mainstream degradation curve narrative, suggesting "context rot" may be task-dependent rather than universal.
Community and developments
The retrieval-head-as-efficient-attention-prior line continues to ferment on the ACL side. RetMask directly links retrieval head sparsity to efficiency gains in long-context optimization, while EvoSparse uses the model's intrinsic retrieval heads as a prior for sparse index propagation across layers, grounding propagation mechanisms in interpretable signals rather than heuristic similarity—complementing the frequency-axis interventional metrics of Modern Transformers this issue, but both lack compute-matched controls and degradation curves, so efficiency claims remain preliminary.
On the evaluation validity side, RULER vs Needle-in-a-Haystack reveals that GPT-4o's effective context length collapses to about 8K under strict NoLiMa rules, and NeedleChain advances evaluation from single-needle retrieval to multi-hop integration, showing models fail on 200-token relevant contexts. On the KV compression side, OjaKV uses online Oja-PCA to adaptively low-rank subspaces, explicitly addressing drift in static low-rank methods, and pairs it with full-rank attention-sink anchors.
Open questions
- If training-free sliding-window attention already beats post-training linearization (Sliding-window beats linear attention), should the value of linearization shift entirely to training-time hybrid design from scratch? Does "lazy large window" imply that SWA's advantage is itself an optimization prior artifact that disappears with sufficient training?
- Last issue's Trust the Mass proved top-mass selection is near-optimal, while this issue's Still amortized synthesis breaks the subset ceiling—does the 8-22 point advantage from synthesis come from escaping the subset ceiling or from synthesis quality itself? Can synthesis and selection be combined?
- The robustness of Gated DeltaNet quantization is explained by delta rule erasure—does this imply the recurrent half's state is inherently low-rank compressible, and can this property be generalized from quantization to KV state compression?
- The null result of Is Context Rot Real conflicts with the mainstream degradation curve narrative—does the discrepancy stem from probes being too simple, or is "context rot" inherently task-dependent rather than universal? This directly affects whether long-context evaluation should use simple probes or complex tasks.
Papers in this issue
Still is a per-layer Perceiver compactor that amortizes KV cache synthesis in one forward pass, beating selection baselines by 8-22 accuracy points across 8x-200x compression.
Editor's noteFirst to advance KV cache compression from per-context synthesis to amortized synthesis: using a per-layer Perceiver to synthesize compact key/value in a single forward pass, breaking the subset ceiling of selection methods and supporting iterative chunked compression for long-range scenarios. Relative to KV-Distill (amortized selection) and Cartridges/Attention Matching (per-context synthesis), its increment is achieving both synthesis and amortization, occupying the speed-quality frontier at 8x-200x compression and 8k-128k contexts, and exceeding the strongest baseline by 8-22 points on the long-context RULER grid, with real degradation evidence for cross-scale/architecture transfer and HELMET summarization retention.
Sliding window attention with attention sinks, requiring zero post-training, outperforms post-trained linear attention methods on both short and long-context tasks while being faster and more memory-efficient.
Editor's noteFirst systematic comparison of post-training linearized models against training-free sliding-window attention (with sink): SWA recovers 99% of baseline average performance on short-context tasks, is 2-10x higher on long-context reasoning, and requires zero post-training and zero specialized kernels. It directly challenges the premise of post-training linearization, corroborating the 'lazy large window' from Issue 1—efficient attention is more about optimization priors than long-range information carriers. Provides compute-matched controls from 1.3B to 70B and real degradation curves for S-NIAH/BABILong, a must-read negative result for anyone doing hybrid architecture or linearization research.
RoPE Transformers implicitly separate retrieval and positional functions across heads, and a head-wise hybrid architecture assigning NoPE attention to global retrieval and linear attention to local position enables zero-shot length extrapolation.
Editor's noteUses interventional metrics RFIS/RPD to establish a complete dichotomous classification of retrieval heads and positional heads on the RoPE frequency axis, discovering a GPBand functional boundary that shifts with training length scale, and proposes head-granularity, layer-specific hybrid design principles (NoPE FA for global retrieval, LA for local positional modeling). Relative to Issue 2's 'Does RoPE Prevent or Degrade Retrieval Heads?' frequency-axis mechanism analysis, its increment is advancing frequency dependence from norm approximations to faithful interventional metrics, and directly instantiating it as a pre-trainable HwH hybrid architecture, with compute-matched controls for FA:LA<1:3 and zero-shot length extrapolation degradation curves.
NVFP4 W4A4 quantization matches BF16 accuracy on a hybrid 27B LLM because Gated DeltaNet's gating and delta-rule recurrence make it more robust to quantization than attention layers.
Editor's noteFirst systematic proof that the recurrent half of Gated DeltaNet in hybrid LLMs can be losslessly quantized under NVFP4 W4A4, providing a four-part mechanistic explanation (block scaling, gate nonlinearity compression, delta rule erasure, context flushing), and identifying the global scale mismatch between fused-GEMM and per-module calibration as a serving pitfall. Relative to Issue 2's KV-CAT training-time compressibility incentives, it is pure PTQ and focuses on the quantization robustness of the recurrent operator itself, with increments in causal decomposition at the mechanism level and serving-stack measurement pitfalls, paired with 4K/32K perplexity and RULER position-decomposed degradation curves.
Faster Flash Decoding uses 2-bit quantization and a top-delta selection rule to achieve up to 11.6x kernel speedup on 256K-context LLMs without accuracy loss.
Editor's noteProposes a top-δ selection strategy, using local pseudo-maxima (sink+local) to replace global softmax synchronization for distribution-adaptive sparsity that is parallelizable, and uses 2-bit quantized K cache as a proxy for content-aware scanning, fusing selector and calculator kernels. Relative to Quest's top-k fixed budget and Twilight's top-p global synchronization, top-δ is a new selection mechanism; provides RULER/LongBench degradation curves and compute-matched controls, a complementary contribution to the training-free route, contrasting with trainable indexer routes.
CRISP replaces indirect JSD routing and cumulative coverage thresholds with a structural-mass proxy and sink-aware noise-floor threshold, achieving parity with dense attention and up to 5.30x speedup at 512k tokens.
Editor's noteIdentifies a specific failure mode of cumulative coverage thresholds in dynamic sparse attention—the post-softmax mass cliff—replacing the JSD routing signal with the structural proxy C_struct, and proves that strict cumulative coverage thresholds accumulate O(n) background noise under long contexts. Relative to Issue 2's COBS second-order cumulant selector, it targets the threshold mechanism itself rather than selector order, providing a sink-aware calibration-free fix, recovering up to +28.0 pp on retrieval tasks and achieving 5.30x attention speedup at 512k, with RULER/InfiniteBench/LongBench degradation curves.
The horizon residual, a log-ratio of composed short-stage success to end-to-end rollout success, diagnoses true long-horizon failures by separating them from ordinary error compounding.
Editor's noteProposes the 'horizon residual' Γ_H—the log ratio of a product baseline matching short-stage success probabilities to actual end-to-end rollout success rates—to distinguish error accumulation from true long-range failure mechanisms, and decomposes 'longer is harder' into work exposure, dependency depth, and local difficulty. Relative to Issue 3's ATLAS length-aware AUC and Distractor-Aware Truncation, its increment is an explicit compositional counterfactual baseline and pre-registered checkpoint protocol, advancing degradation curves from description to a falsifiable diagnostic framework, a methodological blueprint for long-range evaluation validity research.
