Issue 1: The Compound Effects of Architectural Priors and Cross-Layer Reuse of Indexers
Highlights of This Issue
This week's research on efficient sequence modeling presents three clear threads. The first is the "compound effects of architectural priors": Cracks in the Foundation uses 26 7B models with fully aligned data and optimization in a four-factor ablation, demonstrating that combinations of "small choices" such as QK norm, GQA, sliding window, and pretraining length can suppress long-context scalability by up to 47% in an additive manner, while short-context metrics are almost unable to predict long-context performance—sounding an alarm for evaluation validity across the field. In a related vein, Rethinking the Role of Efficient Attention proposes the "large-window laziness" phenomenon, suggesting that efficient attention primarily acts as an optimization prior shaping the formation trajectory of retrieval heads rather than directly carrying long-range information, representing the negative-result category this direction aims to collect.
The second thread concerns indexers and KV compression moving toward "cross-layer reuse" and "theorization." You Only Index Once extends the reuse of KV-sharing architectures from memory to routing, allowing a single indexer's top-k results to be shared across multiple cross-decoder layers; LongCat Sparse Attention and SparDA respectively address the overhead and memory-access bottlenecks of indexers from system-level (streaming-aware indexing, cross-layer distillation) and decoupled look-ahead (Forecast projection) directions. On the KV compression side, NOVA-KV for the first time derives a closed-form non-orthogonal optimal transform from attention product distortion, converting the attention-aware objective into MSE, providing quantization methods with the theoretical foundation previously lacking in heuristic approaches like QuaRot/OSCAR.
The third thread is a re-examination of evaluation validity and training objectives. How Query Visibility Changes KV-Cache Compression Rankings uses a matched-budget protocol to prove that query visibility can flip the rankings of compression methods and identifies attention backend confusion as a reproducible evaluation pitfall; Does Accuracy Equal Evidence? reveals the "answer-evidence gap"—compressed models can preserve the final answer while quietly undermining the faithfulness of the reasoning chain. Information Abundance Paradox offers a counterintuitive conclusion from the training side: richer informational context actually weakens the model's incentive to parameterize knowledge. These works collectively point to a single direction: single-point scores (whether NIAH or accuracy) are insufficient to support "lossless" conclusions.
Community and Developments
On the systems side, LMSYS's Unified Radix Cache uses a single shared radix topology to simultaneously support three types of reuse semantics—full attention, sliding window, and Mamba checkpoints—so that hybrid models no longer need to maintain separate cache trees for each attention combination, and natively incorporates HiCache's multi-tier storage into the component lifecycle. In evaluation discussions, benchr's The million-token context: a marketing number continues the critique of saturation in needle-in-a-haystack metrics, noting that synthetic tasks degrade before retrieval ability beyond 500K, and advocates positioning long context as a supplement to retrieval rather than a replacement.
Two ACL 2026 papers touch on two other axes of this direction: Threshold Differential Attention uses learnable thresholds combined with differential attention to explicitly remove attention sinks while maintaining non-diffuse ultra-sparse patterns; Native Hybrid Attention introduces per-token learnable gating between linear attention and softmax attention, advancing hybrid designs from inter-layer allocation to intra-head mixing. The former directly responds to the architectural axis of "deliberately avoiding sinks," while the latter provides a new intra-layer mixing dimension for hybrid architecture design.
Open Questions
- If short-context metrics (loss, perplexity, in-loop) systematically fail while long-context evaluation is contaminated by query visibility and backend confusion, what kind of "minimally sufficient" evaluation protocol can stably predict long-context degradation within a budget-constrained setting? Can the matched-budget protocol of the Query Visibility audit be generalized into a community standard?
- Cross-layer index reuse (CLSA, LongCat) amortizes indexing costs across multiple layers, but can the KL alignment between the indexer and the main attention distribution still preserve the specificity of retrieval heads in each layer after cross-layer sharing? Will shared indexing reproduce the "large-window laziness"-style delay in retrieval head formation?
- NOVA-KV's attention-aware non-orthogonal transform is theoretically superior to orthogonal rotations, but when combined with trainable indexers (rather than pure quantization), is there provable synergy or conflict between the attention distortion objective and the indexer alignment objective?
Papers in this issue
Four architectural choices—normalization, GQA, sliding window attention, and pretraining context—compound to slash long-context performance by up to 47%, undetectable via short-context metrics.
Editor's noteConducts a four-factor ablation on 26 7B models with fully aligned data and optimization, demonstrating that combinations of "small architectural choices" such as QK norm, GQA, sliding window, and pretraining length suppress long-context scalability by up to 47% in an additive manner, and short-context metrics cannot predict long-context performance. It provides real degradation curves for HELMET/RULER/LongPPL rather than NIAH scores, and releases the OlmPool model suite, serving as an essential comparison benchmark for long-context scaling and efficient attention research.
Long-context training shifts models from internalizing knowledge in weights to relying on context, degrading performance when context is absent or misleading.
Editor's noteProposes the "information abundance paradox": the richer the training context, the less incentive the model has to parameterize knowledge, instead relying on context, leading to performance degradation when context is missing or misleading at inference. It provides compute-matched comparisons across context lengths with degradation curves, and offers mechanistic evidence from gradient allocation, module-constrained fine-tuning, and attention allocation, directly relevant to the design of long-context pretraining curricula.
Cross-Layer Sparse Attention shares a single top-k routing index across decoder layers, delivering up to 7.6× decoding speedup at 128K context with near-lossless quality.
Editor's noteExtends the reuse of KV-sharing architectures (YOCO-like) from memory to routing—a single indexer computes token-level top-k once, and the results are shared across multiple cross-decoder layers, amortizing the expensive per-token routing cost. It provides compute-matched comparisons with dense baselines and degradation curves for RULER/validation loss, representing a key work on the new axis of cross-layer index reuse.
NOVA-KV derives non-orthogonal optimal transforms for KV cache quantization, enabling fixed-width vector quantization that preserves attention accuracy at 2 bits, outperforming prior methods on long-context retrieval.
Editor's noteFor the first time derives a closed-form non-orthogonal optimal transform for KV quantization from attention product distortion, proving that the optimal key transform satisfies a generalized Parseval relation, converting the attention-aware objective into MSE, paired with volume-balanced vector quantization. It represents a genuine theoretical increment over heuristic rotations in QuaRot/OSCAR/TurboQuant, and the orthogonal baseline completely collapses on GPT-OSS-20B hybrid MoE.
Long-context capability in hybrid attention models is carried by full attention layers, while efficient attention acts only as an optimization prior that shapes how quickly full attention learns long-range retrieval.
Editor's noteProposes the "large-window laziness" phenomenon—larger sliding windows delay the formation of retrieval heads in full attention layers, indicating that efficient attention acts more as an optimization prior than as a direct carrier of long-range information. It provides compute-matched scaling comparisons between SWA and recurrent mixers with LongPPL degradation curves, and finds that applying NoPE to full attention layers improves long-context performance in small-window hybrid models, representing the negative-result category this direction aims to collect.
MSA matches full attention quality while cutting attention FLOPs by 28x at 1M context, enabling 14x prefill and 7.6x decoding speedups on a 109B MoE model.
Editor's noteIntroduces per-GQA-group independent block-level top-k selection, paired with KL alignment, gradient isolation, and indexer warm-up, co-designed with a KV-outer sparse kernel, achieving 14.2× prefill speedup at 1M context. The increment over MoBA/NSA is the combination of "per-group selection + block-level granularity," and provides compute-matched comparisons with full attention at 109B scale along with RULER/HELMET degradation curves.
SparDA adds a lightweight Forecast projection that predicts next-layer KV block selections, enabling lookahead prefetching that hides PCIe latency and delivers up to 5.3x higher decode throughput.
Editor's noteUses a trainable fourth projection (Forecast) to decouple block-level sparse selection from attention queries, enabling one-layer-ahead CPU→GPU KV prefetch overlap, paired with a compact indexer with one head per GQA group. Relative to InfiniGen's training-free proxy and DSA's token-level indexer, it advances the decoupled indexer idea to block sparsity and provides RULER degradation curves from 32K to 128K.
LongCat Sparse Attention: Taming the Lightning via Streaming-aware Hierarchical Cross-Layer Indexing
LongCat Sparse Attention achieves near-lossless quality with 1.42-3.60x prefill and 1.25-1.40x decode speedups over DeepSeek Sparse Attention via streaming-aware, cross-layer, and hierarchical indexing.
Editor's noteIntroduces three orthogonal system-level optimizations for DSA-style trainable sparse attention—streaming-aware indexing (combining sink/window fixed regions with dynamic selection for coalesced memory access), cross-layer indexing (cross-layer distillation allowing one indexer to serve multiple layers), and hierarchical indexing (coarse-to-fine page-then-token selection reducing per-query selection cost from O(L) to O(L/P+MP)). It directly addresses DSA's quadratic indexer overhead and non-coalesced memory access issues, and provides HELMET/NIAH/MRCR degradation curves.
Under a deployment-realistic query-agnostic protocol, only KeyDif beats trivial baselines (31/36 cells), while SnapKV loses to them, with per-method drops ordered by how much the question is visible in each scoring signal.
Editor's noteFor the first time quantifies under matched budgets how query visibility (query-aware vs query-agnostic) flips KV compression rankings, and reveals two reproducible evaluation pitfalls—attention backend confusion (eager vs sdpa can shift scores by 0.29, larger than most inter-method gaps) and tokenizer-dependent benchmark lengths. It demonstrates that query-aware scores partially measure query relevance rather than information importance, serving as a critical warning for both the design and evaluation of compression methods.
Accuracy alone masks severe reasoning degradation in KV-compressed large reasoning models, so evaluation must pair accuracy with faithfulness metrics like reasoning correctness and fidelity.
Editor's noteProposes the "answer-evidence gap" as a new failure mode for KV compression—compressed models can preserve the final answer while quietly undermining the faithfulness of the reasoning chain. It uses a fixed-trajectory replay protocol with three complementary metrics (accuracy, answer-chain consistency, perturbation faithfulness) to measure this gap, and finds that eviction-based methods harm reasoning faithfulness while quantization methods relatively preserve it, providing an evidence dimension beyond accuracy for evaluating compression methods.
