STILL: Content-Aware Intra-Layer Hybrid Attention Cuts Memory 80%, Speeds Prefill 83%
Researchers from Harbin Institute of Technology (Shenzhen) and collaborators present STILL, a NeurIPS 2026-accepted architecture that routes tokens to Softmax or linear attention within the same layer based on content importance via a self-saliency score. It achieves 80% memory reduction and 83% prefilling speedup at 16K context, with only 0.04B training tokens for linearization, and up to 86.2% improvement over prior linearization methods on long-context benchmarks.
Coverage timeline
机器之心机器之心
本文一作孟维康是哈尔滨工业大学（深圳）博士研究生，主要研究方向是高效基础模型架构，线性 / 混合注意力机制。通讯作者张正教授是哈尔滨工业大学（深圳）长聘教授、博士生导师，长期从事高效能多模态机器学习研究。 Softmax 注意力的平方复杂度，一直是大模型迈向长上下文的最大拦路虎。线性注意力虽快，从头训练一个大型线性模型却动辄需要成千上万 GPU 时。于是，「把现成的预训练 LLM 直接线性化」成为最具性价比的路线 —— 但现有方法要么按位置粗暴划窗、把远处关键证据挡在高质量注意力之外，要么在特征映射中破坏预训练模型的范数结构，长文能力断崖式下跌。 STILL 近日被 NeurIPS 2026 正式接收。STILL 的核心贡献可以概括为两点： 首个真正的同层混合注意力框架 。 此前的层内方案（LoLCATs、Liger 等）只是把固定滑窗内的 token 交给 Softmax、窗外的交给线性注意力 —— 在作者看来，这种按位置一刀切的做法只是两种算子的「拼接」，算不上真正的混合。STILL 首次让 token 凭 内容重要性 在同一层内分工：重要的进 Softmax，其余的进线性注意力； 用局部窗口预测未来有用的历史信息 。 STILL 提出「自显著性分数（Self-Saliency Score）」，只需在局部滑窗内计算，就能预判一个 token 成为远端历史之后「未来还会不会被关注」，理论上保证局部 - 全局排名一致，实证重合率高达 84.8%。 配合保范数的特征映射 NP-Map 与统一训练 - 推理的分块并行实现，STILL 仅用 0.04B 训练 token 就完成线性化，在 16K 上下文下实现 80% 显存削减、83% 预填充加速 ，并在长上下文基准上相对已有线性化方法最高取得 86.2% 的提升。 论文题目：STILL: Selecting Tokens for Intra-Layer Hybrid Attention to Linearize LLMs 作者：Weikang Meng, Liangyu Huo, Yadan Luo, Jiawen Guan, Jingyi Zhang, Yingjian Li, Zheng Zhang 单位：哈尔滨工业大学（深圳），鹏城实验室，昆士兰大学，瑞汀灵动（上海） 论文链接：https://arxiv.org/
Related stories
VisInteract: Interactive Text-to-Visualization for Imperfect Queries, NeurIPS 2026 Spotlight
Researchers from Hong Kong Polytechnic University and ByteDance propose VisInteract, a new paradigm for dynamic interactive text-to-visualization that handles imperfect user queries. They introduce VisInteract-Bench, a benchmark acknowledging ambiguous, incomplete, or erroneous inputs, and Vis-MCTS, a Monte Carlo tree search-based method that branches and backtracks between data retrieval, chart generation, and clarification. The work was accepted as a Spotlight paper at NeurIPS 2026 (292 out of 30,709 submissions).
1 reports · 1 sources
Stanford, Together AI propose Next-Concept Modeling for parallel text generation, 10-31x speedup
Researchers from Stanford University and Together AI introduced Next-Concept Modeling (NCM), a coarse-to-fine parallel text generation paradigm adapted from next-scale prediction used in image generation. In experiments, NCM achieved 10-31x speedup over diffusion language models with comparable or better quality, offering an alternative to both autoregressive and diffusion approaches.
SAGE topological guidance mitigates long-horizon reasoning biases, NeurIPS 2026
Researchers from Virginia Tech, UW-Madison, and Dartmouth propose SAGE, a method that uses symbolic closure analysis and topological guidance to mitigate exploration and accumulation biases in long-horizon reasoning. Evaluated on 12 benchmarks across 7 model families, SAGE outperforms compared post-training methods; on Andrews–Curtis instances, Qwen3's Lean verification pass rate is nearly 8 times that of the base model. The paper is accepted at NeurIPS 2026.
Kaiming He's Team Unveils VISTA, a Visual Harness for Multimodal Reasoning
Kaiming He's team released a paper introducing VISTA, a visual-native harness framework that lets multimodal models observe environments, store raw frames, and revisit them during reasoning without retraining. In evaluations, VISTA enabled Claude Opus 5.0 to complete all 25 public ARC-AGI-3 games with a perfect human-relative efficiency score of 100, using 57.4% fewer actions than first-time human baselines; GPT-5.6 Sol also cleared all games with a score of 99.
1 reports · 1 sources
News updates every hour. Subscribe in your RSS reader to get new stories automatically.
Subscribe