Back to News

Prefrontal AI and ETH Zurich unveil Latent Sparse Attention, speeding up 10M-token prefill by up to 72x

#sparse attention#long context#prefill#efficiency

Prefrontal AI and ETH Zurich published a paper introducing Latent Sparse Attention (LSA), a method that accelerates prefill for ultra-long contexts by up to 72.4x at 10 million tokens. The approach addresses the quadratic growth of the indexing cost in sparse attention, which dominates prefill time at such scales. LSA speeds up a single attention layer of GLM-5.3-Flash by 22.7x and Qwen3.8-27B by 72.4x while maintaining accuracy.

Coverage timeline

  1. 机器之心机器之心

    编辑|Panda 如今,百万 token 上下文已经从卖点变成了标配。4 月,DeepSeek-V4 把 1M 上下文定为所有官方服务的默认规格;8 月,智谱发布的 GLM-5.3-Flash 同样支持 1M 上下文,并首次在 GLM 系列中引入稀疏注意力与线性注意力的混合架构。这些模型背后有一个共同的选择: 让每个 query 不再看遍所有历史 token,只挑出少数最相关的去「看」。 这条路线在 1M 量级上已经被反复验证。可当上下文继续往 1000 万 token 推进时,一笔原本不起眼的开销开始失控。以 GLM-5.3-Flash 的一个全局注意力层为例,在 10M token 下做 prefill, 98% 的时间并没有花在注意力计算上,而是花在决定「该看哪些 token」的索引器上 。 问题的性质因此变了。稀疏注意力早已证明「只看一小部分」足以保住精度,真正卡住超长上下文的是 「找到那一小部分」依然会随上下文长度二次方增长。 现在,伦敦初创公司 Prefrontal AI 与苏黎世联邦理工学院(ETH Zurich)联合发布论文,提出了一种名为 Latent Sparse Attention(LSA,隐空间稀疏注意力) 的新方法。 它在 10M token 下把 GLM-5.3-Flash 单个注意力层的 prefill 提速 22.7 倍 ,把 Qwen3.8-27B 的提速 72.4 倍 ,同时在一系列任务上保持了与原模型相近的精度。 论文标题:Latent Sparse Attention: Towards Subquadratic Ultra-Long Context Attention 论文地址:https://arxiv.org/abs/2610.42147 代码地址:https://github.com/Prefrontal-AI/latent-sparse-attention 稀疏注意力的「隐形账单」 先回顾一下 注意力的成本结构 。在标准的全注意力里,每个 query 都要和之前所有的 key 计算一遍相关性,上下文长度为 T 时,prefill 的计算量是 O(T²),每解码一个 token 是 O(T)。上下文从 100 万涨到 1000 万,prefill 的注意力计算量就要涨 100 倍。 稀疏注意力 的思路是分工:先

Related stories

  • PKU Researchers Propose MapFlow to Reuse Attention Maps Across LLMs, Cutting Prefill Latency 15-20%

    Researchers at Peking University's PKU-DAIR lab propose MapFlow, a method that reuses attention maps as intermediate states across different LLMs to avoid recomputing context when switching models. In evaluations, MapFlow reduces prefill latency by 15-20% and boosts throughput by up to 25% under a P90 first-token latency constraint in multi-turn routing scenarios, while maintaining generation quality. The paper has been accepted at SIGMOD 2027 and the code is open-sourced.

    1 reports · 1 sources
  • DeepSeek open-sources Super-agent architecture for autonomous bug-fixing agents

    DeepSeek has publicly released the architectural blueprint for its Super-agent, a research prototype of a personal agent that maintains cross-session context to handle multi-task coordination. Internal tests show the agent can autonomously identify bugs and submit pull requests without being explicitly prompted, with a median task completion time of 12.4 minutes in simulated multi-task environments. DeepSeek positions this as a step toward reducing human attention bottlenecks in multi-task workflows.

    3 reports · 3 sources
  • Autonomous Agent ForgeKernel Writes FlashMLA Kernel 9.8% Faster Than Human Version

    In a 22.5-hour livestream, the autonomous agent ForgeKernel, developed by ModelBest and OpenBMB, wrote a FlashMLA Prefill kernel for DeepSeek-V4.1 from scratch without source code or papers. The final version achieved 472 microseconds versus FlashMLA's 518 microseconds, a 9.8% speedup, using one Nvidia B200 and 22 GPU hours over 42 iterations. The system was open-sourced after the livestream, and it previously topped all three Ascend CANNBench 950 rankings.

    1 reports · 1 sources
  • Prime Intellect open-sources fused MoE kernels with overlapping backward pass, up to 3.11x faster

    Prime Intellect extended DeepSeek's Mega MoE kernel with a fast overlapping backward pass, making it suitable for training. The company reports up to 3.11x faster MoE kernels and up to 1.34x faster end-to-end training in Prime-RL, with the kernel code open-sourced.

    1 reports · 1 sources

News updates every hour. Subscribe in your RSS reader to get new stories automatically.

Subscribe