New Route Linearizes Softmax Attention to TTT, Unifies DeltaNet and MesaNet
In a blog post dated 2026-11-01, Su Jianlin proposes a new derivation that linearizes Softmax Attention into Test-Time Training (TTT) via three averaging steps. Within this unified framework, DeltaNet and MesaNet emerge as natural special cases, offering deeper insight into the approximations and trade-offs behind linear attention.
Coverage timeline
科学空间 (苏剑林)苏剑林
在文章 《将Softmax Attention线性化为Gated DeltaNet》 中，我们提供了一个将Softmax Attention线性化为Gated DeltaNet的参考过程，它演示了我们可以从Softmax Attention出发，逐步引入近似，最后得到一个线性模型。这个结果并不一定有多实用，但它能帮助我们更好地理解Linear Attention背后的近似与代价。 本文则提供一条新的线性化路线，它从分布推广出发，引入三步平均操作，建立起了Attention与TTT的近似联系，在这个大框架下，DeltaNet、MesaNet自然地作为它的特例出现。 推广定义 给定向量$\boldsymbol{q},\boldsymbol{k}_1,\cdots,\boldsymbol{k}_n,\boldsymbol{v}_1,\cdots,\boldsymbol{v}_n$，Softmax Attention定义为 \begin{equation}\boldsymbol{o} = \sum_{i=1}^n a_i \boldsymbol{v}_i, \qquad a_i = \frac{e^{\boldsymbol{q}\cdot \boldsymbol{k}_i}}{Z},\qquad Z = \sum_{i=1}^n e^{\boldsymbol{q}\cdot \boldsymbol{k}_i}\end{equation} [...]
Related stories
Continuum Attention Paper Proposes Continuous Softmax Attention Formulation
A Chinese blog post by 苏剑林 analyzes the Continuum Attention paper, which proposes a continuous formulation of softmax attention. The post discusses this within the broader 'continuize-discretize' research path, citing prior examples like EFLA and early SSM work.
STILL: Content-Aware Intra-Layer Hybrid Attention Cuts Memory 80%, Speeds Prefill 83%
Researchers from Harbin Institute of Technology (Shenzhen) and collaborators present STILL, a NeurIPS 2026-accepted architecture that routes tokens to Softmax or linear attention within the same layer based on content importance via a self-saliency score. It achieves 80% memory reduction and 83% prefilling speedup at 16K context, with only 0.04B training tokens for linearization, and up to 86.2% improvement over prior linearization methods on long-context benchmarks.
1 reports · 1 sources
Apple and CMU propose SSR framework cutting multimodal search agent inference latency by over 90%
Researchers from Apple and Carnegie Mellon University propose SSR (Selection-based Structured Reasoning), a framework that replaces free-form token-by-token reasoning with selection from predefined reasoning options in multimodal search agents. In experiments across 7 multimodal search benchmarks, SSR reduces per-step inference latency by over 90% while maintaining task success rates comparable to free-form thinking.
Xiaohongshu DoSA sparse attention cuts TTFT by up to 48% via score bounds
Xiaohongshu's Dots team introduced DoSA (Dots Sparse Attention), a sparse-attention method that accelerates the indexer using score bounds while preserving exact Top-K selection results. In tests at ~1M-token context lengths, it reduced first-token latency by about 37%, 46%, and 48% for Dots3-Note-preview, GLM-5.2, and DeepSeek-V3.2 respectively, compared to their SGLang baselines, and filtered roughly 97% of candidates early.
News updates every hour. Subscribe in your RSS reader to get new stories automatically.
Subscribe