Continuum Attention Paper Proposes Continuous Softmax Attention Formulation
A Chinese blog post by 苏剑林 analyzes the Continuum Attention paper, which proposes a continuous formulation of softmax attention. The post discusses this within the broader 'continuize-discretize' research path, citing prior examples like EFLA and early SSM work.
Coverage timeline
科学空间 (苏剑林)苏剑林
Softmax Attention想必大家已经不陌生，它是定义在有限向量集合下的一个加权混合算子。文章 《Continuum Attention》 提出了一个很有意思的想法——将注意力连续化。这让笔者想起，之前在 《为什么DeltaNet要加L2 Normalize？》 介绍的 EFLA 也是将离散RNN连续化，然后又重新离散化，得到一些有意思的结果，类似的还有早期的 SSM 系列工作。 这样看来，“连续化—离散化”似乎是一条屡试不爽的探索路径。那么，Softmax Attention连续化又能得到什么有意思的结果呢？让我们一起来品鉴一番。 问题描述 标准的Softmax Attention定义为 \begin{equation}\boldsymbol{o}_t = \frac{\sum_{i=1}^t e^{\boldsymbol{q}_t\cdot\boldsymbol{k}_i} \boldsymbol{v}_i}{\sum_{i=1}^t e^{\boldsymbol{q}_t\cdot\boldsymbol{k}_i}}\end{equation} [...]
Related stories
STILL: Content-Aware Intra-Layer Hybrid Attention Cuts Memory 80%, Speeds Prefill 83%
Researchers from Harbin Institute of Technology (Shenzhen) and collaborators present STILL, a NeurIPS 2026-accepted architecture that routes tokens to Softmax or linear attention within the same layer based on content importance via a self-saliency score. It achieves 80% memory reduction and 83% prefilling speedup at 16K context, with only 0.04B training tokens for linearization, and up to 86.2% improvement over prior linearization methods on long-context benchmarks.
1 reports · 1 sources
GDN2 update matrix numerical stability issues analyzed
A technical blog post analyzes numerical stability issues in the GDN2 linear attention update matrix, which generalizes DeltaNet's I - eta k k^T form to I - eta k j^T. The post notes that while DeltaNet's original form with L2 normalization and eta <= 2 ensures stable products, GDN2's generalization may introduce new instability problems, and it proposes ways to make GDN2 more stable.
News updates every hour. Subscribe in your RSS reader to get new stories automatically.
Subscribe