Stanford PhD Alex Wang derives Transformer architecture from memory storage perspective
On October 15, 2026, Stanford CS PhD and former Cartesia research scientist Alex Wang published a blog post titled 'You could have invented transformers from memory', proposing that linear layers, MLPs, and self-attention can all be viewed as associative memory stores. The post derives Transformer components from memory capacity considerations, showing that softmax and the 1/d_k scaling factor emerge naturally from interference suppression. It reports experimental capacities: a simple linear memory stores about 64 items, a GLU-style MLP about 2,700, and self-attention with fixed scaling about 1.2 million, sparking discussion in the AI community.
Coverage timeline
机器之心机器之心
如果 2017 年那篇《Attention Is All You Need》从未发表，给你一个问题：「如何让机器记住尽可能多的东西，并且想取哪条就取哪条」，你能 从零开始，把 Transformer 一层一层地推导出来吗？ 很多人可能觉得一头雾水。10 月 15 日，斯坦福大学计算机科学博士、前 Cartesia 研究科学家 Alex Wang 发表了一篇博客《 You could have invented transformers from memory 》（你本可以凭「记忆」发明 Transformer），迅速在社交媒体引发关注。 正如标题所展示的，这篇文章做了一件很「野心勃勃」的事：它提出一个统一的理解视角 —— 我们熟悉的神经网络层，可以被看作不同形式的「联想记忆存储器」。 线性层、MLP、自注意力，看似差异很大的三种结构，其实都可以从「存记忆、取记忆」的角度理解。衡量它们记忆能力的关键指标，是信息之间的干扰：存储的记忆越多，彼此干扰越严重。谁能在干扰淹没目标信号之前装下更多记忆，谁的「容量」就越大。 沿着这个思路，作者像搭积木一样完成了一次有趣的推导：从最朴素的外积记忆出发，引入非线性激活，就自然「长」出了 MLP；进一步提高非线性程度，又与 GLU 等门控结构建立了联系。 而当你想要「用上所有幂次」来压制干扰时，指数函数登场了。经过归一化处理，便得到了 Softmax，自注意力的数学形式也随之浮现。 效果差异是可以量化的。在作者设定的随机记忆实验中，同样使用 64 维的 Key，最简单的线性存储器容量约为 64 条，平方 ReLU 构成的 GLU 式 MLP 能存储约 2700 条，而采用固定缩放系数的 Self-Attention，理论容量达到约 120 万条。 从 64 到 120 万，容量差了近两万倍。当然，这里的容量对应的是理想化条件下的随机向量记忆任务，并不意味着真实的 Transformer 能记住比 MLP 多两万倍的知识。 更有意思的是，Transformer 中著名的 1/d_k 缩放因子，以及 Qwen 的 LogN-Scaling、YaRN 等长上下文扩展方法采用的注意力缩放策略，也能从记忆干扰的角度得到解释。 而这篇文章与 2017 年 Transformer 原始论文之间，还有一段鲜为人知的联系。Alex Wang 发现，《A
Related stories
Anthropic: Small AI models may self-replicate on IoT devices within months
Anthropic researchers report that small AI models may be months away from self-replicating on commercial IoT devices like smart ovens and off-brand Android TV boxes. In tests, a 9B model ran inference on 8 of 9 hardware types, and GLM-5.2 autonomously spread an 8B model to all 8 nodes in 4 of 5 attempts. The researchers predict models under 30B could achieve end-to-end replication by the end of 2026.
6 reports · 6 sources115 points · 113 comments
SparkDiffusion: Unified Sparse Attention, Distillation, FP8 Speeds Video Generation 265x
Researchers from Peking University, Tsinghua University, and Alibaba propose SparkDiffusion, a unified framework integrating sparse attention, few-step distillation, and FP8 quantization to accelerate video generation. It addresses the 'high-sparsity trap' where extreme sparsity (97%) degrades output quality despite falling training loss, using staged sparse warm-up and trajectory-mixed distillation. The system achieves 265x speedup, generating a 5-second 720P video in 18 seconds, with open weights and code.
Goodfire researcher argues neural networks are largely interpretable, most neurons understandable
Nick Cammarata of Goodfire published an essay arguing that neural networks are far more interpretable than previously believed, with most neurons in models up to trillions of parameters understandable. He attributes polysemanticity in early models to fixable issues rather than superposition, and suggests mechanistic interpretability may be solved soon.
.png)
News updates every hour. Subscribe in your RSS reader to get new stories automatically.
Subscribe