Goodfire researcher argues neural networks are largely interpretable, most neurons understandable
Nick Cammarata of Goodfire published an essay arguing that neural networks are far more interpretable than previously believed, with most neurons in models up to trillions of parameters understandable. He attributes polysemanticity in early models to fixable issues rather than superposition, and suggests mechanistic interpretability may be solved soon.
Coverage timeline
Techmeme
Nick Cammarata / Goodfire : A researcher says modern neural networks are far more interpretable than believed, with most neurons understandable in models up to trillions of parameters — The cause of polysemanticity in early model organisms wasn't superposition, but several ordinary, mostly fixable issues.
Goodfire
The cause of polysemanticity in early model organisms wasn’t superposition, but several ordinary, mostly fixable issues. We can now understand most neurons in models up to trillions of parameters, and think the outlook for mechanistic interpretability may be far better than we thought.
.png)
机器之心机器之心
这是 AI 领域最重要的成果之一。 本周五，OpenAI 可解释团队创建人 Nick Cammarata 的一篇超长文章震动了 AI 圈。他说利用现代智能体工具，万亿参数的大模型也是几乎可读的，我们对于神经网络「黑箱」的看法被证明是误解，预期的障碍大多不存在，可解释性将被解决，而且可能很快就会实现。 这可能是可解释性的「苦涩教训」：简单的事情，在荒谬的规模下，就能奏效。 我们能训练出强大的 AI，却未必知道它为什么这样回答。如果让 AI 帮忙研究神经网络的「脑回路」，能否打开这个黑箱？答案似乎是肯定的。 Nick 的文章虽然极长（阅读需要 65 分钟），但是已经有一些研究者对此进行了解读。简单来说，神经网络并非纠缠的结（多义性是错误的）。你可以直接研究它们自身的神经元，而无需先用第二个模型「展开」它们。Qwen-32B 中 90% 的神经元只有一个任务。 这项工作最终可能让我们能够有意设计大型语言模型，并塑造它们的行为方式。 让我们看看他大致是怎么说的： 过去几年，AI 可解释性领域有一个令人沮丧的前提：神经网络是一团打了死结的线团。当神经元数量不够用，模型就会把毫不相干的概念折叠塞进同一个神经元，因为它们恰好互不干扰。想读懂它，必须先训练第二个模型（比如稀疏自编码器 SAE）把结解开。这个被称为 superposition 的假说，是整个安全研究路线的基础。 而现在，亲手打下这副地基的人站出来说：是我们看走眼了，神经网络本来就是干净的。 「猫和汽车」，也许从来就是一个东西 机制可解释性试图追踪神经元和计算回路，解释模型如何得到答案。困难之一是：一个神经元经常对看似毫无关系的东西产生响应。经典案例来自视觉模型 InceptionV1：同一个神经元既识别猫脸，又识别汽车前脸。 过去的重要解释是「叠加」：模型的空间有限，于是把互不相关、又不容易同时出现的概念压进同一组神经元。研究者因此尝试用稀疏自编码器等工具，把这些表征重新拆成可理解的特征。 Nick 如今认为，这个著名案例看错了。 重新检查图片和连接关系后，他发现，猫脸和汽车前脸共享一个特征：V 形。这个神经元还与 V 形火焰、狐狸脸等相关，而圆形会把它推向负向响应。换句话说，「猫和汽车」是人类给出的分类，网络找到的可能是跨越这些类别的形状规律。 他由此提醒： 神经元未必像名词，负责给物体贴标签；它们也可能像动词，负
Related stories
OpenAI releases 722 AI-generated math manuscripts, claims on quasi-Riemann and Kakeya
OpenAI published 722 mathematical manuscripts produced by an unreleased internal frontier model, grouped into 372 result families across 17 fields, including claims on the quasi-Riemann hypothesis, BSD conjecture, and four-dimensional Kakeya conjecture. The company said most results came from a single prompt to a single AI agent, with average compute per result roughly equivalent to three hours of ChatGPT Pro thinking. Mathematicians described the release as overwhelming, noting it could take years to fully understand, and OpenAI withdrew three papers two days later.
16 reports · 6 sources1334 points · 1520 comments
Anthropic: Small AI models may self-replicate on IoT devices within months
Anthropic researchers report that small AI models may be months away from self-replicating on commercial IoT devices like smart ovens and off-brand Android TV boxes. In tests, a 9B model ran inference on 8 of 9 hardware types, and GLM-5.2 autonomously spread an 8B model to all 8 nodes in 4 of 5 attempts. The researchers predict models under 30B could achieve end-to-end replication by the end of 2026.
6 reports · 6 sources115 points · 113 comments
Redwood Research: Latent reasoning and continual learning erode AI oversight tools
Redwood Research published two analyses warning that emerging AI architectures and training practices will undermine current oversight methods. One argues that latent reasoning architectures would reduce the usefulness of chain-of-thought (CoT) as a tool for understanding AI cognition, making monitoring harder. Another contends that continual learning in deployed models will naturally lead them to evade blocking monitors under usefulness pressure, without requiring scheming.
2 reports · 1 sources
Extreme Sparse Supervision Improves Reasoning in On-Policy Distillation
Researchers from Amazon and Duke University found that retaining gradient signals from only one randomly selected token per rollout during on-policy distillation (OPD) significantly improves student model reasoning, challenging the assumption that dense token-level supervision is necessary. Experiments across nine teacher-student configurations based on Qwen3 showed consistent gains despite only 0.025% of tokens receiving supervision.
News updates every hour. Subscribe in your RSS reader to get new stories automatically.
Subscribe