Stanford and Tsinghua researchers find sparse value and dopamine neurons in LLMs, mapping a reward subsystem
Researchers from Stanford and Tsinghua University identified two types of neurons in large language models that encode a reward subsystem: value neurons, which predict the expected value of the current state, and dopamine neurons, which encode stepwise temporal difference errors. Intervention experiments on Qwen models showed that zeroing the top 1% of value neurons in a layer dropped MATH500 accuracy from 75.2% to 20.3%, while random zeroing had little effect, providing causal evidence for their role.
Coverage timeline
机器之心机器之心
编辑丨% 大型语言模型的隐藏状态里，早已被发现包含一些与答案正确性、模型置信度和奖励相关的信息。但此前的研究通常直接对完整隐藏状态训练一个探测器，只能说明这些信息「能够被读出来」，却很难进一步回答究竟是模型中的哪些神经元在编码这些信号？ 斯坦福与清华大学在一项研究中出了一种更细的观察方式。他们发现，与奖励相关的信息并没有平均分散在整个隐藏状态中，而是集中在一小部分神经元里，并将其称为 reward subsystem，即奖励子系统。 论文链接： https://arxiv.org/abs/2602.00986 模型中的神经元 论文最终找到的两类神经元，分别对应状态价值和逐步奖励预测误差。 第一类是 value neurons（价值神经元）。它们编码当前状态的预期价值，也就是从当前推理状态继续生成下去，最终得到正确答案的概率。 第二类被称为 dopamine neurons（多巴胺神经元）。它们编码的是逐步的时分误差（TD error），反映某一步推理之后，模型预期获得正确答案的概率发生了多大变化。 表 1：奖励子系统的概述。 价值神经元 想要找到这些神经元，研究团队首先训练一个简单的价值探针，从模型隐藏状态预测当前状态的价值。随后，他们逐渐剪掉探测器输入中的神经元，只保留最重要的部分。 结果显示，即使只留下不到 1% 的神经元，模型依然可以较好地预测状态价值。这个现象不仅出现在 Qwen-2.5-14B 模型和 GSM8K、MATH500 任务中，研究人员还在 ARC、MBPP+ 和 IFEval 等不同任务，以及 Qwen、Phi 和 Llama 等不同模型中观察到了类似的稀疏价值神经元。 他们随后对这些神经元进行直接干预，在 Qwen-2.5-7B-SimpleRL-Zoo 上，原始 MATH500 准确率为 75.2%。研究人员只将某一层中排名最高的 1% 价值神经元置零，平均准确率下降到 20.3%；而随机置零同样比例的神经元时，准确率基本保持不变。 图 1：GSM8K 和 MATH500 数据集上 Qwen-2.5-14B-SimpleRL-Zoo 模型第 2–4 层的 AUC 曲线。 不同于一般意义上的「重要神经元」，这些价值神经元被破坏后出现的巨大性能下降，为它们确实承担奖励相关功能提供了因果证据。 多巴胺神经元 价值神经元关注的是当前状态最终能否成功
Related stories
UCL team unveils Large Discovery Models for experiment design
Researchers at University College London led by Wang Jun released Large Discovery Models (LDMs), a recursive foundation-model architecture that combines generative foundation models with Gaussian process reward models to design next experiments in scientific discovery. Validated on neural-network training, antibody design, and molecular optimization, LDM v0.1 reportedly outperforms pure LLM reflection baselines, with antibody binding energy reduced by 18.2% on average and molecular multi-objective hypervolume improved by over 60%.
1 reports · 1 sources
Study Characterizes Massive Activations in Hybrid Linear Attention LLMs
Researchers from StartLux, Tsinghua University, and other institutions published a study using massive activations (MAs) as probes to systematically characterize cross-layer traces of full attention layers in hybrid linear attention LLMs, identifying two core patterns: pre-attention spikes (PAS) and inter-spike plateaus (ISP). The study covers five linear attention architectures, six hybrid configurations, five data domains, and 12 public HLA LLM checkpoints, with model sizes ranging from 1.2B to 397B parameters.
Stanford's Virtual Biotech: Multi-Agent AI Framework for Drug Discovery Published in Science
Stanford researchers published a paper in Science on September 17, 2026, describing Virtual Biotech, a multi-agent AI framework that simulates a biotech company's R&D organization for therapeutic discovery. The system comprises four research departments and 11 main agents, with a virtual chief scientific officer coordinating tasks such as target discovery, safety assessment, and clinical development. It uses Claude Sonnet 4.5 and Haiku 4.5 models, and in a data-curation example, deployed 37,075 parallel agent instances to process clinical trial records.
Peking University and YIXIN AI Lab identify 'readout bottleneck' in LLM reasoning at EMNLP 2026
Researchers from Peking University and YIXIN AI Lab presented a paper at EMNLP 2026 introducing the 'readout bottleneck' phenomenon, where LLMs may encode correct answers in hidden states but fail to express them due to systematic biases in the output layer. They propose a staged diagnostic method to distinguish between genuine capability deficits and expression failures. The paper, titled 'Wrong Prediction, Right Answer: Recovering Evidence from Collapsed LLM Sequence Scores', was accepted to the main conference with a 15.4% acceptance rate.
News updates every hour. Subscribe in your RSS reader to get new stories automatically.
Subscribe