DeepMind Researchers Find Numerical Precision Bugs in Softmax and Cross-Entropy Across Major Frameworks
Google DeepMind researchers identified two numerical precision bugs in the standard Softmax and cross-entropy implementations used by PyTorch, JAX, and TensorFlow. The bugs can subtly alter gradients during training, potentially causing loss spikes or capability regression. The team proposed a fix that adjusts the computation without increasing floating-point precision or redesigning the loss function; the paper was submitted to arXiv on October 22.
Coverage timeline
机器之心机器之心
深度学习中最基础、使用最广泛的两个函数，竟然藏着两个隐蔽的 Bug。 昨日，Google DeepMind 首席研究科学家 Jonathan T. Barron 在 X 上公布了一项新研究。 他指出，包括 PyTorch、JAX、TensorFlow 在内的主流深度学习框架，普遍采用的 Softmax 和交叉熵（Cross-Entropy）标准实现中，存在两个容易被忽视的数值精度 Bug。 这些 bug 平时很难察觉，却可能在模型训练的特定阶段悄悄改变梯度，造成损失突然飙升，甚至让模型已经学到的能力出现退化。 更有意思的是，研究团队发现，只需调整计算方式，就能修复这两个 bug，无须提高浮点精度，也不需要重新设计损失函数。 相关论文已于 10 月 22 日提交至 arXiv，除了 Jonathan T. Barron，还有另外两位研究工程师与研究科学家 Dan Gnanapragasam 和 Michael Tschannen。 论文标题：A Fix for Softmax and Cross-Entropy 论文地址：https://arxiv.org/pdf/2610.31642 并且按照 Barron 的说法， 从发现问题、开展实验到撰写论文，Gemini 在这项研究中承担了相当多的工作。 模型越自信 梯度反而算错了 第一个 bug 叫做 Softmax Collapse（Softmax 坍缩） 。 要理解它，先得看看 Softmax 在做什么。 训练分类模型时，神经网络通常会为每个候选类别输出一个原始分数，也就是 Logit。Softmax 负责将这些分数转换成概率，再由交叉熵衡量模型的预测与正确答案之间的差距。比如，一张图片中有一只猫，模型给猫的分数远高于狗和鸟。经过 Softmax 转换，猫的预测概率可能达到 99.99%。 随着训练继续，模型对正确答案越来越有把握，正确类别的 Logit 与其他类别之间的差距也越来越大。问题就出在这里。 现有实现通常会先从所有 Logit 中减去最大值，再进行指数计算。这是一种经典的数值稳定性处理，可以防止指数运算发生溢出。但减去最大值后，最大 Logit 对应的指数值变成了 1，其他类别则是一些很小的正数。在正确类别分数最高的情况下，Softmax 的分母实际上变成了： 1 + 一个很小的数。 浮点数的精度有限。当这
Related stories
Google open-sources TorchTPU, making TPU a native PyTorch device
At PyTorch Conference 2026, Google announced the open-sourcing of TorchTPU, a software stack enabling native PyTorch execution on TPUs, developed with Meta and integrated by vLLM, SGLang, Hugging Face, Ray, and TorchTitan. The project, available on GitHub with a 23-page technical report on arXiv, allows deployment to TPU via a single parameter in Hugging Face Transformers v5, positioning TPU alongside GPU and CPU as a native PyTorch option.
1 reports · 1 sources
PHP2Shell critical RCE flaws patched; exploited before disclosure
PHP released security updates (8.5.12, 8.4.27, 8.3.36, 8.2.35) fixing two critical unauthenticated remote code execution vulnerabilities, collectively dubbed PHP2Shell, affecting up to 70% of websites. The flaws were discovered in August by OpenAI's GPT-5.6 Sol and independently by Google's Wiz, and AWS reported exploitation starting October 17, four days before public disclosure.
Google launches TorchTPU, vLLM adds TPU backend
Google announced TorchTPU, an open-source version of PyTorch that lets AI applications run on TPUs and Google Cloud via simple configuration changes without rewriting code. Concurrently, vLLM introduced vllm-torchtpu, a new backend for Google Cloud TPUs built on TorchTPU, featuring Pallas kernels, multi-host parallelism, KV cache offloading, and disaggregated serving. These moves, following Amazon's similar tool, challenge Nvidia's dominance in AI development.
3 reports · 3 sources
Google unveils Gemini 4 Argon, limits early access to cyber defenders
Google DeepMind announced Gemini 4 Argon, its next frontier model, on September 30, 2026, with initial access restricted to a set of trusted cybersecurity defenders via the Fairwind Program. The model features a 1M-token output limit, priced at $2/1M input and $10/1M output tokens initially, rising later, and reportedly outperforms GPT-6 Astra on coding and knowledge benchmarks. Internal skepticism exists, with some employees claiming real-world coding struggles, which Google disputes.
10 reports · 7 sources1691 points · 1177 comments
News updates every hour. Subscribe in your RSS reader to get new stories automatically.
Subscribe