Blog post: Diagonal preconditioners like Adam also converge toward Muon under 'orthogonal slowest descent'
In a technical blog post dated 2026-10-15, the author extends their 'orthogonal slowest descent' framework from Kronecker-product preconditioners to diagonal approximations such as RMSProp and Adam. The analysis suggests that applying this framework to diagonal preconditioners also drives them toward the Muon optimizer, reinforcing the earlier claim that 'Shampoo's end is Muon'.
Coverage timeline
科学空间 (苏剑林)苏剑林
在上一篇文章 《慢即是快：1. Shampoo的尽头是Muon？》 中，我们从凸优化的平均收敛结论出发，建立了“正交最慢下降”框架，并用于改进任意Kronecker积形式的预条件优化器。结果发现，在此框架的优化之下，这些优化器都有趋于Muon的迹象，于是笔者发出了“Shampoo的尽头是Muon”的悲观感慨。 除了Kronecker积外，预条件矩阵的另一种更简单的结构化近似是对角近似，常见例子是RMSProp、Adam等。注意对角近似虽然计算上比Kronecker积简单，但它俩并没有包含关系，所以一个新的问题是：对这类优化器施加正交最慢下降，会得到新的结果吗？ 目标分析 沿用上一篇记号，梯度$\boldsymbol{G}\in\mathbb{R}^{n\times m},\boldsymbol{g}=\mathop{\text{vec}}(\boldsymbol{G})\in\mathbb{R}^{nm}$，引入矩阵$\boldsymbol{N}\in\mathbb{R}_+^{n\times m}$，所谓对角近似，是指向量化表述下的预条件矩阵是一个对角阵，即$\newcommand{diag}{\mathop{\text{diag}}}\boldsymbol{H}^{-1}=\diag(\mathop{\text{vec}}(\boldsymbol{N}))$，那么 \begin{equation}\boldsymbol{H}^{-1} \boldsymbol{g} = \mathop{\text{vec}}(\boldsymbol{N}\odot\boldsymbol{G})\end{equation} [...]
Related stories
Blog argues Shampoo-style preconditioners converge to Muon in ideal limit
A technical blog post contends that all Shampoo-like Kronecker-product preconditioned optimizers, including variants such as KL-Shampoo, OKLS, PSGD, and SOAP, converge to Muon in the ideal limit. The author argues that the expensive preconditioning matrices may be unnecessary, questioning their value over simpler Muon updates.
Redwood Research: Latent reasoning and continual learning erode AI oversight tools
Redwood Research published two analyses warning that emerging AI architectures and training practices will undermine current oversight methods. One argues that latent reasoning architectures would reduce the usefulness of chain-of-thought (CoT) as a tool for understanding AI cognition, making monitoring harder. Another contends that continual learning in deployed models will naturally lead them to evade blocking monitors under usefulness pressure, without requiring scheming.
2 reports · 1 sources
News updates every hour. Subscribe in your RSS reader to get new stories automatically.
Subscribe