Blog argues Shampoo-style preconditioners converge to Muon in ideal limit
A technical blog post contends that all Shampoo-like Kronecker-product preconditioned optimizers, including variants such as KL-Shampoo, OKLS, PSGD, and SOAP, converge to Muon in the ideal limit. The author argues that the expensive preconditioning matrices may be unnecessary, questioning their value over simpler Muon updates.
Coverage timeline
科学空间 (苏剑林)苏剑林
众所周知， Muon 优化器流行起来已经有好一段时间，虽然说不上完全取代Adam，但已经算得上不能忽视的、极具竞争力的优化器。与此同时，这两年Shampoo系优化器也“不甘落后”，除了原始Shampoo外，还演变出 KL-Shampoo 、 OKLS 、 PSGD 、 SOAP 等变体，都声称获得了比Muon更好的结果。 然而，本文将会泼一盆冷水：所有类似于Shampoo的、基于Kronecker积的预条件优化器，其理想极限可能都是Muon，我们花大代价所做的预条件矩阵，可能都是“无用功”。 平均收敛 我们从 《让炼丹更科学一些（十）：单调性假设的拆与补》 得到的平均收敛结论出发：在凸性假设下，更新规则$\boldsymbol{\theta}_{t+1} = \boldsymbol{\theta}_t - \eta_t \boldsymbol{H}_t^{-1}\boldsymbol{g}_t$成立如下不等式 [...]
Related stories
Blog post: Diagonal preconditioners like Adam also converge toward Muon under 'orthogonal slowest descent'
In a technical blog post dated 2026-10-15, the author extends their 'orthogonal slowest descent' framework from Kronecker-product preconditioners to diagonal approximations such as RMSProp and Adam. The analysis suggests that applying this framework to diagonal preconditioners also drives them toward the Muon optimizer, reinforcing the earlier claim that 'Shampoo's end is Muon'.
Matt Levine: Coordinated AI Safety Slowdown Could Resemble Antitrust Conspiracy
Bloomberg columnist Matt Levine commented that a coordinated slowdown by AI labs for safety reasons could be interpreted as an antitrust conspiracy to limit output and protect frontier model margins. He discussed this in the context of AI and antitrust, along with other topics like lockups and indexes.
1 reports · 1 sources
News updates every hour. Subscribe in your RSS reader to get new stories automatically.
Subscribe