Back to News

Classic Adaptive Gradient Algorithm AdaGrad and Its Legacy

#adagrad#optimizer#machine-learning#gradient-descent

The ninth article in the series introduces AdaGrad, the foundational adaptive gradient algorithm from a 2011 paper, which generalizes SGD's convergence results to preconditioned forms and derives the optimal preconditioner as the inverse square root of the gradient's second moment. It highlights AdaGrad's influence on subsequent optimizers like RMSProp and Adam, despite its own limited popularity.

Coverage timeline

  1. 科学空间 (苏剑林)苏剑林
    让炼丹更科学一些（九）：经典自适应梯度算法

    本系列前面八篇文章，都是在围绕SGD及其学习率讨论。而从本文开始，我们将正式进入自适应梯度算法的世界。可以说，现在所有的自适应梯度算法，都有一个共同的源头，那就是2011年的经典之作 《Adaptive Subgradient Methods for Online Learning and Stochastic Optimization》 ——也就是著名的AdaGrad论文。 这篇论文将SGD的经典收敛结论推广到一般的预条件矩阵形式，继而推导出最优的预条件矩阵正是梯度的二阶矩（的$-1/2$次幂），然后取对角线近似得到AdaGrad。虽然AdaGrad本身不算太流行，其启发的后续工作如RMSProp、Adam等，是当之无愧的主流优化器。不仅如此，后续的Shampoo、KL-Shampoo等优化器，都还算是它的传承。 本文就让我们一起来重温这篇经典作品。 [...]

Related stories

  • Chinese blogger relaxes convergence assumptions for adaptive gradient algorithms

    A Chinese machine-learning blogger extended convergence theory for adaptive gradient algorithms by removing two restrictive assumptions from a previous framework: constant learning rate and monotone preconditioner. The new analysis covers non-constant learning rate schedules and sliding-average methods like RMSProp and Adam, which were previously excluded.

    1 reports · 1 sources

  • Making Training More Scientific (VIII): Learning Rates for Multi-Stage Training

    This blog post rethinks learning-rate scheduling from a multi-stage training perspective. Instead of aiming for a single optimal final point, it proposes a compromise objective where each stage ends near-optimal, making schedule-free training more practical and simplified.

    1 reports · 1 sources

  • Anthropic reports automated AI researchers mitigate alignment failures across all 10 benchmarks

    Anthropic released a report on its Automated Alignment Researcher (AAR) project, where Claude agents autonomously search papers, propose methods, generate data, and train models to fix alignment failures. The system improved performance on all 10 misalignment benchmarks without degrading overall performance, and outperformed human researchers on some tasks at a fraction of the cost.

    4 reports · 4 sources

  • New Interpretation of Optimizer Momentum as Online Regression Solution

    In a blog post, Su Jianlin proposes viewing the momentum term in optimizers not merely as a gradient average but as the solution to an online regression problem. This perspective naturally leads to recent optimizer works, offering a new theoretical lens for understanding and designing optimizers.

    1 reports · 1 sources

News updates every hour. Subscribe in your RSS reader to get new stories automatically.

Subscribe