Back to News

Chinese blogger relaxes convergence assumptions for adaptive gradient algorithms

#adaptive gradient#convergence theory#machine learning#optimization

A Chinese machine-learning blogger extended convergence theory for adaptive gradient algorithms by removing two restrictive assumptions from a previous framework: constant learning rate and monotone preconditioner. The new analysis covers non-constant learning rate schedules and sliding-average methods like RMSProp and Adam, which were previously excluded.

Coverage timeline

  1. 科学空间 (苏剑林)苏剑林
    让炼丹更科学一些（十）：单调性假设的拆与补

    上文 《让炼丹更科学一些（九）：经典自适应梯度算法》 中，我们为形如$\boldsymbol{\theta}_{t+1} = \boldsymbol{\theta}_t - \eta_t \boldsymbol{H}_t^{-1}\boldsymbol{g}_t$的一般更新规则建立了收敛框架，并以它为出发点重新理解了AdaGrad等经典自适应梯度算法。不过，当时的完整结论依赖于两个前提：一是学习率$\eta_t$取常数，二是预条件矩阵$\boldsymbol{H}_t$关于$t$单调不减（半正定意义下）。 前者限制了学习率策略的自由度，后者则把RMSProp、Adam这类滑动平均型算法排除在了理论之外。本文尝试逐一拆掉这两个“脚手架”，把收敛理论推广到更一般的情形。 遗留问题 先回顾上一篇文章的核心结论。对一般的更新规则 \begin{equation}\boldsymbol{\theta}_{t+1} = \boldsymbol{\theta}_t - \eta_t \boldsymbol{H}_t^{-1}\boldsymbol{g}_t\end{equation} [...]

Related stories

  • GDN2 update matrix numerical stability issues analyzed

    A technical blog post analyzes numerical stability issues in the GDN2 linear attention update matrix, which generalizes DeltaNet's I - eta k k^T form to I - eta k j^T. The post notes that while DeltaNet's original form with L2 normalization and eta <= 2 ensures stable products, GDN2's generalization may introduce new instability problems, and it proposes ways to make GDN2 more stable.

    1 reports · 1 sources

  • Sakana AI releases paper and code for augmented Lagrangian predictive coding

    Sakana AI has published a paper and open-sourced code for an augmented Lagrangian predictive coding (PC-ALM) method, proposed as an alternative to backpropagation for training neural networks. The method is detailed in an arXiv paper and the implementation is available on GitHub.

    1 reports · 1 sources126 points · 45 comments

  • Making Training More Scientific (VIII): Learning Rates for Multi-Stage Training

    This blog post rethinks learning-rate scheduling from a multi-stage training perspective. Instead of aiming for a single optimal final point, it proposes a compromise objective where each stage ends near-optimal, making schedule-free training more practical and simplified.

    1 reports · 1 sources

  • Classic Adaptive Gradient Algorithm AdaGrad and Its Legacy

    The ninth article in the series introduces AdaGrad, the foundational adaptive gradient algorithm from a 2011 paper, which generalizes SGD's convergence results to preconditioned forms and derives the optimal preconditioner as the inverse square root of the gradient's second moment. It highlights AdaGrad's influence on subsequent optimizers like RMSProp and Adam, despite its own limited popularity.

    1 reports · 1 sources

News updates every hour. Subscribe in your RSS reader to get new stories automatically.

Subscribe