Back to News

Making Training More Scientific (VIII): Learning Rates for Multi-Stage Training

#learning-rate#multi-stage-training#schedule-free#optimization

This blog post rethinks learning-rate scheduling from a multi-stage training perspective. Instead of aiming for a single optimal final point, it proposes a compromise objective where each stage ends near-optimal, making schedule-free training more practical and simplified.

Coverage timeline

  1. 科学空间 (苏剑林)苏剑林
    让炼丹更科学一些（八）：多阶段训练的学习率

    上篇文章 《让炼丹更科学一些（七）：步长调度与权重平均》 我们简单介绍了无调度（Schedule-Free）学习率的工作，它试图通过某种权重平均来替代学习率调度，实现常数学习率就可以训出最优模型的效果。然而，上文我们也提到，如果不引入额外假设，这个最优的常数学习率也是依赖于训练步数的，所以也做不到真正的无调度。 这篇文章我们从多阶段训练角度，来重新思考这个问题，主要思想是将调度目标折中为“每个阶段结束时都接近最优”，使其在实践上更为简化与可行。 [...]

Related stories

  • Tencent Hunyuan analyzes batch size impact on scaling RL training at 10K-100K GPU clusters

    Tencent Hunyuan's team published a technical analysis on scaling reinforcement learning (RL) training for clusters of 10,000 to 100,000 GPUs, focusing on batch size tuning and scaling. They found that moderately increasing batch size with adjusted learning rates improves throughput, but at 16B batch size the additional sample cost outweighs the gains, slowing time-to-target.

    2 reports · 2 sources

  • Chinese blogger relaxes convergence assumptions for adaptive gradient algorithms

    A Chinese machine-learning blogger extended convergence theory for adaptive gradient algorithms by removing two restrictive assumptions from a previous framework: constant learning rate and monotone preconditioner. The new analysis covers non-constant learning rate schedules and sliding-average methods like RMSProp and Adam, which were previously excluded.

    1 reports · 1 sources

  • Hacker News Discusses Claim of Over 10x Pretraining Efficiency

    A Hacker News thread on September 8, 2026, discusses a claim of more than 10x efficiency improvement in pretraining, likely referencing a specific technical method or result. The discussion focuses on the feasibility and implications of such a gain for AI research and development.

    1 reports · 1 sources117 points · 62 comments

  • Classic Adaptive Gradient Algorithm AdaGrad and Its Legacy

    The ninth article in the series introduces AdaGrad, the foundational adaptive gradient algorithm from a 2011 paper, which generalizes SGD's convergence results to preconditioned forms and derives the optimal preconditioner as the inverse square root of the gradient's second moment. It highlights AdaGrad's influence on subsequent optimizers like RMSProp and Adam, despite its own limited popularity.

    1 reports · 1 sources

News updates every hour. Subscribe in your RSS reader to get new stories automatically.

Subscribe