Back to News

Variational Method Proposed for Optimal Learning Rate Scheduling in Weight-Normalized Training

#learning rate scheduling#variational method#weight normalization#optimization

A blog post explains and extends a paper that estimates optimal learning rate schedules in weight-normalized training by simplifying training dynamics into a one-dimensional differential equation and applying variational principles to maximize a target under that constraint. The approach offers a method to translate training intuitions into schedule shapes rather than providing a single formula.

Coverage timeline

  1. 科学空间 (苏剑林)苏剑林

    近日,笔者刷到一篇论文 《Your LR Schedule May Not Be Principled: Effective Learning Rate Scheduling in Weight-Normalized Training》 ,里边分享了一种估计最优学习率的新思路:先将模型的训练动态简化成一个一维微分方程,然后在这个微分方程约束之下,基于变分思想去最大化某个目标,从而得到最佳调度。 这个过程最吸引人的地方,不是得到了一个具体公式,而是提供了一种把“训练直觉”翻译成“调度形状”的方法。这篇文章分享笔者阅读过后的理解,并由此做一些延伸。 更新规则 原论文考虑的是Weight Norm(下面简称“WN”)类优化器,所以我们也从WN出发,其更新规则为 \begin{equation}\boldsymbol{\theta}_{t+1} = \gamma\times\frac{\boldsymbol{\theta}_t + \eta_t \boldsymbol{u}_t}{\Vert\boldsymbol{\theta}_t + \eta_t \boldsymbol{u}_t\Vert}\end{equation} [...]

Related stories

  • Conic-Lifting Method Achieves Schedule-Free Convergence in Convex Optimization

    A blog post by Su Jianlin analyzes the paper 'Conic-Lifting for Learning-rate-free Convex Optimization', which introduces a lifting construction that enables schedule-free learning rates. Under bounded-domain and convexity assumptions, the method provably converges to the optimal solution at an ideal rate without presetting a learning-rate curve or training steps. The post discusses the theoretical guarantees and potential implications for deep learning training.

    1 reports · 1 sources
  • Blog Post Links Parameter Symmetry to Effective Learning Rate

    A blog post by Su Jianlin synthesizes recent papers (Hyperball, HyperTransfer, ELR) that analyze model optimization through the parameter symmetry assumption l(w)=l(cw), arguing that optimizer designs aligned with this symmetry yield more robust training and reduce hyperparameter tuning burden. The author also discusses controversies around the symmetry assumption and reflects on how to build a more reliable foundation for this research direction.

    1 reports · 1 sources
  • MIT: GPT-6 Astra edits Qwen weights, boosting MATH-500 from 7.9% to 48.1%

    MIT CSAIL researchers demonstrated that GPT-6 Astra can directly read and modify the weights of Qwen3.5-0.8B, improving its MATH-500 score from 7.9% to 48.1% by altering only 0.016% of parameters, without training or gradient descent. The team named this capability Weight-Space Programming. However, spaCy author Matthew Honnibal publicly challenged the paper, arguing that the experimental conditions across the three claimed abilities differ significantly and that the presentation is misleading.

    6 reports · 5 sources2949 points · 892 comments
  • UCSB and Microsoft Propose ScholarEvolve for AI Agents to Read Papers and Self-Modify Code

    Researchers from UC Santa Barbara and Microsoft introduced ScholarEvolve, a method that automates the research process of reading literature, implementing algorithms, and evaluating results to enable AI agents to evolve their harness code without modifying underlying model parameters. The approach addresses the limitation of prior coding-agent-based harness optimization, which relied on the agent's existing knowledge and observed failures, by allowing agents to draw on the latest academic findings. The paper and code are publicly available.

    2 reports · 1 sources

News updates every hour. Subscribe in your RSS reader to get new stories automatically.

Subscribe