Back to News

Conic-Lifting Method Achieves Schedule-Free Convergence in Convex Optimization

#schedule-free#convex optimization#learning rate#conic lifting

A blog post by Su Jianlin analyzes the paper 'Conic-Lifting for Learning-rate-free Convex Optimization', which introduces a lifting construction that enables schedule-free learning rates. Under bounded-domain and convexity assumptions, the method provably converges to the optimal solution at an ideal rate without presetting a learning-rate curve or training steps. The post discusses the theoretical guarantees and potential implications for deep learning training.

Coverage timeline

  1. 科学空间 (苏剑林)苏剑林
    让炼丹更科学一些（十一）：无调度学习率新思路

    前面的多篇文章中，我们已经表明了学习率调度对终点收敛的重要性。与此同时，“无调度（Schedule-Free）学习率”一直也是训练人的理想追求：不预设学习率曲线，也不预设训练步数，训练过程随时终止，都能得到截止到当前步数的近乎最优的解。 笔者此前对这条路线的态度是比较悲观的，参考 《让炼丹更科学一些（八）：多阶段训练的学习率》 。但近日，论文 《Conic-Lifting for Learning-rate-free Convex Optimization》 通过一个精妙的提升构造实现了这一点！可以证明，在有界域和凸优化假设下，它真的能无调度地以理想速度收敛到最优解！ 它究竟是怎么做到的？深度学习的训练能否从中借鉴什么吗？本文一起来学习一下。 基础回顾 考虑常数学习率的投影更新规则 \begin{equation}\boldsymbol{\theta}_{t+1} = \Pi_{\boldsymbol{\Theta}}(\boldsymbol{\theta}_t - \eta \boldsymbol{g}_t)\end{equation} [...]

Related stories

  • Continuum Attention Paper Proposes Continuous Softmax Attention Formulation

    A Chinese blog post by 苏剑林 analyzes the Continuum Attention paper, which proposes a continuous formulation of softmax attention. The post discusses this within the broader 'continuize-discretize' research path, citing prior examples like EFLA and early SSM work.

    1 reports · 1 sources

  • Blog Post Links Parameter Symmetry to Effective Learning Rate

    A blog post by Su Jianlin synthesizes recent papers (Hyperball, HyperTransfer, ELR) that analyze model optimization through the parameter symmetry assumption l(w)=l(cw), arguing that optimizer designs aligned with this symmetry yield more robust training and reduce hyperparameter tuning burden. The author also discusses controversies around the symmetry assumption and reflects on how to build a more reliable foundation for this research direction.

    1 reports · 1 sources

  • New Route Linearizes Softmax Attention to TTT, Unifies DeltaNet and MesaNet

    In a blog post dated 2026-11-01, Su Jianlin proposes a new derivation that linearizes Softmax Attention into Test-Time Training (TTT) via three averaging steps. Within this unified framework, DeltaNet and MesaNet emerge as natural special cases, offering deeper insight into the approximations and trade-offs behind linear attention.

    1 reports · 1 sources

  • UCSB and Microsoft Propose ScholarEvolve for AI Agents to Read Papers and Self-Modify Code

    Researchers from UC Santa Barbara and Microsoft introduced ScholarEvolve, a method that automates the research process of reading literature, implementing algorithms, and evaluating results to enable AI agents to evolve their harness code without modifying underlying model parameters. The approach addresses the limitation of prior coding-agent-based harness optimization, which relied on the agent's existing knowledge and observed failures, by allowing agents to draw on the latest academic findings. The paper and code are publicly available.

    2 reports · 1 sources

News updates every hour. Subscribe in your RSS reader to get new stories automatically.

Subscribe