Back to News

Blog Post Links Parameter Symmetry to Effective Learning Rate

#parameter symmetry#learning rate#optimization#hyperparameter tuning

A blog post by Su Jianlin synthesizes recent papers (Hyperball, HyperTransfer, ELR) that analyze model optimization through the parameter symmetry assumption l(w)=l(cw), arguing that optimizer designs aligned with this symmetry yield more robust training and reduce hyperparameter tuning burden. The author also discusses controversies around the symmetry assumption and reflects on how to build a more reliable foundation for this research direction.

Coverage timeline

  1. 科学空间 (苏剑林)苏剑林
    从参数对称性到有效学习率

    近来，越来越多的论文或者博客，开始基于参数的对称性来分析和指导模型的优化，比如 Hyperball 、 HyperTransfer 、 ELR 等，它们总体都是围绕着$\ell(\boldsymbol{w})=\ell(c\boldsymbol{w})$这一对称性假设来展开探索。如果我们让优化器的设计也“顺应”相应的对称性，往往能让训练过程更为稳健，也能降低超参数调优的心智负担。 然而，对称性假设本身也有一些争议的地方，如何为这条路找到更可靠的基础，让它可以更合理地走得更远？这篇文章分享一下笔者的思考。 结果回顾 这一节我们先来整理一下开头提到的三篇文章 Hyperball 、 ELR 、 HyperTransfer 的逻辑线。 [...]

Related stories

  • vLLM Achieves Near-Zero-Downtime Scaling with Asynchronous Preparation and NIXL EP

    The vLLM team published a blog post describing how asynchronous preparation and NIXL EP enable near-zero-downtime scaling for Elastic Expert Parallelism (EP). The approach reduces downtime during dynamic scaling of expert-parallel models, improving operational efficiency for large-scale inference.

    1 reports · 1 sources

  • UCSB and Microsoft Propose ScholarEvolve for AI Agents to Read Papers and Self-Modify Code

    Researchers from UC Santa Barbara and Microsoft introduced ScholarEvolve, a method that automates the research process of reading literature, implementing algorithms, and evaluating results to enable AI agents to evolve their harness code without modifying underlying model parameters. The approach addresses the limitation of prior coding-agent-based harness optimization, which relied on the agent's existing knowledge and observed failures, by allowing agents to draw on the latest academic findings. The paper and code are publicly available.

    2 reports · 1 sources

  • Goodfire researcher argues neural networks are largely interpretable, most neurons understandable

    Nick Cammarata of Goodfire published an essay arguing that neural networks are far more interpretable than previously believed, with most neurons in models up to trillions of parameters understandable. He attributes polysemanticity in early models to fixable issues rather than superposition, and suggests mechanistic interpretability may be solved soon.

    3 reports · 3 sources

News updates every hour. Subscribe in your RSS reader to get new stories automatically.

Subscribe