Back to News

Forward-only LLM adaptation via SVD rank selection proposed at NeurIPS 2026

#llm#adaptation#forward-only#svd

A NeurIPS 2026 paper from Lund University and Google DeepMind introduces an efficient LLM adaptation method that uses only forward passes, avoiding backpropagation. The approach leverages SVD rank selection based on activation angle concentration to identify and prune weight matrices during inference, enabling task-specific adaptation without extra resource overhead. Experiments show lower peak memory and higher speedup compared to gradient-based methods like LASER-Grad.

Coverage timeline

  1. 机器之心机器之心

    随着人工智能(AI)系统的大规模部署,AI 服务正成为能源消耗增长的关键驱动因素,尤其是在大型语言模型(LLM)推理服务方面。LLM 服务过程中,基于真实用户交互生成的大规模 LLM 推理促使我们重新思考 如何利用推理过程本身来改进服务系统 ,特别是实现高效的特定任务 LLM 自适应。 主流的 LLM 自适应方法基于调优技术(例如,参数高效微调(PEFT)、提示调优等)和知识增强技术(例如,检索增强生成(RAG)、提示工程等)。知识增强技术主要通过注入知识来实现,并且与 LLM 推理过程中修改内部权重无关。调优技术通常依赖于反向传播,与推理本身相比会产生额外的资源开销。相反, 新兴的仅前向学习方法[1],仅依赖于前向传播,正成为 LLM 推理中一种很有前景的自适应方法。 一句话概括:LLM 在推理的过程中不断进化, 在仅前向(无后向传播)中实现对下游任务的自适应 。 论文标题:Efficient LLM Adaptation with Forward-Only Passes 研究团队:隆德大学(瑞典), Google DeepMind (纽约) 论文链接:https://arxiv.org/abs/2611.10610 NeurIPS 官方页面:https://neurips.cc/virtual/2026/loc/atlanta/poster/148809 核心贡献:利用 LLM 前向传播中激活值的角度集中度来衡量参数 SVD 中不同 rank 对激活值的贡献并对相应的 ranks 进行裁剪 本文旨在实现 LLM 推理服务过程中仅通过前向传播即可高效自适应 LLM 。我们提出仅前向搜索来加速所选权重矩阵的识别,其方法是利用奇异值分解(SVD)分量引起的激活与隐藏状态之间的角度集中度差异。每个 SVD 分量引起的激活的角度集中度用于衡量其对隐藏状态角度集中度的贡献——分散或集中。我们的前向搜索包含以下步骤: 第一,具有多个 SVD 分量的 SVD 分解 第二,所有 SVD 分量的激活值之和可恢复标准的前向传播 第三,角度集中度计算 前向传播测量角度集中度,以确定哪个权重矩阵具有最高的离散度,该权重矩阵随后用于降秩运算。 结果 结果表明,与基于梯度的方法(LASER-Grad)相比,我们的前向方法具有更低的峰值内存和更高的加速比;与穷举搜索方法(LASER)相比,我们

Related stories

  • Attention Sinks Induce Gradient Sinks; Massive Activations Regulate Gradients via RMSNorm

    In a NeurIPS 2026 paper, researchers from Tsinghua and Peking University show that attention sinks concentrate training signals on a few tokens during backpropagation, forming 'gradient sinks.' They find that massive activations can attenuate these locally concentrated gradients through RMSNorm, suggesting that large activations may act as gradient regulators.

    1 reports · 1 sources
  • NV-CoT: Multimodal LLMs Predict Continuous Coordinates for Visual Chain-of-Thought, NeurIPS 2026 Oral

    Researchers from NTU, USTC, NUS, and Microsoft propose NV-CoT (Numerical Visual Chain-of-Thought), a method that lets multimodal LLMs directly predict continuous numerical coordinates for region selection instead of generating bounding boxes as text tokens. The work, accepted as a NeurIPS 2026 Oral (top 0.4%), supports supervised fine-tuning and reinforcement learning, addressing mismatches between token-level training and spatial error.

    1 reports · 1 sources
  • Tsinghua's SkillMaster Trains LLM Agents to Create and Refine Skills via RL

    Researchers from Tsinghua University and collaborators present SkillMaster, a NeurIPS 2026 paper that trains LLM agents to autonomously create, update, and retain skills through reinforcement learning, integrating skill management into the agent's own policy. The method uses two learning signals—making previously failed similar tasks easier and previously successful tasks more efficient—to drive skill editing, enabling skill evolution and task execution to be learned jointly.

    1 reports · 1 sources
  • Lev: Recurrent Decision Model Adapts Compute per Problem, No Chain-of-Thought

    Researchers from Lehigh University, UBC, and Vector Institute propose Lev, a decision model that uses recurrent computation instead of chain-of-thought to adaptively spend more compute on hard problems. Lev reads out option probabilities after each recurrent pass, trains jointly across depths, and uses a lightweight stopper to decide how many passes to run, enabling a single set of weights to answer at arbitrary depth. The authors describe this as System 1.5, preserving the direct interface of System 1 while allowing extra compute on difficult cases like System 2.

    1 reports · 1 sources

News updates every hour. Subscribe in your RSS reader to get new stories automatically.

Subscribe