HKU team unveils GRPO's 'Implicit Advantage Symmetry' and proposes A-GRAE at NeurIPS 2026
Researchers from the University of Hong Kong, led by Prof. 屈靓琼, presented a unified perspective at NeurIPS 2026 explaining two known flaws of GRPO—insufficient exploration and difficulty insensitivity—under the name 'Implicit Advantage Symmetry'. They proposed the A-GRAE algorithm, which improves both Pass@1 accuracy and Pass@k exploration without extra computational cost. The paper and open-source code are available.
Coverage timeline
机器之心机器之心
随着 DeepSeek-R1 的出圈,强化学习尤其是 GRPO 已经作为大模型后训练的一种标准范式。然而,GRPO 并非完美无缺,近期的研究指出了它的两个明显缺陷: 能力边界没有拓宽(探索不足) :GRPO 更多是在模型现有的能力圈子里做 “跷跷板” 游戏(探索与利用的权衡),甚至在大采样预算下(如 Pass@256),其表现反而低于基础模型。 不懂得 “因材施教”(难度不敏感) :GRPO 的奖励机制忽略了题目的实际难度和模型当前的掌握程度,导致训练效率低下。 针对这一现状,近日来自香港大学的屈靓琼教授团队首次提出了一个统一的视角来解释这两大缺陷,并将其命名为 “ 隐式优势对称性(Implicit Advantage Symmetry) ”。基于这一理论视角,团队提出了全新的算法 A-GRAE ,在不增加额外计算成本的前提下,同时显著提升了模型的准确率(Pass@1)与探索能力(Pass@k)。论文获 NeurIPS 2026 录用,代码已开源。 论文题目:Unveiling Implicit Advantage Symmetry: Why GRPO Struggles with Exploration and Difficulty Adaptation 录用会议:NeurIPS 2026 论文地址:https://arxiv.org/pdf/2602.05548 代码仓库:https://github.com/HKU-HealthAI/A-GRAE 第一作者:香港大学博士生余志齐 一、研究动机: GRPO 为何会陷入双重瓶颈? 在可验证奖励强化学习中,GRPO 会让模型针对同一道题生成多条回答,再由验证器打分。以二值奖励为例,答对得 1 分,答错得 0 分。组相对优势估计将每条回答的奖励减去组内均值,再除以标准差。一个简单的标准化步骤,同时决定了组内如何奖惩、题目之间如何分配学习信号。研究团队指出,这种组相对优势中隐藏着一种未被察觉的 “对称性”,这正是阻碍模型进一步进化的罪魁祸首。这种对称性体现在两个层面: 图 1:GRPO 中的双重隐式优势对称性问题。在组内级别,正负轨迹的优势权重相等,导致低概率正确路径的 logits 无法更新,阻碍探索;在样本级别,中等难度样本的绝对优势值之和最大,导致困难数据训练不足。 组级别的对称 :在组内相对优势计算中,正确轨迹和

Related stories
FAME: Flow-Driven Multi-Anchor Exploration for Active 3D Mapping Accepted as NeurIPS 2026 Spotlight
Researchers from Tianjin University and Hefei University of Technology proposed FAME, a flow-driven multi-anchor exploration method for open-environment active 3D mapping. Unlike traditional methods that predict a single exploration goal, FAME uses Conditional Flow Matching to generate multiple possible goals, combined with path planning, exploration mode clustering, and hierarchical selection. The paper was accepted as a Spotlight at NeurIPS 2026, with 292 out of 7,900 accepted papers selected for this distinction.
1 reports · 1 sources

UCSB and Microsoft Propose ScholarEvolve for AI Agents to Read Papers and Self-Modify Code
Researchers from UC Santa Barbara and Microsoft introduced ScholarEvolve, a method that automates the research process of reading literature, implementing algorithms, and evaluating results to enable AI agents to evolve their harness code without modifying underlying model parameters. The approach addresses the limitation of prior coding-agent-based harness optimization, which relied on the agent's existing knowledge and observed failures, by allowing agents to draw on the latest academic findings. The paper and code are publicly available.
2 reports · 1 sources

Tsinghua's SkillMaster Trains LLM Agents to Create and Refine Skills via RL
Researchers from Tsinghua University and collaborators present SkillMaster, a NeurIPS 2026 paper that trains LLM agents to autonomously create, update, and retain skills through reinforcement learning, integrating skill management into the agent's own policy. The method uses two learning signals—making previously failed similar tasks easier and previously successful tasks more efficient—to drive skill editing, enabling skill evolution and task execution to be learned jointly.
1 reports · 1 sources

Zhejiang University and TeleAI Propose GR-BC for Robotic Manipulation, Accepted at CoRL 2026
Zhejiang University and TeleAI proposed Graph-Reweighted Behavior Cloning (GR-BC), a method that builds a graph from multiple robot attempts and uses graph search to identify effective actions for precise manipulation. The work was accepted at CoRL 2026, with paper, code, and project page publicly available.
1 reports · 1 sources

News updates every hour. Subscribe in your RSS reader to get new stories automatically.
Subscribe