Study: Complex Harness Evolution May Not Beat Simple Retrying for Agents
Researchers from AI2 and the University of Washington published a paper rethinking how to evaluate harness evolution for AI agents. Under matched feedback and inference budgets, complex self-modification did not consistently outperform simple test-time scaling (retrying tasks multiple times), and gains largely vanished on new tasks. The findings question the validity of reported improvements from agent self-evolution.
Coverage timeline
机器之心机器之心
论文标题：Rethinking the Evaluation of Harness Evolution for Agents 论文链接：https://arxiv.org/abs/2607.12227 代码：https://github.com/rethinking-harness-evolution 博客：https://yikee.github.io/harnessevolution/ 一个 Agent 第一次没把任务做对。 接下来，你有五份额外的推理预算，可以有很多种花法。 你可以让它把同一道题重新做几遍，从多个结果里挑最好的；可以把上一次失败的过程交给它，让它接着修；也可以做一件听起来更 “高级” 的事 —— 让 Agent 分析自己的失败，修改 Prompt、工具、Memory 和控制逻辑，甚至自动重写整个运行框架。 最后一种思路，正在成为 Agent 研究里一个很热门的方向： Harness Evolution 。 它背后的愿景非常吸引人。今天是工程师观察 Agent 为什么失败，然后修改 Prompt、添加工具、设计 Memory、调整 workflow；未来，也许这些工作可以交给 Agent 自己。Agent 不只是完成任务，还能够修改支撑自己工作的系统，从而一轮比一轮更强。 但这里隐藏着一个很容易被忽略的问题： 当 “自我进化” 的 Agent 得分提高时，我们怎么知道它是真的学会了更好的工作方式，而不是因为它比普通 Agent 多获得了几次尝试机会？ AI2 和华盛顿大学的一项最新研究，专门把这个问题摆到了台面上。 研究者做的事情并不复杂：把 Harness Evolution 和最简单的 “多跑几遍” 放到尽可能公平的条件下，给它们相同的反馈、相近的 inference budget，然后看看究竟谁更有效。 结果有些出人意料。在他们的实验中，复杂的 Harness Evolution 并没有稳定胜过简单的 test-time scaling。更值得注意的是，当进化出来的 Harness 被拿去解决从未参与优化的新任务时，提升只剩下了很小的一部分。 这意味着，我们可能需要重新理解过去一些所谓 “Agent 自我进化” 的提升。 Harness，其实就是模型外面的那套 “操作系统”。一个大模型真正成为 Agent，通常不只是因为模型本身。模型外面还
Related stories
Google Researchers Propose RRSI to Curb Overfitting in Recursive Self-Improving Agents
Google researchers have introduced RRSI (Regularized Recursive Self-Improvement), a method that adds regularization to recursive self-improving agents to prevent overfitting. The approach targets harness evolution, where agents iteratively modify their own prompts, tool usage, and memory management based on a fixed set of tasks, which can lead to benchmark-specific fitting and evaluation noise chasing. The paper, code, and project page are available online.
1 reports · 1 sources
NTU Study: Agent Harness Quality Depends on Model-Task Combination
A team led by An Bo at Nanyang Technological University published a report systematically studying model-harness interactions across agent tasks. The study compared four configurable harnesses (OpenHands, DSH, PI, openJiuwen) with five models and two native pairings, concluding that harness quality is not a fixed property but depends on the specific model-task combination, recommending joint evaluation over seeking a universal harness.
ModularRSI: Frozen-Weight Agents Improve via Harness Self-Modification
IQuest Research, with Beihang University and the University of Manchester, released ModularRSI, a method for agent continuous self-improvement that modifies the execution harness while keeping the base model frozen. On Terminal-Bench 2.0, accuracy rose from 47.57 to 52.43 through harness evolution, with improvements transferring to unseen tasks, domains, and base models. The work defines an agent as A=(M,H), where H is the harness controlling environment observation, context maintenance, tool invocation, failure recovery, and task termination.
Shanghai AI Lab Proposes SHE and SafeEvolve to Evolve Agent Safety Harnesses from Execution Trajectories
Shanghai AI Lab, with Fudan University, Shanghai Jiao Tong University, Hong Kong University of Science and Technology, and Zhejiang University, proposed SHE and SafeEvolve, two methods that evolve AI agent safety harnesses from full execution trajectories. SHE decomposes the harness into four components—System Prompt, Rule Bank, Safety Memory, and Tool Policy—to enable targeted updates, while SafeEvolve explores how these experiences can be absorbed by the policy model. This addresses multi-step agent risks such as hidden instructions, tool over-permission, and memory drift, which traditional prompt-only checks fail to cover.
News updates every hour. Subscribe in your RSS reader to get new stories automatically.
Subscribe