Back to News

NTU Study: Agent Harness Quality Depends on Model-Task Combination

#agent#harness#model-evaluation#ntu

A team led by An Bo at Nanyang Technological University published a report systematically studying model-harness interactions across agent tasks. The study compared four configurable harnesses (OpenHands, DSH, PI, openJiuwen) with five models and two native pairings, concluding that harness quality is not a fixed property but depends on the specific model-task combination, recommending joint evaluation over seeking a universal harness.

Coverage timeline

  1. 机器之心机器之心

    开发 Agent 应用时,大家一定都为选择 Harness 纠结过:同一个模型接入不同 Harness,工具调用、上下文管理和错误恢复的方式都会发生变化,最终表现也可能相差很大。一套 Harness 在某类任务上表现突出,换到另一类任务后,却未必仍是最佳选择。 那么,不同任务究竟该选择哪套 Harness?是否存在一套能够稳定适配多种任务的通用方案?模型厂商提供的原生 Harness,又是否一定更适合自家模型? 新加坡南洋理工大学安波教授团队在最新报告《Finding the Right Fit: Model–Harness Interactions across Agent Tasks》中,对这些问题进行了系统研究。 报告链接: https://arxiv.org/abs/2610.00917 实验代码仓:https://github.com/liyix/finding-the-right-fit 数据集:https://huggingface.co/datasets/yixuanli97/finding-the-right-fit 这篇报告不是在做一张简单的 Harness 排行榜,而是把模型与 Harness 视为一个整体进行比较。实验覆盖 OpenHands、DSH、PI 和 openJiuwen 四套可配置 Harness,并将 Codex–GPT 与 Claude Code–Claude 两组原生搭配作为参照。报告最终指向了一个很实际的结论:Harness 的好坏并不是固定属性,而是取决于它与模型、任务之间的具体组合。与其寻找一套 “放之四海而皆准” 的 Harness,不如围绕真实任务,对模型与 Harness 进行联合评估。 01 实验背景 为了考察 Harness 对 Agent 表现的影响,研究首先选取了四种可自由配置模型的 Harness: OpenHands、DeepSeek Harness(DSH)、PI 和 openJiuwen。 实验分别为这四种 Harness 接入 Claude Opus 5、GPT-6 Astra、GLM-5.3、Kimi K3 和 DeepSeek V4 Pro。这样的交叉设计,可以观察同一个模型更换 Harness 后的表现变化,也能比较同一套 Harness 面对不同模型和任务时是否依然稳定。 除此之外,研究

Related stories

  • Theory of Agent Survey: How to Split Capabilities Between Model and Harness

    Researchers from eight institutions, including Beihang University and the University of Edinburgh, published a survey titled 'Theory of Agent: The Science of Internalization and Externalization for LLM-based Agents'. The survey reviews about 600 recent studies, proposing a unified framework of internalization and externalization to analyze how LLM agents should decide which capabilities to embed in the model versus delegate to external harnesses, covering learning, alignment, evolution, and evaluation.

    1 reports · 1 sources
  • Study: Complex Harness Evolution May Not Beat Simple Retrying for Agents

    Researchers from AI2 and the University of Washington published a paper rethinking how to evaluate harness evolution for AI agents. Under matched feedback and inference budgets, complex self-modification did not consistently outperform simple test-time scaling (retrying tasks multiple times), and gains largely vanished on new tasks. The findings question the validity of reported improvements from agent self-evolution.

    1 reports · 1 sources
  • Shanghai AI Lab Proposes SHE and SafeEvolve to Evolve Agent Safety Harnesses from Execution Trajectories

    Shanghai AI Lab, with Fudan University, Shanghai Jiao Tong University, Hong Kong University of Science and Technology, and Zhejiang University, proposed SHE and SafeEvolve, two methods that evolve AI agent safety harnesses from full execution trajectories. SHE decomposes the harness into four components—System Prompt, Rule Bank, Safety Memory, and Tool Policy—to enable targeted updates, while SafeEvolve explores how these experiences can be absorbed by the policy model. This addresses multi-step agent risks such as hidden instructions, tool over-permission, and memory drift, which traditional prompt-only checks fail to cover.

    1 reports · 1 sources
  • ModularRSI: Frozen-Weight Agents Improve via Harness Self-Modification

    IQuest Research, with Beihang University and the University of Manchester, released ModularRSI, a method for agent continuous self-improvement that modifies the execution harness while keeping the base model frozen. On Terminal-Bench 2.0, accuracy rose from 47.57 to 52.43 through harness evolution, with improvements transferring to unseen tasks, domains, and base models. The work defines an agent as A=(M,H), where H is the harness controlling environment observation, context maintenance, tool invocation, failure recovery, and task termination.

    1 reports · 1 sources

News updates every hour. Subscribe in your RSS reader to get new stories automatically.

Subscribe