Harness Attribution: From Scores to Reliability
Highlights
This issue's strongest signal is the shift in harness attribution from "does changing the harness change the score" to "does the harness itself faithfully fulfill its obligations." Trial by Trace is the first to make harness reliability a two-layer evaluation benchmark independent of task success rate, using a seven-dimensional reliability taxonomy (context/execution/governance/lifecycle/observability/tools/verification) to construct 95 mechanism-oriented tasks, scoring the Agent layer and Harness layer separately. The core evidence is that 31.1% of runs succeed on the task yet violate at least one harness standard—hard evidence of the blind spot in outcome-only evaluation. When Agents Fail provides from the other side the "Valid Completion Rate (VCR)" and "model-harness fit," quantifying on 52,168 trajectories the phenomena where non-clean runs still earn task credit and clean runs fail to solve the task, and finding that model-harness interaction residuals change sign across benchmarks. Rethinking Verification treats verification as a systematically tunable test-time scaling axis, proving on a 17 design × 7 model matrix that independent review outperforms self-checking at only 1.00–1.25× cost, and changes-based review outperforms trajectory-based—directly challenging the intuition that giving reviewers more trajectory context is necessarily better.
Skeptical evidence is equally dense this issue. Harness Updating Is Not Harness Benefit uses controlled decomposition to split self-evolving agent gains into "harness updates" and "harness benefits," finding the former flat on base capabilities, the latter non-monotonic, and weak models often unable to activate harness artifacts. SMU's 44-point full factorial study quantifies on 6 harnesses × 3 models, 1,800 runs, that the harness alone can swing SWE-bench scores by 44 percentage points, and harness rankings flip across models. MCISLAB DeepRead performs a statistical reanalysis of existing harness evolution papers, proving that the reported 16-percentage-point resolve rate dispersion falls within binomial sampling noise for 50 tasks, while token efficiency trends are robust—reminding us to audit power even for negative results.
On the cost and stopping strategy side, two interlocking findings emerge. Continue or Restart uses a large-scale multi-setting empirical study on 147,002 trajectories to prove that early execution prefixes have almost no discriminative power for attempts within the same task (AUC≈0.50), and provides an exact algebraic decomposition: the gain of learned stopping over random restart comes almost entirely from inter-task allocation (avoiding stops on simple issues), while intra-task attempt selection contributes nearly zero (-0.11 to +0.24pp); it also proves that on the shared budget frontier, uninterrupted continuation outperforms cold restart—directly challenging the value of early stopping. The Physical Cost of Context treats "prefix cache hit rate" as a first-class variable in physical cost accounting, proving that deleting history can reverse token savings into an explosion of uncached prefill (Mask-5 saves 33.3% of prompt but increases uncached prefill by 608.5%). Compact-Bench uses a paired continuation protocol of full-history vs summary-only on the same repository snapshot to quantify the stable damage range (5–64%) across 14 summarizers, and proves that "preserving repair evidence ≠ preserving continuation progress."
Benchmark validity and open-world directions also see progress. Finding Side-Effect Loopholes is the first to port mutation testing to RLVR verifier auditing, specifically targeting "side-effect loopholes"—where an agent completes the task but produces unrequested side effects while the verifier only checks postconditions. On AppWorld and WorkArena, it finds repeated writes, extra fields, and other side effects that bypass the verifier, and three RL methods' reward formulas give full marks for repeated writes. PROBE uses "bugs actually fixed by maintainers" as the reference set for open-world bug discovery, proving that bugs found by agents and bugs fixed by maintainers are systematically different in type distribution—even if agents find real bugs, they may not be the ones maintainers care about. SWE-Shift treats "task-relevant changes during execution" as a first-class evaluation object, with all 17 systems degrading and zero paired success on F06/F11. On the community side, an answer leak audit finds that the current MiMo mirror still leaks the scope of hidden test edits through file modification times.
Community and Dynamics
Harness Updating Is Not Harness Benefit at EMNLP provides a controlled decomposition of self-evolving agent gains: harness update capability is flat on base capabilities, harness benefits are non-monotonic, and weak models often cannot activate harness artifacts—directly responding to "harness update ≠ harness benefit." SMU's full factorial study uses 6 harnesses × 3 models, 1,800 runs, to quantify that the harness alone can swing SWE-bench scores by 44 percentage points, and harness rankings flip across models—the strongest quantitative upper bound on model-harness confounding to date. The answer leak audit independently checks task mirrors of SWE-bench Verified/Pro and MiMo, finding that the v1 mirror of SWE-bench Verified still carries the entire future history of the project, while the current MiMo mirror, though cleaned of Git, still has file modification times pointing to the scope of hidden test edits. MCISLAB DeepRead performs a statistical reanalysis of existing harness evolution papers, proving that the reported 16-percentage-point resolve rate dispersion falls within binomial sampling noise for 50 tasks, while token efficiency trends are robust—reminding us to audit power even for negative results.
Open Questions
- Continue or Restart proves that early failure prediction has almost no discriminative power for attempts within the same task (AUC≈0.50), and the gain of learned stopping comes almost entirely from inter-task allocation. If stopping strategies cannot distinguish attempts within a task, should the computational budget shift entirely to inter-task allocation—and should "when to give up on a task" not be decided by the execution prefix at all?
- Trial by Trace proves that 31.1% of runs succeed on the task yet violate harness standards, and When Agents Fail quantifies that non-clean runs can still earn task credit. When outcome-only metrics are completely blind to harness reliability, should all harness comparison reports be required to include reliability/valid completion dimensions, rather than just pass rates?
- The Physical Cost of Context proves that deleting history can reverse token savings into a 608.5% explosion of uncached prefill. If "token savings ≠ physical cost savings," should reports on compression and memory methods uniformly switch to prefix-cache-aware physical accounting, and does the assumption that "compression is a free lunch" still hold under cache pricing?
- PROBE proves that bugs found by agents and bugs fixed by maintainers are systematically different. If the "usefulness" of open-world bug discovery is defined by maintainer priorities rather than agent self-reports, what is the correct success criterion for un-prompted bug discovery tasks—and how can it be made into a hackability-auditable benchmark?
Papers in this issue
Early execution prefixes from coding agents cannot distinguish successful from failed attempts on the same task, making uninterrupted continuation superior to cold restarts across nearly all budget configurations.
Editor's noteThe first large-scale, multi-setting empirical proof that early execution prefixes have almost no discriminative power for attempts within the same task (AUC≈0.50), with an exact algebraic decomposition: the gain of learned stopping over random restart comes almost entirely from inter-task allocation (avoiding stops on simple issues), while intra-task attempt selection contributes nearly zero (-0.11 to +0.24pp). It also proves that on the shared budget frontier, uninterrupted continuation outperforms cold restart—directly challenging the value of early stopping. For any researcher designing stopping strategies or auditing cost/throughput trade-offs, its decomposition template and budget-constrained replay are directly reusable.
TraceTrial-Bench reveals that 31.1% of AI agent runs succeed despite harness failures, proving task outcomes alone cannot characterize harness reliability.
Editor's noteThe first to make "harness reliability" a two-layer evaluation benchmark independent of task success rate: a seven-dimensional reliability taxonomy constructs 95 mechanism-oriented tasks, with trajectory tracing and multi-source evidence acquisition, scoring the Agent layer and Harness layer separately, and using explicit mechanism chains for failure attribution. The core evidence is that 31.1% of runs succeed on the task yet violate at least one harness standard—hard evidence of the blind spot in outcome-only evaluation. Its two-layer scoring and evidence boundary protocol can be directly reused for any harness comparison or audit. Limitations: the main benchmark has no held-out control, a single Judge identity, and no contamination audit.
All 17 coding agents fail to retain success when tasks change mid-execution, with tool/session changes and stale evidence causing zero paired success across all systems.
Editor's noteThe first to treat "task-relevant changes during execution" as a first-class evaluation object: using a matched Clean/Shifted continuation protocol to fork from the same saved state, separating "task failure itself" from "changes breaking successful continuation," covering 12 change families and 9 intervention positions. The result that all 17 systems degrade and F06/F11 have zero paired success is strong evidence for "harness update ≠ harness benefit," and proves that passing the original task tests can mask failures such as the removal of collaborator-side changes. The matched continuation protocol is a reproducible template for isolating "change effects" from "task difficulty." Limitations: the main benchmark is rebuilt from published benchmarks, a single update, and no held-out control.
Compact-Bench shows context compaction damages 5–64% of coding repairs, proving that retaining code details does not guarantee preserving task progress.
Editor's noteThe first to use "paired continuation of full-history vs summary-only on the same repository snapshot" as a first-class protocol for compression evaluation, providing three dimensions—task retention, loop detail retention, and production chain retention—using executable tests to determine whether compression breaks continuation ability, quantifying the stable damage range (5–64%) across 14 summarizers. The core new finding is that "preserving repair evidence ≠ preserving continuation progress": pytest-7571 retains 5/5 audit tokens yet still fails, directly supporting the decoupling of "compression information retention" from "task success." The paired-replay protocol and stable damage determination (2/3 runs) are directly reusable. Limitations: the main task set is only a 35-task SWE-bench subset, with no held-out control.
Independent review by a separate critic at the final stage delivers the best accuracy-to-cost tradeoff, improving agent performance by up to 9.5 points at only 1.25x baseline cost.
Editor's noteTreats "verification" as a systematically tunable test-time scaling axis in the harness, providing four-layer (L0-L3) accuracy-cost attribution on a large matrix of 17 designs × 7 models × 2 coding benchmarks + 18 vision tasks. The core finding is that independent review (separate critic) gives stable gains (TB +5.9~9.5pp, SWE +3.9~6.6pp) at only 1.00–1.25× cost, outperforming self-checking, and "changes-based review outperforms trajectory-based review" directly challenges the intuition that giving reviewers more trajectory context is necessarily better. Limitations: the main benchmark SWE-bench Verified is saturated and lacks a held-out contemporaneous new benchmark control.
Mutation testing of RLVR verifiers reveals all false positives stem from unrequested side-effect changes, not missing work, and simple count assertions can fix these loopholes.
Editor's noteThe first to systematically port mutation testing to RLVR verifier auditing, specifically targeting "side-effect loopholes"—where an agent completes the task but produces unrequested side effects while the verifier only checks postconditions, not frame conditions. Using "end-state checking" as an oracle to distinguish harmless edits from false positives, it proves that all false positives come from unrequested side effects rather than missing or incorrect work; on AppWorld and WorkArena, it finds repeated writes, extra fields, and other side effects that bypass the verifier, and three RL methods' reward formulas give full marks for repeated writes. The end-state checking protocol and count assertion fixes are directly reusable for any RLVR benchmark audit. Limitations: only two benchmarks, hand-written edits, and no trained policy to verify whether RL would actually exploit them.
Agent benchmark scores measure model-harness pairs jointly, not model capability alone; harness choice can reverse model rankings.
Editor's noteThe first to make "Valid Completion Rate (VCR)" and "model-harness fit" first-class evaluation objects, systematically distinguishing task success from execution validity, providing cross-model, cross-benchmark interaction residuals and resource profiling, based on a large-scale paired evaluation of 52,168 trajectories. It quantifies the phenomena where non-clean runs still earn task credit and clean runs fail to solve the task, and the finding that model-harness interaction residuals change sign across benchmarks. "Task success and valid completion are separable" is an important warning for any evaluation using pass rate as the sole metric. Limitations: the main benchmark SWE-bench-45 is a subset of a saturated benchmark with no held-out control.
PROBE benchmark shows frontier coding agents find under 21% of maintainer-fixed bugs, discovering real but systematically different bugs focused on local errors rather than input handling.
Editor's noteThe first to use "bugs actually fixed by maintainers" as the reference set for repository-level open-world bug discovery, systematically quantifying the systematic difference in type distribution between bugs found by agents and bugs fixed by maintainers (agents over-report local logic errors, maintainers focus more on input handling and public API defects), and agents are highly consistent with each other but differ from humans. Even if agents find real bugs, they may not be the ones maintainers care about—complementing the skeptical evidence for "harness update ≠ harness benefit." Its benchmark construction, matching protocol, and independent verification panel all meet hard standards, with public code and trajectories, making it a template for new benchmarks.
Retrospective harness self-evolution rarely reduces repeated work; only adaptations covering encountered instances succeed, while incomplete ones can add work.
Editor's noteThe first to perform per-proposal qualitative coding and controlled transfer evaluation of single adaptations for retrospective harness self-evolution mechanisms like /retro, distinguishing experience-driven from prior-driven triggers, and finding that lint-type proposals are often unrelated to session events. The core increment is proving that the coverage of an adaptation (whether it covers the specific instances encountered in subsequent sessions) determines whether it reduces duplicate work, and when it does not cover, it may instead increase irrelevant work. The transfer evaluation protocol (encounter vs coverage separation, control tasks, consulted records) is directly reusable for any harness self-evolution or memory system evaluation. Limitations: single agent, single /retro version, small sample size.
Quality-gated evaluation of coding agents reveals that editing reusable history can increase uncached prefill by 608% despite cutting prompt tokens, so physical work must be measured, not just token counts.
Editor's noteThe first to treat "prefix cache hit rate" as a first-class variable in physical cost accounting for context policies, proposing a quality-gated physical resource accounting protocol that decomposes execution prefill into logical tokens minus cached reuse tokens, and proving that "deleting history" can reverse token savings into an explosion of uncached prefill (Mask-5 saves 33.3% of prompt but increases uncached prefill by 608.5%). The causal attribution of "token savings ≠ physical cost savings" and the break-even condition (ρ≈0.55) are directly usable analytical tools. Limitations: the main benchmark is a localization task, single device (DGX Spark), and no held-out control.
