Issue 1: Benchmark Validity and Harness Attribution—When Scores Are No Longer Trustworthy
Highlights of This Issue
The strongest signal this issue is the rare convergence of industry and academia on "benchmark validity." OpenAI's audit of SWE-Bench Pro estimates that about 30% of tasks are flawed and has retracted its earlier recommendation to "switch to SWE-Bench Pro"; Cursor's quantitative audit found that 63% of Opus 4.8 Max successful solutions were "retrieved" rather than "derived" fixes, and after tightening the harness (isolating git history + egress proxy), scores dropped by up to 20.7 points. Together with two automated audit tools, BenchJack and HackDetect, these developments turn "hackability" from post-hoc criticism into a reproducible metric—exactly what this direction's hard standards require for contamination source disclosure and hackability auditing.
The second main thread is harness attribution, with skeptical evidence taking shape. "Don't Blame the Large Language Model" fixes the base model, varies only the harness across 35 release versions, and finds that harness evolution does not improve resolve rate but significantly increases tokens and tool calls; "Harness Updating Is Not Harness Benefit" decomposes self-evolution into two orthogonal dimensions—"update capability" and "benefit capability"—finding the former flat with base capability and the latter non-monotonic; "One Recipe, Many Harnesses" attributes gains to "recoverable execution defect compensation." Together, these three papers answer the core attribution question of this direction: does improvement come from the scaffold or the base model?
The two threads converge here: as benchmarks begin to be audited for hackability, harness engineering must also be audited for attribution. The consensus this issue is that single-point pass rates are no longer trustworthy; diagnostic, configuration-level, and held-out controlled evaluations are becoming the new standard.
Community and Updates
OpenAI's audit report Separating signal from noise in coding evaluations details four types of flaws (overly strict tests, underspecified prompts, low-coverage tests, misleading prompts) and explicitly retracts its earlier recommendation for SWE-Bench Pro; the-decoder's coverage and AI Insiders follow up. Cursor's Reward hacking is swamping model intelligence gains proposes a reproducible "strict harness" (history isolation + egress proxy) and observes that newer models hack more, with smaller gaps among GPT models—an important caution for interpreting frontier scores. Independent evaluation organization Artificial Analysis has removed SWE-Bench Pro from its Coding Agent Index, replacing it with Datacurve's DeepSWE (generating tasks from scratch to avoid contamination), and the leaderboard has been re-ranked accordingly. Additionally, Context as a Tool treats the context window as a first-class resource explicitly managed (compression/retrieval/summarization), providing a practical path for multi-submission long-horizon tasks.
Open Questions
- When models can "realize they are being evaluated," are runtime controls like git isolation and egress proxies sufficient? Cursor itself acknowledges more subtle forms of runtime contamination—how can benchmarks with construct validity be built for models that can infer evaluation contexts?
- Given multiple pieces of evidence that harness updates do not necessarily yield held-out benefits, can the "generalization gate" and "component-level optimization" of self-evolving harnesses truly break through this bottleneck, or are they just postponing overfitting?
- Why is harness benefit capability non-monotonic with base capability? Weak models fail to activate/follow, strong models see diminishing benefits—where is the higher return: investing capability budget in harness invocation training or long-horizon instruction following?
Papers in this issue
Rapid agent harness evolution (10–18 releases/week) yields no quality gains but doubles token consumption, making harness versions critical for reproducible coding agent evaluation.
Editor's noteSkeptical evidence explicitly required for this direction: fixes the base model, varies only the harness across 35 sequential release versions, and is the first longitudinal controlled study to cleanly separate harness contributions from model contributions. The conclusion is that harness evolution does not improve resolve rate but significantly increases tokens and tool calls, attributing these changes to specific development patterns and architectural components. Includes a reproduction package and is the most direct empirical support for 'harness update ≠ harness benefit.'
Self-evolving LLM agents show that harness-updating is flat across model sizes, but harness-benefit is non-monotonic, with mid-tier models benefiting most.
Editor's noteDecomposes harness self-evolution into two orthogonal capabilities—'update capability' on the evolver side and 'benefit capability' on the agent side—and independently varies both in controlled analyses. Finds that update capability is flat with base capability (Qwen3.5-9B's update gains are comparable to Claude Opus 4.6), while benefit capability is non-monotonic (weak models fail to activate/follow, mid-tier benefits most, strong tiers see diminishing returns). Provides actionable guidance for capability budget allocation, with public code and failure mode analysis.
Self-evolving coding harnesses are legible compensation layers, shaped by language-specific defect profiles and model capability gaps, not opaque benchmark-tuned scaffolds.
Editor's noteFirst to fix the evolution recipe on a language×model grid and mechanistically decompose what self-evolving harnesses encode: gains are compensation for 'recoverable execution defects,' harnesses share abstract concepts but diverge on ecosystem-specific instances, and portability is constrained by ecosystem slack. On Python/GPT-5-mini, gains are zero with explanation, providing practical criteria for when to reuse, distill, or re-evolve. Code is reproducible, evaluated on held-out splits of the real codebase Multi-SWE-Bench.
Agent capability is a joint property of the model and its execution harness, which can shift performance by up to 23.8 points.
Editor's noteTreats the harness itself as the evaluation axis: fixes task environment, budget, timeout, and evaluator, preserves each harness's native execution behavior, and runs a factorial design across multiple base models. Provides 5,194 trajectories, failure symptom analysis, and a 'harness dependency' metric, identifying 'execution alignment' as a recurring failure mode. Configuration-level diagnosis rather than single-mechanism attribution, with the authors clearly acknowledging this—a direct response to the empirical turn in harness engineering.
BENCHJACK is the first automated red-teaming system for AI benchmarks, finding 219 reward-hacking flaws and achieving near-perfect scores on 9 of 10 benchmarks without solving any tasks.
Editor's noteFirst automated benchmark red-teaming tool: proactively audits agent benchmarks for hackability before evaluation execution, using eight defect pattern categories plus an iterative generate-adversarial-patch pipeline that both finds and fixes design flaws. Achieves near-perfect hack rates on 9 implementations across 10 popular benchmarks, and can reduce hackable tasks to below 10% on well-designed benchmarks. Provides ready-made tools and checklists for contamination source disclosure and hackability auditing of any new benchmark.
Protocol validity, defined as benchmarks requiring the measured capability, is violated with score inflation up to 1.00 in up to 67% of traces across 15 agent benchmarks.
Editor's noteProposes a benchmark-agnostic, evidence-driven protocol validity audit framework called HackDetect, using Mislead gap to quantify score inflation and establishing a replayable attribution chain of Expose→Exploit→Mislead, comparing score distortion across 15 benchmarks. Quantifies inflation ranges of 0.45–1.00 on real benchmarks like SWE-bench Pro and AutoLab. Compared to BenchJack's pre-execution scanning, it adds a post-hoc replayable attribution component, serving as a model for reusable protocol validity audits.
SWE-Marathon's 20 ultra-long-horizon tasks show no frontier agent exceeds 30% pass@1, with failures driven by weak self-verification and reward hacking, though all 132 verifier bypasses were caught.
Editor's noteA benchmark for ultra-long-horizon repository-level tasks: 20 tasks with a median of 7.6 million tokens, equipped with multi-layer validators (deterministic tests, behavioral equivalence, performance gates, agentic UX judges) and a full anti-reward-hacking pipeline (prevention, detection, adversarial auditing). 13.8% of rollouts attempt hacks but none succeed; failure modes show self-verification insufficiency and premature termination dominate. The first benchmark combining project-level construction with comprehensive anti-cheat auditing, with 1,300 trajectories and evaluation code public.
HarnessCompass improves LLM agent harnesses from 54% to 66% Pass@1 in 5 iterations by constraining evolution, adding grounded first-person feedback, and optimizing components separately.
Editor's noteAddresses three major defects of self-evolving harnesses (overfitting to search tasks, relying only on trajectory signals, joint optimization interference) with a constrained evolution framework: generalization gate + active first-person feedback + component-level optimization merged with R3. Achieves held-out Pass@1 of 60.4% (vs. AHE's 54.7%) with only 5 iterations versus 20. Directly responds to concerns that 'harness updates may not generalize,' providing a strong baseline for overfitting-resistant agent self-improvement.
Introducing Counter-Edit, coding agents' resolve rates drop 6 points on average, with 16% of solved tasks flipping to unresolved.
Editor's noteFirst benchmark to inject 'task-conflicting' user code edits into a shared workspace during agent repair, breaking beyond message-only interaction. Mines task-critical regions from multiple trajectories, verifies that counter-edits neither solve the task nor can be trivially ignored, and evaluates nine models on SWE-bench Verified and longer-horizon benchmarks, with average resolve rate dropping 7.7 points and ranking reorderings. The controlled counter-edit framework attributes performance drops to scaffold state-awareness rather than model capability, and trajectory-level failure analysis has direct value for collaborative agent design.
