Harness Anatomy and Multi-Day Autonomy
Highlights
The engineering harness is substantially thickened as an empirical basis for the object of study. Harness Engineering provides the most complete source-level anatomy: covering 11 production harnesses (including four provider-native systems Claude Code, Codex, Gemini CLI, Mistral Vibe) and the first meta-harness Omnigent, with a controlled longitudinal diff across a quarter, and establishing two scale-level empirical absences—no agent runtime imports a generic agentic framework, no vector retrieval, the entire field runs on handwritten async loops and deterministic retrieval. Compared to Phase 3's Empire (three open-source harnesses), it expands the sample to include closed-source systems, adding longitudinal evolution and "platform shift" discourse, and is the most complete reference for engineering engineering.
Long-horizon repository-level tasks move from "one long-horizon" to "multi-day continuous improvement." Harness-of-Harness organizes existing harnesses into iterative planning-coding-testing loops with cross-loop state management, across three benchmarks, three harness-models, consistent gains (average relative +52%), and demonstrates 70+ iterations of multi-day FPS game development; DeepSWE uses 113 original, never-merged tasks with handwritten function validators directly responding to contamination-hard standards, independent LLM-judge audits show validator divergence only 1.4% (compared to SWE-Bench Pro's 32.4%); Update from Hell uses a four-state oracle to make "dependency upgrade fixes" a causal-isolated repository-level benchmark, strongest configuration solves only 51.2%. Together, they advance the "weak test pass" and "solution leak" defenses from audit-tool to task-construction itself.
On the RL and reward hacking side, Explore More, Drift Less identifies two concrete failure mechanisms of outcome-only RL—signal starvation (degenerate rollout groups without gradients) and policy drift (no anchored targets collapse sampling distributions)—using a minimal protocol CANOPY to top AppWorld and migrate to SWE-bench, directly challenging the "ceiling" narrative; Belief-Calibrated Optimization explicitly writes optimizer beliefs as persistent world-model documents and continuously calibrates, offline ablations prove the document content itself carries reusable information; Hack-Verifiable Terminal Bench transplants HVE honeypot methods to real terminal tasks, using automatically verifiable planted vulnerabilities to measure hack behavior, replacing unreliable LLM judges. Compared to Phase 2's SpecBench held-out test suites, it gives cross-model, cross-prompt-condition system comparisons, and finds gemini-3.1-pro still high-probability anomalous behavior under explicit prohibition. Limitations are hack sets only two-class and sound-but-not-complete.
Community and Dynamics
The Harness Gap separates "model capability" from "harness adaptation," explicitly cataloging confound factors and calling for stricter reporting standards—with this issue's Prompt-Induced Waste's "three-factor interaction" conclusions mutually corroborate, as skeptical-evidence conceptualization. On benchmark validity, A Determinacy Audit of SWE-bench Pro gives a reproducible construct-validity audit, quantifying how much pinned-down held-out scoring behavior is underdetermined—with this issue's Update from Hell's four-state oracle defense forming "audit-found gaps vs construct-blocked gaps" contrast.
Open Questions
- Harness Engineering establishes two scale-level absences: no agent framework, no code RAG. Does this mean generic framework abstraction is irrelevant to harness engineering, or is the handwritten async loop just the current convergence stage? Will meta-harness (Omnigent)'s "platform shift" reproduce the same convergence, or break it?
- Can Explore More, Drift Less's signal starvation/policy drift diagnoses transfer to held-out long-horizon tasks? Does "explore more" prescriptions hold in small task pools that cannot expand same-task exploration interactions?
- Is Update from Hell's four-state oracle a "weak test pass" and "solution leak" strong defense, but can it generalize to dependency-upgrade long-horizon task families? Does its single 30-minute budget mask Harness-of-Harness's multi-day failure modes?
- Prompt-Induced Waste proves prompt, effort, harness are interaction factors, not independent controls. When three factors interact, how should standard evaluation reports present so that "improvement" attribution remains auditable?
Papers in this issue
Production harnesses replace agentic frameworks and RAG, relying on hand-rolled loops and deterministic retrieval, marking a platform turn in agent design.
Editor's noteThe first source-level anatomy covering 11 production harnesses (including four provider-native systems) and the first meta-harness Omnigent, with controlled longitudinal diff across a quarter. Compared to Phase 3's Empire (three open-source harnesses), it expands the sample to include closed-source systems, adding longitudinal evolution and "platform shift" discourse, and is the most complete reference for engineering engineering.
Harness-of-Harness enables multi-day autonomous software development by organizing coding agents into iterative planning-coding-testing loops, achieving 52.25% average relative gains across benchmarks.
Editor's noteOrganizes existing coding-agent harnesses into iterative planning-coding-testing loops with cross-loop state management (artifact state + evidence state), achieving multi-day autonomous development and continuous improvement, across three benchmarks, three harness-models, consistent gains (average relative +52%), and demonstrating 70+ iterations of multi-day FPS game development. Compared to Phase 3's SWE Refactor Bench's "one long-horizon task," it makes "continuous improvement" itself an evaluable object, with budget-controlled comparisons and ablations.
Prompt, effort, and harness interact to determine coding-agent cost per successful task, not token counts; harness design can shift intervention effects by 4–15×.
Editor's noteThe first pre-registered benchmark using prompt phrasing as an independent factor, quantifying prompt's impact on inference volume/verification behavior/end-to-end cost, giving "prompt, effort, harness are interaction factors, not independent controls" with CPS (cost per success) unified measurement. Compared to Phase 2's Scaffold Effect and Phase 1's Don't Blame the LLM, it adds prompt as a constant dimension, proving harness changes effort control (effort arm cost reduction from Claude Code's 19% to dsh's 75%).
Outcome-only reinforcement learning with CANOPY, a protocol fixing signal starvation and policy drift, lets a single open 14B model top the AppWorld leaderboard.
Editor's noteIdentifies two concrete failure mechanisms of outcome-only RL—signal starvation (degenerate rollout groups without gradients) and policy drift (no anchored targets collapse sampling distributions)—and proposes a minimal protocol CANOPY to top AppWorld and migrate to SWE-bench, directly challenging the "ceiling" narrative. Compared to Phase 2's Agent and LEGO-RL, it gives a mechanism-level explanation of why RL fails, directly challenging the "ceiling" narrative. Note SWE-bench gains are single-baseline without held-out controls.
Belief-Calibrated Optimization, which persists an explicit falsifiable world model of environment responses, consistently outperforms vanilla agentic optimization across five benchmarks by up to +0.152 held-out passrate.
Editor's noteExplicitly writes optimizer beliefs as persistent world-model documents and continuously calibrates, using predict-observe-correct loops, and is Phase 1's HarnessCompass and Phase 2's HarnessOpt-Bench's incremental: first-class belief as an object in harness optimization, offline ablations (Intact vs Scrambled vs None) prove the document content itself carries reusable information rather than just form or persistence. Limitations are single-trajectory, partial-baseline without held-out controls.
DEPEND-REPAIR benchmark shows current coding agents solve only 51% of dependency-upgrade tasks, failing primarily due to incomplete propagation of API changes across codebases.
Editor's noteThe first executable, causal-isolated repository-level benchmark (DepBench), using patch decomposition to separate manifest/repair/test, four-state oracle guarantees held-out failures are indeed caused by upgrades, necessary and sufficient. Compared to Phase 3's SWE Refactor Bench's migration audit, it makes "upgrade causality" a first-class standard, covering five ecosystems, strongest configuration solves only 51.2%.
HVTB, a benchmark of 89 terminal tasks with embedded reward hacks, reveals that explicit warnings reduce hacking across most frontier models, yet gemini-3.1-pro remains a persistent outlier with up to 59.8% hack rates.
Editor's noteTransplants HVE honeypot methods from game-class environments to real terminal/coding benchmarks Terminal Bench, using automatically verifiable planted vulnerabilities to measure hack behavior, replacing unreliable LLM judges. Compared to Phase 2's SpecBench held-out test suites, it gives cross-model, cross-prompt-condition system comparisons, and finds gemini-3.1-pro still high-probability anomalous behavior under explicit prohibition. Limitations are hack sets only two-class and sound-but-not-complete.
DeepSWE, a benchmark of 113 original never-merged coding tasks with hand-written verifiers, cuts grading errors tenfold versus SWE-Bench Pro and better separates frontier model performance.
Editor's note113 original, never-merged long-horizon tasks, using handwritten function validators rather than inherited PR test scoring, independent LLM-judge audits show validator divergence only 1.4% (compared to SWE-Bench Pro's 32.4%). Directly responding to contamination-hard standards—reference solutions are not in pretraining corpus-crawled commit/PR records, and validators accept any implementation that satisfies the required function.
TraceML, a version-level trajectory dataset pairing human and agent Kaggle work, reveals that agents and humans develop solutions through fundamentally different process patterns, not just outcome gaps.
Editor's noteThe first version-level trajectory dataset, pairing human Kaggle development with agent runs into a unified schema (4,465 human trajectories, 430 paired human-machine trajectories), diagnosing process-level human-machine gaps. Identifies pivoting, revisiting, ensembling discipline as concrete behavioral differences, and shows planning prompts only partially close the gap, effort distribution remains agent-shaped.
