# Harness Handbook: Making Evolving Agent Harnesses Readable,Navigable, and Editable > Harness Handbook bridges behavior and code, improving plan quality by up to 18.9 percentage points while reducing token usage by 12.7%. - **Source:** [arXiv](https://arxiv.org/abs/2607.13285) - **Published:** 2026-07-17 - **Permalink:** https://picx.dev/p/G9H8Ma - **Whiteboard:** https://picx.dev/p/G9H8Ma/image ## Summary ## Summary (Overview) - Identifies **behavior localization**—finding all code locations that implement a given behavior—as a central bottleneck in evolving production-scale agent harnesses. - Introduces **Harness Handbook**, a behavior-centric representation that reorganizes repository knowledge around runtime behavior and links each behavior directly to its source code. - Proposes **Behavior-Guided Progressive Disclosure (BGPD)**, a coarse-to-fine navigation workflow that guides coding agents from high-level behavior descriptions to relevant implementation details. - Provides an automated construction pipeline using static program analysis and LLM-assisted behavioral structuring, with automatic resynchronization after code changes. - Evaluation on two open-source harnesses (Terminus-2, Codex) shows Handbook-Assisted planning improves plan quality (win rate +10.0 pp on Codex, +18.9 pp on Terminus-2) while reducing planner token usage (−12.7% and −8.6% respectively). --- ## Introduction and Theoretical Foundation Modern AI agents depend not only on their foundation model but also on the **harness**—the software layer that constructs prompts, manages state, invokes tools, and coordinates execution. As models, APIs, environments, and application requirements evolve, the harness must be continually adapted. This process, called **harness evolution**, requires adding capabilities, modifying existing behaviors, or refining workflows. A key difficulty is that modification requests describe *what* the system should do, but repositories are organized by files, functions, and modules. Before any edit can be planned, the developer or coding agent must locate **all** code locations that implement the target behavior—a task the paper terms **behavior localization**. In production-scale harnesses, a single behavior may be scattered across many files, functions, execution stages, and state transitions, making this mapping costly and error-prone. Existing approaches (code search, repository maps, long-context editing) help inspect code but remain implementation-centric; they do not explicitly connect behavioral descriptions to distributed implementation sites. The paper addresses this gap by introducing an **operational behavior representation** that bridges the gap between “what the system should do” and “where that behavior is implemented.” --- ## Methodology ### 1. Harness Handbook Representation The Handbook is a three-level document tree **D** plus a complementary **state-register view Z**: - **L1 (System Overview)**: architecture, execution model, major stages, global data flow. - **L2 (Component Overview)**: responsibilities, inputs/outputs, dependencies, local state of each stage. - **L3 (Unit Deep Dive)**: source-grounded implementation entries (functions or files) linked to specific code locations. - **State-Register View Z**: records state relationships that cross stage boundaries. Two design rules ensure utility: **progressive disclosure** (readers move from L1 to L3 only when needed) and **behavior–implementation alignment** (every L3 locator is validated against the current repository). ### 2. Construction Pipeline Construction proceeds in three phases: - **Phase I – Static Fact Extraction**: Language-specific adapters parse the repository to extract functions, source locations, signatures, call edges, and external boundaries. The resulting program graph **G** keeps only internal calls; unresolved calls are logged. This phase is deterministic (no LLM calls). - **Phase II – Behavioral Organization**: Source units are organized into an execution-stage skeleton **S** using one of two leaf modes: - **Function-as-leaf**: A seed skeleton is refined through LLM-assisted proposal-review of function-to-stage assignments. - **File-as-leaf**: Files are summarized and combined with the program graph to infer the stage skeleton; an optional proposal-review loop refines it. - **Phase III – Hierarchical Synthesis and Packaging**: The skeleton and source organization are converted into the L1–L3 document tree and state-register view. Each L3 entry is linked to a statically identified source location and validated against the repository. ### 3. Modification Workflow with BGPD Given a request **q**, the workflow proceeds in four steps: 1. **Behavior Localization with BGPD**: - Use L1/L2 to identify relevant execution stages. - Follow **Z** to include stages coupled through shared state. - Select relevant L3 entries and retrieve source locators. - Expand candidate set using the call graph (function or file level). - Verify candidates against the current repository to produce evidence **b̂E_q**. 2. **Edit Planning**: The planner converts evidence into an edit plan **P** with action declarations **Γ** (modify/add/remove). 3. **Execution**: The executor applies **P** to the repository, producing diff **Δ**. 4. **Resynchronization**: For non-empty **Δ**, the handbook is updated incrementally (reparse changed source, match units via fingerprints/content hashes, refresh only affected entries). Unparseable content is frozen rather than guessed. --- ## Empirical Validation / Results ### Experimental Setup - **Benchmarks**: Two open-source harnesses—Terminus-2 (function-as-leaf) and Codex (file-as-leaf). Each contributes 30 modification requests grouped by type: Query (Q), Cross-file (CF), Search-Hostile (SH), and labeled Easy/Medium/Hard by localization difficulty. - **Planner**: DeepSeek-V4-Pro with NexAU framework; two arms: Baseline (direct repository exploration) vs. Handbook-Assisted (BGPD navigation). - **Evaluation**: Three independent judges (GPT-5.5, Opus 4.8, DeepSeek-V4-Pro) score plans on three dimensions using a weighted score: $$ S = 0.5 \, S_{\text{Loc}} + 0.25 \, S_{\text{Scope}} + 0.25 \, S_{\text{Reason}} $$ Win declared if score difference ≥ 3 points. Additionally, localization accuracy is measured against reference plans from Opus 4.8 and GPT-5.5 (Recall, Precision, F1, Wrong). ### Key Results #### RQ1: Better Plans at Lower Token Cost | Harness | Baseline Win Rate | Handbook Win Rate | Gain (pp) | Baseline Tokens/Req | Handbook Tokens/Req | Reduction | |---------|------------------|-------------------|-----------|---------------------|---------------------|-----------| | Codex | 28.3% | 38.3% | +10.0 | 0.102M | 0.089M | –12.7% | | Terminus-2 | 26.7% | 45.6% | +18.9 | 0.058M | 0.053M | –8.6% | - Gains consistent across all three judges (e.g., +10.0 pp for every judge on Codex; +13.3 to +26.7 pp on Terminus-2). - Decomposition by dimension: Localization (+12.2 pp on Terminus-2), Scope Control (+6.7 pp), Reasoning (+4.5 pp). #### RQ2: Weaker Planner Matches Stronger Models **Table 1: Reference-plan localization metrics (%).** Predicted implementation sites compared against independent reference plans from Opus 4.8 and GPT-5.5 at file and symbol granularity. All 24 Recall/Precision/F1 comparisons favor Handbook-Assisted. | Harness | Level | Metric | Opus 4.8 Baseline | Opus 4.8 Handbook | Gap | GPT-5.5 Baseline | GPT-5.5 Handbook | Gap | |----------|--------|---------|-------------------|-------------------|------|------------------|------------------|-------| | Codex | File | Recall | 55.4 | 69.7 | +14.3| 49.4 | 49.8 | +0.4 | | | | Precision | 46.1 | 60.2 | +14.1| 53.0 | 62.2 | +9.2 | | | | F1 | 46.6 | 61.8 | +15.2| 47.3 | 52.3 | +5.0 | | | | Wrong ↓ | 37.0 | 14.8 | –22.2| 21.4 | 21.4 | 0.0 | | | Symbol | Recall | 47.1 | 65.4 | +18.3| 46.4 | 49.1 | +2.7 | | | | Precision | 38.0 | 55.3 | +17.3| 48.4 | 60.4 | +12.0 | | | | F1 | 38.3 | 57.1 | +18.8| 43.8 | 51.2 | +7.4 | | | | Wrong ↓ | 44.4 | 18.5 | –25.9| 28.6 | 21.4 | –7.2 | | Terminus-2 | File | Recall | 74.7 | 83.9 | +9.2 | 76.1 | 87.5 | +11.4 | | | | Precision | 74.1 | 86.2 | +12.1| 78.3 | 93.3 | +15.0 | | | | F1 | 74.1 | 84.7 | +10.6| 76.5 | 89.3 | +12.8 | | | | Wrong ↓ | 24.1 | 13.8 | –10.3| 20.0 | 6.7 | –13.3 | | | Symbol | Recall | 64.7 | 76.3 | +11.6| 73.0 | 87.5 | +14.5 | | | | Precision | 65.1 | 78.4 | +13.3| 73.9 | 93.3 | +19.4 | | | | F1 | 64.8 | 77.1 | +12.3| 73.0 | 89.3 | +16.3 | | | | Wrong ↓ | 24.1 | 13.8 | –10.3| 20.0 | 6.7 | –13.3 | - F1 gains range from 5.0 pp to 18.8 pp. - **Wrong** (zero overlap with reference) decreases by up to 25.9 pp on Codex symbol-level against Opus 4.8. #### RQ3: Consistent Gains Across Request Types and Difficulty | Harness | Request Type | Baseline Win Rate | Handbook Win Rate | Gain (pp) | |------------|--------------|-------------------|-------------------|-----------| | Codex | Query (Q) | – | – | +26.7 | | | Cross-file (CF) | – | – | +16.3 | | | Search-Hostile (SH) | – | – | +16.7 | | Terminus-2 | Query (Q) | – | – | +3.3 | | | Cross-file (CF) | – | – | +20.0 | | | Search-Hostile (SH) | – | – | +33.3 | All six harness-by-type comparisons favor Handbook-Assisted. Gains also hold across difficulty levels (Easy, Medium, Hard), ranging from +3.7 pp to +33.3 pp. --- ## Theoretical and Practical Implications - **Theoretical**: The work formalizes **behavior localization** as a prerequisite for harness evolution and demonstrates that an explicit behavior-centric representation can significantly improve the accuracy and efficiency of modification planning. - **Practical**: Harness Handbook reduces the cognitive burden on developers and coding agents by organizing knowledge around runtime behavior rather than code structure. It lowers planning token costs, enables weaker models to match stronger ones in localization, and reduces complete localization failures. - **Broader applications**: Beyond modification, the Handbook can support behavior auditing, regression-impact analysis, and autonomous harness self-evolution by serving as a shared behavioral memory. --- ## Conclusion This work identifies **behavior localization** as a central challenge in evolving production-scale agent harnesses and introduces **Harness Handbook**, a behavior-centric representation that links behavioral descriptions to their distributed source implementations. The Handbook is constructed automatically via static analysis and LLM-assisted structuring, and supports modification through **Behavior-Guided Progressive Disclosure (BGPD)** and automatic resynchronization. Experiments on two open-source harnesses show that Handbook-Assisted planning: - Improves plan quality while reducing token usage (win rate +10.0 pp to +18.9 pp, tokens –8.6% to –12.7%). - Enables a weaker planner to match the localization performance of substantially stronger models (all 24 Recall/Precision/F1 metrics improve, Wrong decreases by up to 25.9 pp). - Delivers consistent gains across request types (Query, Cross-file, Search-Hostile) and localization difficulty levels. These results indicate that making the relationship between behavior and implementation explicit improves both modification planning and localization efficiency without requiring a larger planning-token budget. Future work will apply the Handbook to **harness self-evolving**, enabling an agent to autonomously close the loop of localization, planning, execution, and resynchronization. --- _Markdown view of https://picx.dev/p/G9H8Ma, served by PicX — AI-generated visual whiteboard summaries of research papers._