# Elicitation or Expansion? Demystifying Gains from Agent Harness Optimization

> Harness optimization mostly elicits existing capabilities rather than expanding solvable task range, with pass@1 gains reversing at higher attempt budgets.

- **Source:** [arXiv](https://arxiv.org/abs/2610.27889)
- **Published:** 2026-10-24
- **Permalink:** https://picx.dev/p/PuI0BW
- **Whiteboard:** https://picx.dev/p/PuI0BW/image

## Summary

# Summary of "Elicitation or Expansion? Demystifying Gains from Agent Harness Optimization"

## Summary (Overview)

- This paper systematically analyzes whether gains from harness optimization (optimizing prompts, context, tool use, and execution workflows without updating model weights) stem from **elicitation** (making success more frequent on already-solvable tasks) or **expansion** (broadening the range of solvable tasks).
- Using **pass@k analysis** (borrowed from RL literature), the authors compare optimized harnesses against their pre-optimization counterparts across static prediction tasks (text classification) and dynamic tasks (multi-hop QA, agentic tasks).
- Key finding: **pass@1 advantages of optimized harnesses over stronger interventions (larger models, higher reasoning effort) can reverse at larger attempt budgets** — an optimized small model may beat a larger model at pass@1 but fall behind at pass@64 or pass@128.
- **Aggregate gains conceal shortfalls on hard tasks**: even when optimized harnesses match or exceed stronger references overall, they consistently fail to surpass them on hard task subsets.
- The paper demonstrates that harness optimization primarily achieves **elicitation** (more reliable success on tasks already within reach) rather than **expansion** (broader task coverage), especially on static tasks.

---

## Introduction and Theoretical Foundation

### Background and Motivation

The design of an agent's **harness** — governing prompting, context management, tool use, and execution logic — significantly influences language model agent performance *without* updating model parameters. Recent harness optimization methods have achieved substantial gains across diverse benchmarks, but aggregate success rates alone cannot reveal *what* improved:

- **Elicitation**: The optimized harness makes success more frequent on tasks already solvable by the baseline.
- **Expansion**: The optimized harness enables solving tasks that were previously unreachable.

This distinction is critical for **recursive self-improvement** and **evolutionary search**, where even rare successes on previously unreachable tasks can provide a foundation for further learning. The paper draws on a parallel debate in reinforcement learning, where comparisons between RL-trained models and their pre-RL counterparts show that pass@1 improvements can diminish or reverse at larger sampling budgets.

### Theoretical Foundation

The formal optimization objective is defined as:

$$
\max_{H} J(\mathcal{M}, H), \qquad J(\mathcal{M}, H) = \mathbb{E}_{x \sim \mathcal{X}, \tau \sim \pi_{\mathcal{M}, H}(\cdot | x)}[r(\tau, x)]
$$

where:
- $\mathcal{M}$ = solver model (fixed)
- $H$ = harness (optimized)
- $x$ = task drawn from distribution $\mathcal{X}$
- $\tau \sim \pi_{\mathcal{M}, H}(\cdot | x)$ = complete execution trajectory
- $r(\tau, x)$ = task-specific scoring function

The paper uses **pass@k** (Chen et al., 2021) to distinguish:
- **pass@1**: mean accuracy (expected number of successful executions)
- **pass@k**: coverage (expected fraction of tasks solved at least once in $k$ complete executions)

---

## Methodology

### Harness Optimization Framework

The authors use **Meta-Harness** (Lee et al., 2026b), where a coding-agent "proposer" iteratively:
1. Inspects previous harness code, evaluation scores, and execution traces
2. Diagnoses failures
3. Proposes revised harnesses

The analysis compares:
- **Baseline system**: $(\mathcal{M}, H_0)$ — initial harness
- **Optimized system**: $(\mathcal{M}, H^\star)$ — harness selected based on validation performance
- **Reference system**: $(\mathcal{M}_{\text{ref}}, H_0)$ — larger model or higher reasoning effort with base harness

### Evaluation Setup

| Benchmark | Task Type | Solver Model | Reference Model | Attempts |
|-----------|-----------|--------------|-----------------|----------|
| Symptom2Disease | Static text classification | GPT-OSS-20B | GPT-OSS-120B | k=64 |
| LawBench | Static text classification | GPT-OSS-20B | GPT-OSS-120B | k=64 |
| USPTO | Static text classification | GPT-OSS-20B | GPT-OSS-120B | k=64 |
| HotpotQA | Dynamic multi-hop QA | Qwen3-8B | Qwen3-32B | k=128 |
| AppWorld | Dynamic agentic task | GPT-OSS-120B (medium effort) | GPT-OSS-120B (high effort) | k=128 |

**OOD Transfer**: The in-distribution (ID) selected harness is transferred to FiNER, GoEmotions, and Banking77.

---

## Empirical Validation / Results

### 5.1 Gains Over Stronger References

**Static tasks**: The optimized GPT-OSS-20B harness's pass@1 advantage over the GPT-OSS-120B reference **reverses at larger k** across all three benchmarks:

- **LawBench**: 16.05 pp pass@1 advantage reverses at $k=11$; at pass@64, optimized harness reaches 56% vs. reference's 60%.
- **Symptom2Disease**: Pass@1 nearly matches reference, but slight decrease in pass@64 coverage.
- **USPTO**: Pass@1 falls below reference; 68 of 100 tasks still yield no success at pass@64.

**Dynamic tasks**: Optimized harnesses track stronger references more closely:
- **HotpotQA**: Both optimized 8B harnesses exceed 32B reference at pass@1, but only one remains above at pass@128.
- **AppWorld**: Both optimized medium-effort harnesses fall below high-effort reference at endpoints.

**Key mechanism**: Task-level success distributions (Figure 4) reveal that optimization primarily **increases the frequency of existing successes** rather than broadening coverage. On LawBench, of 53 tasks solved by baseline within 64 attempts, the optimized harness retains 49 and loses 4, while newly solving only 7 others — retained tasks account for 21.02 pp of the 25.14 pp pass@1 gain.

**Dynamic advantage**: Environment interaction enables expansion. Example from HotpotQA: the base harness restates the question as its second retrieval query, leaving the "George Litto" abstract unreachable in all 128 rollouts. The optimized harness queries the newly resolved bridge entity, reaching the relevant abstract in 125/128 rollouts and answering correctly 122 times.

### 5.2 Gains by Task Difficulty

**Hard task shortfall**: None of the selected harnesses exceeds its stronger reference on hard subsets at any attempt budget:

- **HotpotQA hard**: Optimized harnesses reach 70.73–73.17% at pass@128 vs. baseline's 70.73% and reference's 78.05%.
- **AppWorld hard**: Optimized harnesses reach 88.89–92.06% at pass@128 vs. baseline's 82.54%, at best matching reference's 92.06%.

**All candidates share the shortfall**: Across all 40 candidates from one optimization run, *every* curve remains below the 32B reference on hard questions throughout all budgets.

**Hard-focused optimization is insufficient**: Re-optimizing with a hard-focused task mixture improves hard pass@1 from 56.88% to 59.28% (nearly matching the 32B reference's 59.51%), but pass@128 coverage remains at 73.17% — unchanged and 4.88 pp below the reference.

### 5.3 OOD Transfer

| Dataset | Pass@1 Gain | Pass@64 Gap vs. Baseline |
|---------|-------------|--------------------------|
| FiNER | Small positive | −6.00 pp |
| GoEmotions | Small positive | −8.00 pp |
| Banking77 | Negative | −11.33 pp |

**Mechanism**: Code inspection reveals the optimized harness routes prompts based on label count and average length. Its catalog template permits semicolon-separated labels (compatible with LawBench's multi-charge format) but is incorrectly applied to Banking77, which requires exactly one intent — producing outputs like "order physical card;country support" instead of "country support".

---

## Theoretical and Practical Implications

### Theoretical Implications

1. **Harness optimization primarily elicits, not expands**: Gains concentrate on tasks with existing baseline successes — analogous to the "Matthew effect" in RL (Noukhovitch et al., 2026) where gains favor tasks with greater initial ability.

2. **Static vs. dynamic task asymmetry**: On static tasks, optimization reshapes how the solver uses fixed input, limiting expansion potential. On dynamic tasks, optimization shapes actions and subsequent observations, enabling access to previously unreachable external evidence.

3. **Pass@1 ≠ coverage**: A substantial single-attempt advantage does not translate into broader coverage under repeated attempts. Evaluation must consider the intended attempt budget.

### Practical Implications

1. **Diagnostic framework**: The pass@k curve analysis with stronger references provides a practical toolkit for evaluating whether a harness suits a given attempt budget and task group.

2. **Hard task evaluation**: When hard task coverage matters, evaluate the optimized harness on that subset at the intended budget — aggregate gains can be misleading.

3. **OOD deployment validation**: Validate transferred harnesses against target-domain baselines and use code inspection to identify domain-dependent assumptions.

4. **Selection criteria**: For recursive self-improvement, select harnesses that reward validated successes on previously uncovered hard tasks and retain candidates with complementary successes.

---

## Conclusion

The paper provides a diagnostic lens for understanding harness optimization gains by combining:
- **pass@k curves** to distinguish reliability from coverage
- **Stronger reference systems** (larger models, higher effort) as empirical baselines
- **Task difficulty stratification** to reveal concealed shortfalls

**Main takeaways**:
1. Harness optimization can yield substantial pass@1 gains without matching stronger references at larger $k$, especially on static tasks.
2. Dynamic tasks offer greater expansion opportunities through improved environment interaction.
3. Aggregate gains conceal consistent shortfalls on hard tasks that persist even with targeted optimization.
4. Transferred harnesses may underperform baselines at larger budgets despite small pass@1 gains.

**Future work**: Achieving consistent capability expansion on hard tasks remains an open challenge, motivating search and selection methods that reward validated successes on previously uncovered hard tasks and retain candidates with complementary successes. The authors note their results are bounded by the evaluated benchmarks, solver families, and optimization runs examined.

---

_Markdown view of https://picx.dev/p/PuI0BW, served by PicX — AI-generated visual whiteboard summaries of research papers._