# What Does a Harness Repair? A Preregistered Study of Visibility, Baseline Adequacy and Evaluation Defects

> Preregistered harness-search studies show most accuracy gains are visibility artifacts, and searched harnesses failed to beat trivial constant-label baselines despite large nominal improvements.

- **Source:** [arXiv](https://arxiv.org/abs/2610.05533)
- **Published:** 2026-10-31
- **Permalink:** https://picx.dev/p/kk7VFN
- **Whiteboard:** https://picx.dev/p/kk7VFN/image

## Summary

## Summary (Overview)

- This paper presents two preregistered studies examining **where harness search gains come from**—specifically whether improvements come from genuine quality gains or from making previously unparseable answers visible, weak baselines, or evaluation defects.

- The authors test four hypotheses: (H1) whether thinking-off gains over capped thinking come from visibility, (H2) whether simple baselines are adequate, (H3) whether budget forcing reduces truncation, and (H4) whether known evaluation defects replicate through the same pipeline.

- Across 12 model-benchmark cells (three models of 1.7B–8B parameters plus Qwen3-14B), thinking-off raised accuracy over capped thinking in 7 of 12 cells, **mostly through questions the capped setting left without readable answers** (visibility gains).

- At 14B scale, two GEPA-selected harnesses beat the thinking-off baseline by 30–34 points on longevity letter tasks, but **were not detectably above the strongest constant-label baseline**, answering "Yes" in 95–99% of parsed mortality answers.

- The study reports all **65 primary endpoints** with deviations and addenda, finding that prospective replication confirmed defect distortions in 9 of 20 evaluable pairs (7 of which were fixed by design identities), and that conclusions held across GPUs despite most token sequences changing.

## Introduction and Theoretical Foundation

### Background

A **harness** is defined as "everything around a frozen model that changes what it is scored on: the prompts, reasoning switches, token budgets, retry rules and parsers." **Harness search** proposes changes and keeps them if scores rise—a mechanism central to self-improving AI systems.

### Three Open Questions

The paper identifies three questions that a replicated gain leaves open:

1. **Visibility**: Did answers improve, or did answers that the parent never produced in readable form (due to token limits or non-commitment) become visible to the parser?
2. **Baseline adequacy**: Would a simple setting the search never tried do as well? Does the model beat trivial baselines at all?
3. **Evaluation validity**: Are inputs, gold labels, splits, and the served model what they claim to be?

### Prior Work and Gaps

The authors note that most prior work addresses these questions *after the fact*: harness gains vanish against matched-budget baselines, gains can be completion rather than correctness, evaluation pipelines carry defects, and no earlier study fixed in advance how much of a harness gain is visibility, tested benchmark authors' protocols against searched harnesses with a margin, or injected known defects to test replication. The search was capped at 200 web searches covering 2024-01 to 2026-10-03.

### The Case Study Origin

This work follows a case study of harness search around a frozen longevity language model where:
- Search scored candidates through a caller that sent each benchmark row's reference answer to the model as an earlier assistant turn
- The rescue selected replicated through the same defective caller
- Trivial baselines, input audits, and per-call prompt-token reconciliation detected the defect
- After correction, the starting configuration parsed **none** of its 556 prospective rows

### Preregistration Approach

Two preregistered studies were frozen (first on 30 September 2026, second on 7 October 2026) with:
- Pre-specified hypotheses, endpoints, and verdict rules for all 65 primary endpoints
- Randomization into search, replication, and test partitions (40/20/40)
- Adaptive GEPA arms for harness search
- Six evaluation defects injected one at a time

## Methodology

### Models and Stack

| Component | Specification |
|-----------|---------------|
| First study models | Qwen3-1.7B-Longevity, SmolLM3-3B, Intern-S1-mini |
| Second study models | Qwen3-14B (confirmatory); base and tuned Qwen3-1.7B (descriptive) |
| Serving engine | vLLM 0.30.0, batch-invariant mode, zero temperature, client concurrency 32 |
| GPUs | RTX 5090 (first study), RTX PRO 6000 (second study) |

### Configurations

**Table 1** defines the six configurations (with "tokens" being max_tokens):

| Configuration | Request |
|---|---|
| THINK-64, THINK-512, THINK-1536 | thinking on; 64, 512, or 1,536 tokens total |
| AUTHORS-1500 | thinking off; 1,500 tokens (benchmark authors' protocol) |
| BUDGET-512-1500 | thinking on; vLLM ends thinking after 512 tokens; up to 1,500 answer tokens (2,012 total) |
| RESCUE-1500 | THINK-512; if truncated, AUTHORS-1500 on same row |
| GEPA harnesses | 4 runs per study (clean/broken start × 2 seeds), 6,000 metric calls on 300 search rows |

### H1 Visibility Decomposition

For baseline P and comparison R, paired on n rows:

- **V**: rows only R parses ($\bar{u}_{P,i} u_{R,i}$)
- **B**: rows both parse ($u_{P,i} u_{R,i}$)
- **L**: rows only P parses ($u_{P,i} \bar{u}_{R,i}$)

With $\Delta = \sum_i (c_{R,i} - c_{P,i})/n$, the identity:

$$\Delta = \Delta_V + \Delta_B + \Delta_L$$

is exact, where:

$$\Delta_V = \sum_i V_i c_{R,i}/n, \quad \Delta_B = \sum_i B_i (c_{R,i} - c_{P,i})/n, \quad \Delta_L = -\sum_i L_i c_{P,i}/n$$

The **dominance statistic** is:

$$T = \Delta_V - 0.8(\Delta_V + \Delta_B)$$

which is positive when visibility supplies more than 80% of what R's parsed rows contribute.

### Hypotheses and Endpoints

| Hypothesis | Claim | Verdict Rule | Endpoints |
|---|---|---|---|
| H1 | Most gain from unparsed rows | No gain unless lower bound of Δ > 0; then check T | 9 (first), 3 (second) |
| H2 | AUTHORS-1500 non-inferior to comparator | Holds if lower bound of θ > −δ (δ = 0.05) | 13 (first), 7 (second) |
| H3 | Budget forcing lowers truncation and raises parsing | Holds if both lower bounds > m (m = 0.05) | 9 (first), 3 (second) |
| H4 | Replication confirms defect's distortion | Three intervals: D_S, D_R, Γ = D_R − D_S | 16 pairs (first), 5 (second) |

### Statistical Methods

- **Bootstrap**: B = 50,000 resamples, percentile intervals, Bonferroni family adjustment (levels 99.44%, 99.62%, 99.44%, 99.69% for H1–H4)
- **Resampling units**: participant components for LongevityBench, items for MMLU-Pro and GSM8K
- **Seeds**: hash of partition, model, pool (base 4242)
- Intervals are approximate (percentile bootstrap), so family-wise error is approximately controlled

### Six Injected Defects

- **A**: Reference answer as assistant turn
- **B1**: Gold fallback (historical vs. derived)
- **B2**: Unparsed answered scored as "A"
- **C**: Skewed split (re-implementation of defunct case study split)
- **D**: Broken start (THINK-64)
- **E**: Tied output layer (Qwen3-1.7B-Longevity only)

## Empirical Validation / Results

### H1: Where Gains Come From

**Visibility dominated in 7 of 12 cells** across both studies. Key findings:

- **SmolLM3-3B**: Visibility dominated on all three pools (Δ from +0.060 to +0.403)
- **Intern-S1-mini**: Visibility dominated on letters and MMLU-Pro; ΔV ≥ Δ in all five cells
- **Qwen3-14B**: Visibility dominated on letters (Δ = +0.203 [ +0.186, +0.219]) and MMLU-Pro (Δ = +0.402 [ +0.350, +0.453])

For example, on Intern-S1-mini MMLU-Pro: Δ = +0.403 [ +0.352, +0.456], ΔV = +0.419, ΔB = −0.018, with only 107 rows in B.

The gate failed in 5 of 12 cells, including Qwen3-1.7B-Longevity (parses 1.000 of letter rows, Δ = −0.003) and on GSM8K where AUTHORS-1500 lost accuracy (e.g., Intern-S1-mini: Δ = −0.287 [ −0.355, −0.218] because ΔB = −0.343 was large and negative).

### H2: Baseline Adequacy

**First study**: AUTHORS-1500 was non-inferior to RESCUE-1500 in **7 of 9 cells** and to all four GEPA harnesses. It failed on GSM8K for SmolLM3-3B (θ = −0.089) and Intern-S1-mini (θ = −0.342).

**Second study (Qwen3-14B)**: The result inverted against GEPA harnesses:
- Both clean-start harnesses beat AUTHORS-1500 by more than the margin: θ = −0.337 and −0.342, with test accuracy 0.540 and 0.545 against 0.203
- But when compared to the **strongest constant label** (0.539), the harnesses were not detectably above: +0.001 and +0.006 on test, with intervals containing zero

The broken-start searches could not be repaired at 14B: every proposal scored zero, and 142–137 of 975 proposals failed the interface check.

### H3: Budget Forcing

**BUDGET-512-1500 lowered truncation and raised parsing** in 9 of 12 cells across both studies:
- First study: D_T from +0.260 to +0.979, D_P from +0.151 to +0.662 in six cells
- Second study: D_T from +0.531 to +0.974, D_P from +0.531 to +0.782 in all three cells
- THINK-512 truncated 1.000 of letter calls for Qwen3-14B

### H4: Replication of Defects

**The claim held in 9 of 20 evaluable pairs**, with 7 of these fixed by design identities (stars):

| Defect | Model | D_S | D_R | Γ | Claim |
|---|---|---|---|---|---|
| B1 gold fallback | Qwen3-1.7B | −0.145 | −0.148 | −0.003 | holds |
| B1 gold fallback | Qwen3-14B | −0.097 | −0.090 | +0.007 | holds |
| B2 unparsed-as-A | SmolLM3 | +0.208 | +0.223 | +0.015 | holds * |
| B2 unparsed-as-A | Intern-S1 | +0.289 | +0.292 | +0.003 | holds * |
| B2 unparsed-as-A | Qwen3-14B | +0.237 | +0.247 | +0.009 | holds * |
| D broken start | Qwen3-1.7B | +0.638 | +0.635 | −0.003 | holds * |
| D broken start | SmolLM3 | +0.164 | +0.157 | −0.007 | holds * |
| D broken start | Intern-S1 | +0.124 | +0.125 | +0.001 | holds * |
| D broken start | Qwen3-14B | +0.239 | +0.235 | −0.005 | holds * |

The claim **failed** in 4 pairs (gold fallback in SmolLM3-3B and Intern-S1-mini, skewed split in SmolLM3-3B, tied output layer in Qwen3-1.7B-Longevity) and was **inconclusive** in 5–6 pairs (reference-answer defect, skewed split).

### Descriptive Analyses

**Fine-tuning comparison** (base vs. tuned Qwen3-1.7B, AUTHORS-1500):

| Partition | Pool | Base | Tuned | Tuned − Base | Class |
|---|---|---|---|---|---|
| Test | LB letters | 0.217 | 0.631 | +0.414 [ +0.395, +0.433] | base below tuned |
| Test | MMLU-Pro | 0.384 | 0.220 | −0.164 [ −0.203, −0.125] | base above tuned |
| Test | GSM8K | 0.700 | 0.066 | −0.634 [ −0.677, −0.590] | base above tuned |

**GPU comparison**: Token IDs differed in 58.2% of replication grid calls and 57.6% of test grid calls between RTX 5090 and RTX PRO 6000, yet **all 9 endpoints gave the same verdict** and accuracy differed by at most 0.010.

## Theoretical and Practical Implications

### Conceptual Contributions

1. **Visibility decomposition**: The exact identity Δ = ΔV + ΔB + ΔL provides a principled way to split harness gains into visibility, quality, and format-loss components, applied prospectively rather than retrospectively.

2. **Baseline adequacy must be model-dependent**: The thinking-off baseline beat the strongest constant label only for the model tuned on the task format; GEPA beat the baseline only where the baseline was below that constant. This shows a harness gain must be read against trivial baselines.

3. **Prospective replication of defects**: Injecting known defects one at a time reveals that distortions persist across samples—a replication run through the same pipeline reproduces them instead of revealing them. Seven of the nine confirmed pairs were fixed by design identities.

4. **The constant-label issue**: A 30–34 point gain over a baseline can be meaningless when the strongest constant label achieves the same accuracy—the harnesses matched the "Yes" constant on the mortality task by answering "Yes" almost throughout (94.6–98.2% of parsed answers).

### Audit Check Findings

The empirical false alarms on 7 clean instances:
- IR4 (input reading): fires on all 7 (does not separate defective from clean pipelines)
- SCF1 and SCF2: fire on 4
- TB2 (trivial baseline): fires on 3
- SCF3: fires on 2
- Gain rule of PR: fires on 1
- TR (token reconciliation): fires on none

### Practical Recommendations

The paper suggests cheap checks for every round of self-improving loops:
- Report truncated and non-committed answers separately
- Run a thinking-off baseline
- Run constant-label baselines
- Read the rendered inputs
- Reconcile prompt tokens per call

## Conclusion

### Main Takeaways

1. **Most harness gains can be visibility**: In 7 of 12 cells, the gain of thinking-off over capped thinking came from rows the capped setting left unparsed—a search starting from such a configuration can find "repairs" that a thinking-off baseline already provides.

2. **Baseline adequacy is model-dependent**: AUTHORS-1500 was non-inferior to the rescue in most cells of the small models (7 of 9) and to all four GEPA harnesses at 1.7B, but at 14B clean-start harnesses beat it by 30–34 points—yet those harnesses were not detectably above the strongest constant label.

3. **Replication reproduces defects**: In 9 of 20 evaluable (defect, model) pairs, distortion was larger than five points on the selection sample and reappeared within five points on the check sample—a replication through the same pipeline reproduces the distortion instead of revealing it.

4. **Robustness to GPU**: Conclusions did not depend on token-level agreement across GPUs despite most token sequences changing.

### Future Directions

The paper explicitly states what it does not answer:
- Whether early visibility gains turn into quality gains later in a recursive loop
- Whether a loop can escape a start that GEPA could not repair at 14B
- The space of defects beyond the six studied (H4 is specific to these defects)
- Whether a search that changes request parameters (not just prompt text) could repair the broken 14B start

### Limitations

- Four models in the confirmatory design, three pools, six defects from one line of work
- Several endpoints carry stars: quantities fixed by design identity, so verdicts say little beyond the mechanism
- Power is mixed; H4 error control covers "holds" only, and "fails" is descriptive
- The GEPA arms searched prompt text only, with one proposer
- The freeze guards rules, not model-specific tuning
- The second preregistration was written after the first study's results were known

---

_Markdown view of https://picx.dev/p/kk7VFN, served by PicX — AI-generated visual whiteboard summaries of research papers._