Summary (Overview)

• Core question: Can a frozen-weight AI agent's harness (prompts/notes) improve by learning from the agent's own failure traces, beyond what a matched search without trace access achieves?
• Key finding: Trace-based lessons improved decision accuracy by +4.0 percentage points over a matched no-evidence search on generated retail decisions, but the search alone captured ~60% of the total gain over the standard harness.
• Negative results: No demonstrated improvement on full τ² retail conversations (pass¹: −1.3 [−5.9, +3.3]) or on held-out BFCL tasks (gate winner did not replicate: +0.31 [−3.12, +3.65], Holm p = 1.000).
• Notable observations: A neutral note (no procedure) scored higher than either learned lesson set on held-out BFCL (+4.5 [+0.7, +8.4]), and a blind audit found 146 of 240 (60.8%) held-out BFCL instances defective.
• Methodological contributions: A matched no-evidence search control, a harm-constrained Bayesian gate for lesson admission, and a pre-registered blind scorer audit.

---

Introduction and Theoretical Foundation

The paper addresses a central problem in agent self-improvement: deploying small open models with fixed weights (Qwen3.5-9B here) and asking whether the harness—the prompt, notes, tools, and control logic around the model—can be improved from the agent's own interaction traces.

The authors distinguish two sources of gain when comparing a learned harness to its starting point:

1. Search: A writer that proposes texts and keeps the best-scoring ones on a development set can improve the agent without ever reading failures.
2. Evidence: The writer read the agent's failures and encoded something about them.

The key methodological innovation is a matched no-evidence search control that uses the same writer, loop, selection, budget ceilings, and development feedback, but with episode contents withheld (replaced by a fixed sentence on τ²; replaced by function documentation on BFCL).

The motivation stems from a skeptical literature:

• Harness evolution did not beat parallel sampling under matched budgets [5]
• Failure-derived experience repairs source tasks but rarely transfers to unseen ones [6–8]
• ALTK-Evolve reported pass⁴ rising from 57.5 to 67.5 on τ² retail from GPT-4.1's traces, but without a matched no-evidence control [10, 11]
• SkillsBench: self-written skills scored 42.9% vs 45.5% with no skills [12]
• Multiple late-2026 studies report regressions without significance tests [13–16]

---

Methodology

Agent and Harness

• Agent: Qwen3.5-9B, served with vLLM, built-in reasoning mode ("thinking") off, weights never updated
• Standard harness (B1): τ² benchmark's reference agent with optional "think" tool; BFCL official conversation without notes message
• Teacher writers: Qwen3.8-Flash-Next-FP8 (τ², "T+") and Qwen3.5-397B-A17B (BFCL)

Benchmarks

The Bayesian Gate

Each candidate lesson is switched on/off at random across many runs, with a hierarchical Bayesian model pooling estimates across contexts. A lesson passes only if it has:

• Credible gain > 1 point
• Enough exposure in every context
• Per-context non-inferiority within a registered 1-point margin (probability of loss > 1 point below 0.10 in every context)
• Bounded harm (wrong write on τ², wrong final state on BFCL)
• On τ²: must pass under three named priors (sceptical, weakly informative, slightly negative)

Pre-registration and Auditing

• Every primary contrast, interval method, threshold, and decision rule was hash-chained in a registry before data were opened
• GPU experiments run by an external collaborator from versioned experiment kits
• Blind scorer audit: Two independent model-based reviewers labelled all 300 gate and test instances from benchmark material only, never from agent trajectories; defect lists frozen and registered before outcomes were opened

---

Empirical Validation / Results

Decision-Level Results (τ², 2,997 sealed cases)

Registered primary contrast: Trace lessons − no-evidence search = +4.0 [+2.3, +5.7] (passes superiority; point estimate below the 5-point registered worthwhile gain)

• Wrong writes fell by 3.1 points [−4.6, −1.7] relative to search
• Lessons vs standard harness: +10.5 [+8.5, +12.5]
• The control captures 6.5 of those 10.5 points (~60%) without traces—entirely on the task whose rule is stated in the writer's task description

Conversation-Level Results (τ², 114 tasks × 4 trials)

• Standard harness pass¹: 82.9%; with lessons: 81.6% → −1.3 [−5.9, +3.3] (no demonstrated improvement)
• On 40 never-used tasks: −0.6 [−6.9, +6.3]
• Targeted behavior did change: conversations with item-before-address changes fell from 6 to 0

τ² Gate Results

From 12 candidate teacher lessons (20,400 decisions over 1,200 prefixes), two passed every check under every prior:

Four lessons pushing toward asking/confirming again were rejected (each credibly hurt at least one context). No held-out test was run for these.

BFCL Results

Scorer Audit

• 146 of 240 held-out instances (60.8% [54.5, 66.8]) and 42 of 60 gate instances judged clearly defective, almost all because a correct response is scored wrong
• Reviewer agreement: κ = 0.84
• On clean held-out instances, both learned-lesson arms sat ~6 points below the standard harness:
  • Standard − search lesson: +6.0 [+0.2, +12.0] (Holm p = 0.149)
  • Standard − trace lessons: +6.2 [−0.5, +13.2] (Holm p = 0.153)

---

Theoretical and Practical Implications

1. Controlled comparisons are essential: Without the matched no-evidence search, the decision-level gain would have looked 2.6 times larger. The search alone reached ~60% of the total improvement—a finding that challenges naive attributions of gains to trace evidence.

2. Screens are not certifications: The two admitted τ² lessons remain untested in conversations; the BFCL gate winner failed to replicate on held-out tasks. Selection on noise is a real risk—across four identical trials of published τ² retail configurations, 33–55% of task verdicts flipped [27].

3. Note effects may be non-specific: The neutral note outperforming both lesson sets (descriptively) suggests learned notes should be compared against neutral notes, not only against no note.

4. Benchmark scorers need audits: With 60.8% of BFCL held-out instances defective, the audit fundamentally changed interpretability: the gate winner lost its credible gain on clean instances, and clean-instance analyses reversed the sign of the primary contrast.

5. Practical guidance for practitioners: Report a matched no-evidence search, compare notes with a neutral note, confirm gate winners on held-out data, and audit the scorer blind before the comparison.

---

Conclusion

For a frozen Qwen3.5-9B agent, teacher access to the agent's mistakes provided a modest, controlled gain on generated decisions (+4.0 points over matched search), but:

• No improvement was demonstrated on full conversations or held-out BFCL tasks
• Most of the decision-level gain came from search alone, not from reading failures
• The methods proposed—matched no-evidence search, neutral-note comparison, held-out gate confirmation, and blind scorer audit—are expected to outlast the specific results

What the authors do not claim:

• That lessons have no effect on full conversations or held-out BFCL (intervals allow a few points either way)
• That the two admitted τ² lessons are certified, or that their gain comes from traces
• That results extend beyond one agent, two benchmarks, and one teacher family

Limitations disclosed: One frozen 9B model with thinking off; τ² writers were told the target rules (measures phrasing ability, not discovery); BFCL search writer saw documentation the trace writer did not; several collaborator-run deviations (sampled decoding temperature 0.7 vs official 0.001, a scorer serialization fix, a re-run after folder loss) are disclosed in full in the extended report.