# Agent Self-Improvement with Frozen Weights: The Added Value and Limits of Failure Traces

> Trace-based lessons improved decision accuracy by 4.0 points over matched search, but search alone captured 60% of the gain, with no improvement on conversations or held-out tasks.

- **Source:** [arXiv](https://arxiv.org/abs/2611.11342)
- **Published:** 2026-11-14
- **Permalink:** https://picx.dev/p/Q8I15o
- **Whiteboard:** https://picx.dev/p/Q8I15o/image

## Summary

## Summary (Overview)

- **Core question**: Can a frozen-weight AI agent's harness (prompts/notes) improve by learning from the agent's own failure traces, beyond what a matched search without trace access achieves?
- **Key finding**: Trace-based lessons improved decision accuracy by **+4.0 percentage points** over a matched no-evidence search on generated retail decisions, but the search alone captured ~60% of the total gain over the standard harness.
- **Negative results**: No demonstrated improvement on full τ² retail conversations (pass¹: −1.3 [−5.9, +3.3]) or on held-out BFCL tasks (gate winner did not replicate: +0.31 [−3.12, +3.65], Holm p = 1.000).
- **Notable observations**: A neutral note (no procedure) scored higher than either learned lesson set on held-out BFCL (+4.5 [+0.7, +8.4]), and a blind audit found **146 of 240 (60.8%)** held-out BFCL instances defective.
- **Methodological contributions**: A matched no-evidence search control, a harm-constrained Bayesian gate for lesson admission, and a pre-registered blind scorer audit.

---

## Introduction and Theoretical Foundation

The paper addresses a central problem in agent self-improvement: **deploying small open models with fixed weights** (Qwen3.5-9B here) and asking whether the *harness*—the prompt, notes, tools, and control logic around the model—can be improved from the agent's own interaction traces.

The authors distinguish two sources of gain when comparing a learned harness to its starting point:

1. **Search**: A writer that proposes texts and keeps the best-scoring ones on a development set can improve the agent without ever reading failures.
2. **Evidence**: The writer read the agent's failures and encoded something about them.

The key methodological innovation is a **matched no-evidence search control** that uses the same writer, loop, selection, budget ceilings, and development feedback, but with episode contents withheld (replaced by a fixed sentence on τ²; replaced by function documentation on BFCL).

The motivation stems from a skeptical literature:
- Harness evolution did not beat parallel sampling under matched budgets [5]
- Failure-derived experience repairs source tasks but rarely transfers to unseen ones [6–8]
- ALTK-Evolve reported pass⁴ rising from 57.5 to 67.5 on τ² retail from GPT-4.1's traces, but without a matched no-evidence control [10, 11]
- SkillsBench: self-written skills scored 42.9% vs 45.5% with no skills [12]
- Multiple late-2026 studies report regressions without significance tests [13–16]

---

## Methodology

### Agent and Harness

- **Agent**: Qwen3.5-9B, served with vLLM, built-in reasoning mode ("thinking") off, weights never updated
- **Standard harness (B1)**: τ² benchmark's reference agent with optional "think" tool; BFCL official conversation without notes message
- **Teacher writers**: Qwen3.8-Flash-Next-FP8 (τ², "T+") and Qwen3.5-397B-A17B (BFCL)

### Benchmarks

| Benchmark | Level | Details |
|---|---|---|
| τ²-bench retail [21] | Decision | 2,997 sealed test cases over 3 tasks (address, variant, sequence) |
| τ²-bench retail | Conversation | 114 tasks × 4 trials, v1.0.1 (pinned commit 672227c6) |
| BFCL multi-turn [23] | Multi-turn | 200 roots → 50 discovery, 30 gate, 120 held-out; 60 gate + 240 held-out instances |

### The Bayesian Gate

Each candidate lesson is switched on/off at random across many runs, with a **hierarchical Bayesian model** pooling estimates across contexts. A lesson passes only if it has:
- Credible gain > 1 point
- Enough exposure in every context
- Per-context **non-inferiority** within a registered 1-point margin (probability of loss > 1 point below 0.10 in every context)
- **Bounded harm** (wrong write on τ², wrong final state on BFCL)
- On τ²: must pass under **three named priors** (sceptical, weakly informative, slightly negative)

### Pre-registration and Auditing

- Every primary contrast, interval method, threshold, and decision rule was **hash-chained in a registry** before data were opened
- GPU experiments run by an external collaborator from versioned experiment kits
- **Blind scorer audit**: Two independent model-based reviewers labelled all 300 gate and test instances from benchmark material only, never from agent trajectories; defect lists frozen and registered before outcomes were opened

---

## Empirical Validation / Results

### Decision-Level Results (τ², 2,997 sealed cases)

| Arm | Accuracy |
|---|---|
| Model only (B0) | — |
| Standard harness (B1) | 70.3% |
| No-evidence search | 76.8% |
| Trace-written lessons | 80.9% |

**Registered primary contrast**: Trace lessons − no-evidence search = **+4.0 [+2.3, +5.7]** (passes superiority; point estimate below the 5-point registered worthwhile gain)

- Wrong writes fell by 3.1 points [−4.6, −1.7] relative to search
- Lessons vs standard harness: +10.5 [+8.5, +12.5]
- The control captures 6.5 of those 10.5 points (~60%) **without traces**—entirely on the task whose rule is stated in the writer's task description

### Conversation-Level Results (τ², 114 tasks × 4 trials)

- Standard harness pass¹: 82.9%; with lessons: 81.6% → **−1.3 [−5.9, +3.3]** (no demonstrated improvement)
- On 40 never-used tasks: −0.6 [−6.9, +6.3]
- Targeted behavior did change: conversations with item-before-address changes fell from 6 to 0

### τ² Gate Results

From 12 candidate teacher lessons (20,400 decisions over 1,200 prefixes), **two passed** every check under every prior:

| Lesson | Gain (90% CrI) |
|---|---|
| "Change the address first" | +4.7 [+4.0, +5.4] |
| "List only the first change" | +2.7 [+2.0, +3.4] |
| Both together | +7.4 [+6.3, +8.4] |

Four lessons pushing toward asking/confirming again were rejected (each credibly hurt at least one context). **No held-out test was run** for these.

### BFCL Results

| Stage | Result |
|---|---|
| Gate (60 instances, 2,820 episodes) | 0 of 8 trace lessons passed; only search lesson passed ("when the current directory is unknown, call pwd"): +4.9 [+1.9, +8.0] |
| Gate, clean instances only (18) | +3.9 [−0.4, +8.3] (includes zero) |
| Held-out (120 roots, 240 instances, 5,760 episodes) | Standard harness − search lesson: +0.31 [−3.12, +3.65], Holm p = 1.000 → **did not replicate** |
| Held-out, neutral note vs standard (descriptive) | +4.5 [+0.7, +8.4] |
| Held-out, standard − 8 trace lessons (no gate) | −0.52 [−4.38, +3.33], Holm p = 1.000 |

### Scorer Audit

- **146 of 240 held-out instances (60.8% [54.5, 66.8])** and 42 of 60 gate instances judged clearly defective, almost all because a correct response is scored wrong
- Reviewer agreement: κ = 0.84
- On clean held-out instances, both learned-lesson arms sat ~6 points **below** the standard harness:
  - Standard − search lesson: +6.0 [+0.2, +12.0] (Holm p = 0.149)
  - Standard − trace lessons: +6.2 [−0.5, +13.2] (Holm p = 0.153)

---

## Theoretical and Practical Implications

1. **Controlled comparisons are essential**: Without the matched no-evidence search, the decision-level gain would have looked **2.6 times larger**. The search alone reached ~60% of the total improvement—a finding that challenges naive attributions of gains to trace evidence.

2. **Screens are not certifications**: The two admitted τ² lessons remain untested in conversations; the BFCL gate winner failed to replicate on held-out tasks. Selection on noise is a real risk—across four identical trials of published τ² retail configurations, 33–55% of task verdicts flipped [27].

3. **Note effects may be non-specific**: The neutral note outperforming both lesson sets (descriptively) suggests learned notes should be compared against neutral notes, not only against no note.

4. **Benchmark scorers need audits**: With 60.8% of BFCL held-out instances defective, the audit fundamentally changed interpretability: the gate winner lost its credible gain on clean instances, and clean-instance analyses reversed the sign of the primary contrast.

5. **Practical guidance for practitioners**: Report a matched no-evidence search, compare notes with a neutral note, confirm gate winners on held-out data, and audit the scorer blind *before* the comparison.

---

## Conclusion

For a frozen Qwen3.5-9B agent, teacher access to the agent's mistakes provided a modest, controlled gain on generated decisions (+4.0 points over matched search), but:

- No improvement was demonstrated on full conversations or held-out BFCL tasks
- Most of the decision-level gain came from search alone, not from reading failures
- The methods proposed—matched no-evidence search, neutral-note comparison, held-out gate confirmation, and blind scorer audit—are expected to outlast the specific results

**What the authors do not claim**:
- That lessons have no effect on full conversations or held-out BFCL (intervals allow a few points either way)
- That the two admitted τ² lessons are certified, or that their gain comes from traces
- That results extend beyond one agent, two benchmarks, and one teacher family

**Limitations disclosed**: One frozen 9B model with thinking off; τ² writers were told the target rules (measures phrasing ability, not discovery); BFCL search writer saw documentation the trace writer did not; several collaborator-run deviations (sampled decoding temperature 0.7 vs official 0.001, a scorer serialization fix, a re-run after folder loss) are disclosed in full in the extended report.

---

_Markdown view of https://picx.dev/p/Q8I15o, served by PicX — AI-generated visual whiteboard summaries of research papers._