Full text not available for this paper

Summary of "Do LLM Agents Still Need to Be Told How to Self-Improve?"

Summary (Overview)

  • Core Question: The paper investigates whether sufficiently capable LLM agents can construct their own self-improvement algorithms when given only primitive tools, resources, and a general goal—without being prescribed an explicit learning loop or provided real-world training data.

  • Key Finding: Newer, more capable model generations (Claude Opus-5, GPT 6-Astra, GLM 5.3) successfully self-organize learning procedures and improve their performance on held-out gameplay tasks, while older generations (Claude Opus-4.6, GPT 5.6-Sol, GLM 4.7) fail to do so.

  • Strategy Discovery: Self-improving agents spontaneously employ human-recognizable learning strategies aligned with Zimmerman's self-regulated learning framework, including A/B testing, difficulty-calibrated task generation, and adversarial game design.

  • Transferability: Self-learned text artifacts generalize across different game engines, horizons, representations (including 3D navigation), and even transfer between different models.

  • Information Trade-off: Providing the teacher with more information about the evaluation setting improves in-distribution performance but can encourage overfitting and reduce generalization.

Introduction and Theoretical Foundation

The paper addresses a fundamental question in the evolution of LLM agents: as models become more capable, do they still need humans to specify how they should self-improve? The authors position this work at the intersection of:

  1. Self-improvement methods: Prior work delegates various components of optimization to models themselves (e.g., generating supervised data, constructing environments, rewriting tools).

  2. Safety concerns: Understanding whether self-improvement can occur without explicit loops is critical for developing safeguards against misaligned self-improvement.

  3. Cognitive frameworks: The authors draw on Zimmerman's cyclical phases model of self-regulated learning (SRL), which organizes learning into forethought (goal setting), performance (strategy execution), and self-reflection (evaluation) phases.

  4. Machine learning theory: The work connects to asymmetric self-play and the "Zone of Proximal Development" from developmental psychology, where tasks of intermediate difficulty maximize learning.

The theoretical foundation rests on the observation that as models become more broadly capable, they require fewer human-imposed inductive biases—a trend the authors extend to the meta-optimization layer.

Methodology

Setting: Gameplay with Limited Information

The paper uses turn-based games with undisclosed mechanics as a testbed. Formally, this is a partially observable Markov decision process (POMDP):

  • State: st∈Ss_t \in S at turn tt
  • Observation: ot∈Oo_t \in O received by the player
  • Action: at∈Aa_t \in A chosen by the player
  • Transition: st+1∼Pg(⋅∣st,at)s_{t+1} \sim P_g(\cdot | s_t, a_t)
  • Observation: ot+1∼Og(⋅∣st+1)o_{t+1} \sim O_g(\cdot | s_{t+1})
  • Reward: rt=Rg(st,at,st+1)∈{0,1}r_t = R_g(s_t, a_t, s_{t+1}) \in \{0, 1\} (binary, 1 only upon reaching a winning state)

The player πθ\pi_\theta lacks access to PgP_g and RgR_g, acting based on interaction history ht=(o1,a1,…,ot)h_t = (o_1, a_1, \ldots, o_t) and context cc.

Teacher-Player Architecture

The core experimental design uses two instances of the same frozen LLM:

  • Player agent πplayer\pi_{\text{player}}: Has context c=(cP,cL)c = (c_P, c_L) where cPc_P is the player prompt and cLc_L is auxiliary text files (e.g., skill files).
  • Teacher agent πteacher\pi_{\text{teacher}}: Modifies cc through text edits, with an update defined as a single edit tool-call.

Teacher Environment

The teacher operates in a sandboxed harness (Claude Code) with:

  • Tools: Read/Write, Bash, and Gameplay (which executes games in fresh player sessions and returns scores/transcripts)
  • Budget: 20 updates to player context, 150 gameplays
  • Context: System prompt specifying the goal of improving player performance on unseen games

Self-Organized Self-Improvement Algorithm

The teacher composes its own learning procedure freely:

Algorithm 1: Self-organized self-improvement
Require: Budget B, teacher context I
1: S ← InitSandbox(c₀, I, B)
2: A_T ← {Read, Write, Bash, Gameplay}
3: while B > 0 do
4:   a ∼ π_teacher(· | S), a ∈ A_T
5:   S ← Execute(S, a)
6: end while
7: return player context c_final

Evaluation

Performance is measured via expected reward on held-out test games:

J(c)=1∣Dtest∣∑g∈Dtest[Eπplayer(⋅∣ht,c)(∑t=1Trt)]J(c) = \frac{1}{|\mathcal{D}_{\text{test}}|} \sum_{g \in \mathcal{D}_{\text{test}}} \left[ \mathbb{E}_{\pi_{\text{player}}(\cdot | h_t, c)} \left( \sum_{t=1}^{T} r_t \right) \right]

Self-improvement is measured as J(cfinal)−J(c0)>0J(c_{\text{final}}) - J(c_0) > 0.

Models and Benchmarks

  • Models: Claude Opus-{4.6, 5}, GPT {5.6-Sol, 6-Astra}, GLM-{4.7, 5.3}
  • Benchmarks (increasing distance from training setting):
    • PuzzleScript-t20 (same engine, short horizon)
    • PuzzleScript-t30 (same engine, longer horizon)
    • ARC-AGI-3 L1 (different engine and representation)
    • MazeBench (3D navigation via ASCII projections)

Empirical Validation / Results

1. Newer Models Self-Improve

ModelImprovement over Baseline
Claude Opus-5+20%
GPT 6-Astra+10%
GLM 5.3+6%
Older generationsLittle or no improvement

2. Self-Organized Learning Matches Prescribed Optimization

Claude Opus-5's self-organized learning with self-generated games achieved comparable performance to GEPA (a prescribed evolutionary prompt optimization method) that hill-climbs on 20 real in-distribution games.

3. Recognizable Learning Strategies

Algorithmically measurable behaviors of self-improving models:

  • A/B testing: Testing the same game with and without teacher-written artifacts
  • Fewer trivial games: Reduced generation of games with short solution lengths (< 5) and 100% win rates
  • Smaller context updates (for Opus and GLM)
  • More performance-improving updates (for GPT and GLM)

Self-Regulated Learning alignment: Models that self-improve score higher across most of the 13 subprocesses of Zimmerman's SRL framework, including:

  • Task strategies: Writing procedural game generators with BFS-based difficulty filtering
  • Learning goal orientation: Designing adversarial games to stress-test advice
  • Self-experimentation: A/B testing to evaluate text artifacts

Learnability-seeking task selection: Self-improving agents concentrate replay effort on games with 20-60% observed win rates, exhibiting an "inverted-U" pattern resembling learnability objectives in asymmetric self-play and unsupervised environment design (UED).

4. Transfer Results

Context optimized under PuzzleScript-t20 gameplays improved:

  • PuzzleScript-t20: +20%
  • PuzzleScript-t30: +9%
  • ARC-AGI-L1: +21%
  • MazeBench SPL: +0.15

5. Cross-Model Transfer

PlayerText Artifact SourceWin Rate (%)
Opus 5None25.4 ± 3.6
Opus 5GPT-6 Astra47.5 ± 4.5
GPT-6 AstraNone67.1 ± 3.8
GPT-6 AstraOpus 572.1 ± 4.0

Learned text transfers in both directions, even when the source model has lower baseline performance.

6. Information and Generalization Trade-off

  • Context information: Richer descriptions of the target game domain initially improve both approximation and generalization, but at high detail levels, transfer plateaus or declines.
  • Tool information: Adding tools relevant to both benchmarks helps; imposing t20-specific token/turn budgets improves t20 performance but reduces t30 transfer.

Theoretical and Practical Implications

Theoretical Implications

  1. Meta-capability trajectory: The key finding is not that a single generation self-improves, but that this meta-capability improves across generations. This suggests a scaling law for self-improvement ability itself.

  2. Emergent alignment with cognitive theory: Without explicit instruction, capable models spontaneously implement strategies matching Zimmerman's SRL framework and the Zone of Proximal Development—suggesting these frameworks capture universal principles of effective learning.

  3. Information-optimality trade-off: The results reveal a fundamental tension between providing evaluation-relevant information (which improves in-distribution performance) and maintaining generalization (which suffers from over-specialization).

Practical Implications

  1. Reduced scaffolding requirements: Future agents may require even less human-specified structure for self-improvement. One Claude Opus-5 session with Bash access spontaneously wrote and ran game simulations without any Gameplay tool or prompt reference to game simulation.

  2. Safety considerations: Self-organized improvement is not hypothetical—it occurs in controlled settings. As this capability scales to more complex domains, scientific understanding and characterization of model behavior become critical.

  3. Benchmark design: The transfer results suggest that evaluating self-improvement requires multiple benchmarks at varying distances from the training distribution.

Conclusion

The paper demonstrates that sufficiently capable LLM agents can construct their own self-improvement procedures when given only primitive tools and a general goal, without prescribed learning loops or real-world data. Key takeaways:

  1. Capability threshold: Self-organized self-improvement emerges only in newer, more capable model generations, suggesting this is a meta-capability that improves with model scale.

  2. Strategy convergence: Effective self-improvers converge on similar, human-recognizable strategies across model families, aligning with cognitive frameworks and ML literature on learnability.

  3. Generalization: Self-learned text artifacts transfer across environments, models, and representations, though evaluation-setting awareness creates an overfitting risk.

Future directions identified by the authors:

  • Understanding what enables continued improvement beyond ~20 updates (plateauing effects observed)
  • Disentangling causal relationships between specific strategies and self-improvement outcomes
  • Extending findings to more open-ended domains where models have weaker priors
  • Characterizing less human-interpretable behaviors that may also shape performance

Related papers