Full text not available for this paper
Summary of "Do LLM Agents Still Need to Be Told How to Self-Improve?"
Summary (Overview)
-
Core Question: The paper investigates whether sufficiently capable LLM agents can construct their own self-improvement algorithms when given only primitive tools, resources, and a general goal—without being prescribed an explicit learning loop or provided real-world training data.
-
Key Finding: Newer, more capable model generations (Claude Opus-5, GPT 6-Astra, GLM 5.3) successfully self-organize learning procedures and improve their performance on held-out gameplay tasks, while older generations (Claude Opus-4.6, GPT 5.6-Sol, GLM 4.7) fail to do so.
-
Strategy Discovery: Self-improving agents spontaneously employ human-recognizable learning strategies aligned with Zimmerman's self-regulated learning framework, including A/B testing, difficulty-calibrated task generation, and adversarial game design.
-
Transferability: Self-learned text artifacts generalize across different game engines, horizons, representations (including 3D navigation), and even transfer between different models.
-
Information Trade-off: Providing the teacher with more information about the evaluation setting improves in-distribution performance but can encourage overfitting and reduce generalization.
Introduction and Theoretical Foundation
The paper addresses a fundamental question in the evolution of LLM agents: as models become more capable, do they still need humans to specify how they should self-improve? The authors position this work at the intersection of:
-
Self-improvement methods: Prior work delegates various components of optimization to models themselves (e.g., generating supervised data, constructing environments, rewriting tools).
-
Safety concerns: Understanding whether self-improvement can occur without explicit loops is critical for developing safeguards against misaligned self-improvement.
-
Cognitive frameworks: The authors draw on Zimmerman's cyclical phases model of self-regulated learning (SRL), which organizes learning into forethought (goal setting), performance (strategy execution), and self-reflection (evaluation) phases.
-
Machine learning theory: The work connects to asymmetric self-play and the "Zone of Proximal Development" from developmental psychology, where tasks of intermediate difficulty maximize learning.
The theoretical foundation rests on the observation that as models become more broadly capable, they require fewer human-imposed inductive biases—a trend the authors extend to the meta-optimization layer.
Methodology
Setting: Gameplay with Limited Information
The paper uses turn-based games with undisclosed mechanics as a testbed. Formally, this is a partially observable Markov decision process (POMDP):
- State: at turn
- Observation: received by the player
- Action: chosen by the player
- Transition:
- Observation:
- Reward: (binary, 1 only upon reaching a winning state)
The player lacks access to and , acting based on interaction history and context .
Teacher-Player Architecture
The core experimental design uses two instances of the same frozen LLM:
- Player agent : Has context where is the player prompt and is auxiliary text files (e.g., skill files).
- Teacher agent : Modifies through text edits, with an update defined as a single edit tool-call.
Teacher Environment
The teacher operates in a sandboxed harness (Claude Code) with:
- Tools: Read/Write, Bash, and Gameplay (which executes games in fresh player sessions and returns scores/transcripts)
- Budget: 20 updates to player context, 150 gameplays
- Context: System prompt specifying the goal of improving player performance on unseen games
Self-Organized Self-Improvement Algorithm
The teacher composes its own learning procedure freely:
Algorithm 1: Self-organized self-improvement
Require: Budget B, teacher context I
1: S ← InitSandbox(c₀, I, B)
2: A_T ← {Read, Write, Bash, Gameplay}
3: while B > 0 do
4: a ∼ π_teacher(· | S), a ∈ A_T
5: S ← Execute(S, a)
6: end while
7: return player context c_final
Evaluation
Performance is measured via expected reward on held-out test games:
Self-improvement is measured as .
Models and Benchmarks
- Models: Claude Opus-{4.6, 5}, GPT {5.6-Sol, 6-Astra}, GLM-{4.7, 5.3}
- Benchmarks (increasing distance from training setting):
- PuzzleScript-t20 (same engine, short horizon)
- PuzzleScript-t30 (same engine, longer horizon)
- ARC-AGI-3 L1 (different engine and representation)
- MazeBench (3D navigation via ASCII projections)
Empirical Validation / Results
1. Newer Models Self-Improve
| Model | Improvement over Baseline |
|---|---|
| Claude Opus-5 | +20% |
| GPT 6-Astra | +10% |
| GLM 5.3 | +6% |
| Older generations | Little or no improvement |
2. Self-Organized Learning Matches Prescribed Optimization
Claude Opus-5's self-organized learning with self-generated games achieved comparable performance to GEPA (a prescribed evolutionary prompt optimization method) that hill-climbs on 20 real in-distribution games.
3. Recognizable Learning Strategies
Algorithmically measurable behaviors of self-improving models:
- A/B testing: Testing the same game with and without teacher-written artifacts
- Fewer trivial games: Reduced generation of games with short solution lengths (< 5) and 100% win rates
- Smaller context updates (for Opus and GLM)
- More performance-improving updates (for GPT and GLM)
Self-Regulated Learning alignment: Models that self-improve score higher across most of the 13 subprocesses of Zimmerman's SRL framework, including:
- Task strategies: Writing procedural game generators with BFS-based difficulty filtering
- Learning goal orientation: Designing adversarial games to stress-test advice
- Self-experimentation: A/B testing to evaluate text artifacts
Learnability-seeking task selection: Self-improving agents concentrate replay effort on games with 20-60% observed win rates, exhibiting an "inverted-U" pattern resembling learnability objectives in asymmetric self-play and unsupervised environment design (UED).
4. Transfer Results
Context optimized under PuzzleScript-t20 gameplays improved:
- PuzzleScript-t20: +20%
- PuzzleScript-t30: +9%
- ARC-AGI-L1: +21%
- MazeBench SPL: +0.15
5. Cross-Model Transfer
| Player | Text Artifact Source | Win Rate (%) |
|---|---|---|
| Opus 5 | None | 25.4 ± 3.6 |
| Opus 5 | GPT-6 Astra | 47.5 ± 4.5 |
| GPT-6 Astra | None | 67.1 ± 3.8 |
| GPT-6 Astra | Opus 5 | 72.1 ± 4.0 |
Learned text transfers in both directions, even when the source model has lower baseline performance.
6. Information and Generalization Trade-off
- Context information: Richer descriptions of the target game domain initially improve both approximation and generalization, but at high detail levels, transfer plateaus or declines.
- Tool information: Adding tools relevant to both benchmarks helps; imposing t20-specific token/turn budgets improves t20 performance but reduces t30 transfer.
Theoretical and Practical Implications
Theoretical Implications
-
Meta-capability trajectory: The key finding is not that a single generation self-improves, but that this meta-capability improves across generations. This suggests a scaling law for self-improvement ability itself.
-
Emergent alignment with cognitive theory: Without explicit instruction, capable models spontaneously implement strategies matching Zimmerman's SRL framework and the Zone of Proximal Development—suggesting these frameworks capture universal principles of effective learning.
-
Information-optimality trade-off: The results reveal a fundamental tension between providing evaluation-relevant information (which improves in-distribution performance) and maintaining generalization (which suffers from over-specialization).
Practical Implications
-
Reduced scaffolding requirements: Future agents may require even less human-specified structure for self-improvement. One Claude Opus-5 session with Bash access spontaneously wrote and ran game simulations without any Gameplay tool or prompt reference to game simulation.
-
Safety considerations: Self-organized improvement is not hypothetical—it occurs in controlled settings. As this capability scales to more complex domains, scientific understanding and characterization of model behavior become critical.
-
Benchmark design: The transfer results suggest that evaluating self-improvement requires multiple benchmarks at varying distances from the training distribution.
Conclusion
The paper demonstrates that sufficiently capable LLM agents can construct their own self-improvement procedures when given only primitive tools and a general goal, without prescribed learning loops or real-world data. Key takeaways:
-
Capability threshold: Self-organized self-improvement emerges only in newer, more capable model generations, suggesting this is a meta-capability that improves with model scale.
-
Strategy convergence: Effective self-improvers converge on similar, human-recognizable strategies across model families, aligning with cognitive frameworks and ML literature on learnability.
-
Generalization: Self-learned text artifacts transfer across environments, models, and representations, though evaluation-setting awareness creates an overfitting risk.
Future directions identified by the authors:
- Understanding what enables continued improvement beyond ~20 updates (plateauing effects observed)
- Disentangling causal relationships between specific strategies and self-improvement outcomes
- Extending findings to more open-ended domains where models have weaker priors
- Characterizing less human-interpretable behaviors that may also shape performance
Related papers
- Shared Weights, Selected Computations: How Looped Transformers Route What Each Loop Does
Looped Transformers perform state-dependent computation, not fixed one-loop-one-step algorithms, and a learned linear layer can steer frozen shared weights via attention routing.
- Inside the Loop: A Mechanistic Study of Looped Transformers on Permutation Traversal
Looped transformers solve sequential algorithmic tasks untied models of equal effective depth cannot, by executing one algorithm step per iteration without intermediate supervision.
- From Repositories to Training Environments: Scaling Long-Horizon Coding Agents with Environment Evolution Loop
RepoFactory's proxy-driven environment evolution loop yields RepoHorizen-35B-A3B, a model improving coding-agent benchmarks by 22.6 points on NL2Repo and 8.5 points on average.