SelfImprove-Bench: Evaluating Generalization in Self-Improving Agents

Summary (Overview)

Novel evaluation framework: The paper introduces SelfImprove-Bench, a benchmark for evaluating recursively self-improving (RSI) agents that organizes tasks into three primary capabilities—Exploring, Planning, and Reasoning—instantiated on games for controllability and reproducibility.
Generalization ladder (G0–G4): The framework introduces a five-level generalization hierarchy distinguishing replay performance (G0), unseen instances at same difficulty (G1), new difficulty levels (G2), different tasks within the same capability (G3), and tasks across capability categories (G4).
Key finding—inconsistent generalization: Experiments with three representative RSI methods (ACE, AHE, HyperAgents) reveal that evolutionary gains are inconsistent across generalization levels, with ranking reversals on harder instances and different task types; similar final scores can conceal distinct evolution patterns.
Scalable task generation: The framework includes configurable task generation and orchestration using task templates and difficulty coefficients, supporting sustained evaluation as agents evolve and approach performance ceilings.
Fine-grained trace analysis: A semantic process analysis module characterizes refinement modifications by functional target, applicability scope, and modification surface, revealing method-specific evolution patterns.

Introduction and Theoretical Foundation

Background and Motivation

Recursively self-improving (RSI) systems aim to autonomously improve their capabilities by refining working pipelines and recursively improving the strategies that guide subsequent refinements. A central question is whether these improvements generalize beyond encountered instances to harder tasks, different task types, and ultimately distinct capabilities.

Three Unresolved Challenges in Existing Benchmarks

Systematic generalization assessment: Existing benchmarks examine selected settings (cross-task or cross-difficulty transfer) but lack a unified taxonomy spanning instances, difficulty levels, tasks, and capabilities.
Scalable RSI evaluation: Fixed task pools and limited difficulty ranges constrain sustained evaluation as agents evolve and approach performance ceilings.
Fine-grained process diagnosis: Existing analyses rely primarily on aggregate metrics, selected case studies, or persistence ablations, leaving the relationship between evolution trajectories and generalization outcomes insufficiently understood.

RSI Method Classification

The paper classifies RSI methods into three non-exclusive strategies:

Experience-level RSI: Adapts prompts, memories, and playbooks (e.g., Reflexion, A-MEM, ACE)
Harness-level RSI: Modifies tools, workflows, and executable code (e.g., SICA, Self-Harness, AHE)
Meta-level RSI: Refines candidate generation or selection (e.g., DGM's evolving self-modification agent, HyperAgents)

The paper focuses on non-training RSI methods, whose explicit modifications to prompts, memories, skills, and harnesses support a unified evaluation protocol and interpretable evolution traces.

Methodology

Evolution Protocol

Task construction: Each instance $\tau = G_g(d, \xi)$ is generated from game $g$ , difficulty $d$ , and seed $\xi$ . Configurations specify held-in sequences $S_{in} = (\tau_1, \dots, \tau_N)$ for execution/refinement and held-out sequences $S_{out} = (\tilde{\tau}_1, \dots, \tilde{\tau}_M)$ for evaluation.

Agent initialization: Each RSI method starts with agent $A_0 = (B, H_0)$ , comprising a common backbone model $B$ and method-specific initial harness $H_0$ .

Refinement process: At each step $t$ , agent $A_{t-1}$ executes task $\tau_t$ , producing an interaction trace and score. Each execution (except replays) is followed by refinement using benchmark-provided feedback. Methods may inspect and modify their workspace but cannot access benchmark modules or evaluate held-out instances.

Generalization Ladder

Level	Definition	Comparison
G0	Replay performance on same instances	$A_{replay,j}$ vs. $A_{init,j}$ on replayed task
G1	Unseen instances at same difficulty	$A_T$ vs. $A_0$ on $G_g(d_0, \xi')$
G2	New difficulty levels within same task	$A_T$ vs. $A_0$ on $G_g(d_1, \xi')$ with $d_1 > d_0$
G3	Different tasks within same capability	$A_T$ vs. $A_0$ on game $g'$ with $c(g') = c(g)$
G4	Tasks across capability categories	$A_T$ vs. $A_0$ on game $g'$ with $c(g') \neq c(g)$

HCI Metric

The Headroom-Closed Index (HCI) normalizes gains relative to the initial gap to a reference score:

\mathrm{HCI} = \frac{s_a - s_b}{s_r - s_b}

where $s_b$ and $s_a$ are mean baseline and comparison scores, and $s_r$ is the reference score.

Semantic Process Analysis Dimensions

Semantic relations: Whether a proposal introduces new content, contradicts/revises existing proposals, or duplicates them
Applicability scope: Task-specific, category-general, or fully general proposals
Functional targets: Exploration, reasoning, planning, or execution
Modification surfaces: Contextual artifacts, execution harness, or refinement mechanism

Empirical Validation / Results

Key Results Across Generalization Levels

G0 (Replay): Refinement does not always preserve performance—replay scores fell below historical scores in 8 of 18 method–task pairs.

G1 (Held-out instances): 16 of 18 method–task pairs showed positive HCI. HyperAgents had the highest HCI on Farm, Combat, and Astray; AHE on Sweepmine, Hero, and MC.

G2 (Harder instances): Gains at training difficulty do not necessarily extend. ACE on Astray improved on G1 but declined on G2. Rankings reversed: on Farm, HyperAgents beat AHE at training difficulty but trailed on Hard held-out sets.

G3 (Within-category transfer): Transfer can help or hurt depending on direction. All methods improved from Hero to Sweepmine but degraded in reverse. HyperAgents reached 109.2% HCI from Combat to Farm, exceeding direct refinement (100.0%).

G4 (Cross-category transfer): Five of eight directions had positive HCI, but improvements were inconsistent.

Representative Results Table (G0–G3)

Task	Method	G0 HCI (%)	G1 HCI (%)	G2 HCI (%)	G3 HCI (%)
Farm	ACE	-10.2	+81.3	+14.6	+7.0
Farm	AHE	+58.7	-33.8	-1.8	+15.5
Farm	Hyper	+28.0	+100.0	+19.3	-0.1
Combat	AHE	+93.8	+99.2	+88.4	-24.3
Combat	Hyper	+62.5	+100.0	+100.0	+109.2
MC	AHE	-179.4	+43.4	N/A	+0.0

Trajectory Analysis Findings

Task-specific refinement coexists with transferable gains: Task-specific opinions predominate in all methods; category-general/general opinions make up 14.2% (ACE), 7.8% (AHE), and 36.5% (HyperAgents) of retained opinions.
Refinement emphasizes fixing observed problems: Reasoning and planning appear in 61.2% and 49.6% of retained opinions versus 11.0% for exploration.
Main-experiment changes concentrate on task agent: 87.9% of retained opinions target prompts/memory; 12.1% target execution harness; none modify the improver.
Active revision correlates with better outcomes: ACE, AHE, and HyperAgents revise 14.2%, 30.6%, and 40.2% of recorded opinions respectively, with AHE and HyperAgents generally achieving better performance.

Task Scheduling Results

Schedule	Combat Medium	Combat Hard	Hero Medium	Hero Hard
Random medium	1.433	1.314	0.085	0.082
Fixed medium instance	0.772	0.507	0.100	0.017
Mixed difficulty	1.388	1.195	0.145	0.037
Increasing difficulty	1.332	1.080	0.152	0.024

Execution Harness Refinement

HyperDSH variants (applying HyperAgent's improvement to DeepSeek Harness) all outperformed the original DeepSeek Harness (0.751) on Combat Medium, with the DeepSeek variant best at 1.326, though still below HyperAgents at 1.433. When explicitly allowing improver self-evolution, Meta opinions accounted for 20.5–45.2% of retained opinions.

Theoretical and Practical Implications

Theoretical Contributions

Unified generalization taxonomy: The G0–G4 ladder provides a systematic framework for distinguishing self-improvement (persistent across unseen instances/difficulties/tasks) from task fitting (confined to training instances), addressing a critical gap in RSI evaluation.
Capability-oriented task organization: Categorizing tasks by primary capability demands (exploration, reasoning, planning) enables fine-grained attribution of improvement to specific cognitive functions.
Distinguishing improvement types: The framework formalizes the distinction between genuine self-improvement and task fitting, providing theoretical clarity for evaluating progress toward AGI.

Practical Implications

Benchmark design: Template- and difficulty-controlled task generation enables scalable, sustained evaluation as agents evolve.
Method selection guidance: The finding that "similar final scores can conceal distinct evolution patterns" implies that practitioners should evaluate RSI methods across the full generalization ladder rather than relying on aggregate metrics.
Refinement strategy insights: The observation that more active revision correlates with better outcomes suggests that RSI methods should emphasize revising existing strategies, not just adding new ones.
Task scheduling: Random medium-difficulty training generally outperforms fixed-instance or increasing-difficulty schedules, informing practical training design.

Conclusion

Main takeaways:

SelfImprove-Bench provides a scalable, capability-oriented evaluation framework for RSI systems with a generalization ladder spanning instances, difficulty levels, tasks, and capabilities. The evaluation of three representative RSI methods reveals that refinement gains are inconsistent across generalization levels, with relative rankings reversing on harder instances and different task types. The framework's fine-grained trace analysis enables attribution of improvement to specific modification patterns, revealing that current methods focus predominantly on fixing observed problems through prompt/memory modifications rather than exploring alternatives or self-modifying their improvement mechanisms.

Future directions:

Expanding the framework to additional capabilities and real-world tasks beyond games
Investigating why certain RSI methods fail to generalize across capability boundaries
Exploring mechanisms to encourage meta-level self-modification and exploration-oriented refinement
Developing methods to reduce duplicate opinions and improve revision efficiency in RSI systems