S³Gym: Can LLMs Turn Self-Testing and Self-Judging into Self-Improvement?

Summary (Overview)

• New Benchmark: Introduces S³Gym (Self-Testing, Self-Judging, and Self-Improvement Gym), an interactive benchmark that evaluates whether LLMs can improve their own future behavior through environmental experience, rather than treating them as fixed policies.
• Three Coupled Capabilities: Decomposes experience-driven learning into Self-Testing (exploring strategies), Self-Judging (evaluating outcomes), and Self-Improvement (converting experience into better future decisions).
• Unified Evaluation Protocol: Separates permissive exploration from strict held-out evaluation across seven text-based games (Chess, Minesweeper, Nullify, Tetris, Snake, PvZ, Trust Evolution) with executable environment verifiers.
• Three Experience Pathways Compared: Evaluates History ICL (raw trajectory conditioning), Summary Memory (compressed rules), and parameter Training (SFT on self-generated trajectories) under the same interaction budget.
• Key Finding: Self-improvement is neither automatic nor uniform—the most effective pathway depends strongly on task structure, and accurate self-judging alone is insufficient for reliable improvement.

Introduction and Theoretical Foundation

Background and Motivation

LLMs are increasingly deployed as autonomous agents that interact with external environments through repeated observation–reasoning–action–feedback loops. However, existing agent benchmarks evaluate models as fixed policies, answering only "how capable is the model at the time of evaluation?" rather than whether the model can use its own past interactions to improve future behavior.

Theoretical Basis

The paper draws on experiential learning theory:

"Theories of experiential learning and reflective inquiry suggest that experience becomes useful only when it is actively examined, abstracted, and tested again."

Key theoretical foundations include:

• Dewey and Kolb: Transformation of experience into knowledge through reflection and reapplication
• Popper: Progress as iterative exposure of conjectures to tests that reveal error
• The principle that collecting trajectories alone does not guarantee learning

The Three Capabilities

The authors decompose experience-driven learning into three interdependent capabilities:

1. Self-Testing: The agent explores strategies and gathers diagnostic evidence
2. Self-Judging: The agent evaluates actions, outcomes, and their potential reusability
3. Self-Improvement: The resulting experience alters future decisions

Levels of Experience Integration

Experience can be incorporated at different persistence levels:

• Short-term (in-context): Direct conditioning on previous observations, actions, and scores
• Intermediate (external memory): Compression into natural-language rules, as in Reflexion and Voyager
• Long-term (training): Internalization through parameter updates, following STaR, ReST, and Re-ReST

Methodology

Formal Framework

One self-improvement cycle contains five stages:

$$\mathcal{R}_{t}^{(p)} = \mathrm{Explore} \to \mathrm{Judge} \to \mathrm{Consolidate} \to \mathrm{Update}_{p} \to \mathrm{Evaluate}, \tag{1}$$

where $t$ is the cycle index and $p \in \{\text{History}, \text{Memory}, \text{Training}\}$ denotes the improvement pathway.

Interaction Protocol

Each episode is initialized as:

$$O_{x,0} = \mathrm{Reset}(\mathrm{seed}_{x}, c_{x}), \quad H_{x,0} = \varnothing, \tag{2-3}$$

where $c_x$ denotes the game configuration. At each step, the model jointly produces an action and a self-judged immediate score:

$$(a_{x,i}, s_{x,i}) = \pi_{\theta_t}(O_{x,i}, H_{x,i}; Z_{t}^{(p)}), \tag{4}$$

The environment then returns:

$$(O_{x,i+1}, F_{x,i}, r_{x,i}, d_{x,i}) = \mathrm{Env}(O_{x,i}, a_{x,i}), \tag{5}$$

Critical design choice: The verifier-computed reward $r_{x,i}$ is not exposed to the agent during exploration—only the observable feedback $F_{x,i}$ is. This separation allows comparison of self-judgments against ground truth.

Experience Consolidation

History Context (direct serialization):

$$C_{t+1} = \mathrm{Serialize}(C_{t}, \mathcal{T}_{t}^{\mathrm{exp}}, \mathcal{S}_{t}^{\mathrm{exp}}), \tag{8}$$

Summary Memory (model-compressed experience):

$$M_{t+1} = \mathrm{Summarize}(M_{t}, \mathcal{T}_{t}^{\mathrm{exp}}, \mathcal{S}_{t}^{\mathrm{exp}}) = (R_{t}^{\mathrm{retain}}, R_{t}^{\mathrm{avoid}}, D_{t}^{\mathrm{next}}), \tag{9-10}$$

The summary contains strategies to retain, mistakes to avoid, and concrete directions for the next cycle.

Evaluation Metrics

Three complementary metrics are reported:

$$\mathrm{Avg}_{m,p,g} = \frac{1}{K+1}\sum_{k=0}^{K} y_{m,p,g,k}, \tag{18}$$ $$\operatorname{Max}_{m,p,g} = \max_{0 \leq k \leq K} y_{m,p,g,k}, \tag{19}$$ $$\mathrm{AUC}^{+}_{m,p,g} = \int_{0}^{30} \max(0, y_{m,p,g}(x) - y_{m,p,g,0})\,dx, \tag{20}$$

Game Configurations

Empirical Validation / Results

Main Results (Context-Level Pathways)

Key findings from Table 4:

• No universal winner: Self-improvement potential is jointly determined by base model, experience pathway, and environment
• History ICL: Gemini-3.5-Flash leads Chess, Snake, and Trust Evolution; GPT-5.5 leads Minesweeper, Nullify, and Tetris
• Summary Memory: GPT-5.5 becomes strongest on Chess, Nullify, and Tetris; Gemini-3.5-Flash remains best on PvZ and Trust Evolution

Summary Memory vs. History ICL is task-dependent:

• Summary helps when experience compresses into reusable rules: Gemini-2.5-Flash's Minesweeper AUC⁺ rises from 0.000 (ICL) to 7.794 (Summary); GPT-5.5's Chess AUC⁺ rises from 0.474 to 16.840
• Summary hurts when state-contingent details matter: GPT-5.5's PvZ AUC⁺ drops from 548.499 to 33.219; Gemini-3.5-Flash drops from 12.280 to 0.000 on Chess

RQ1: Can Parameter Training Enable Self-Improvement?

Evaluation of Qwen3-8B over 20 training checkpoints reveals:

• Substantial improvement on Trust Evolution: Score rises from 0 to max 30, with AUC⁺ = 163.5
• Limited/transient gains on Chess and Snake: Occasional successes (Snake reaches 1 at epochs 8, 12, 15) but inconsistent retention
• No improvement on Minesweeper, Nullify, Tetris: Scores remain zero throughout
• Severe negative transfer on PvZ: Score drops from 23 to 6 persistently—parameter updates can overwrite previously effective behavior

RQ2: Is Agent Self-Judging Reliable?

Step-level reliability (Table 5, 116,117 transitions across 98 runs):

Key findings:

• Self-Judging is only partially reliable—high binary agreement often masks poor calibration (PvZ has 0.881 agreement but 0.882 NMAE)
• Chess and Trust are the clearest failures: near-random agreement, high over-confidence—reward depends on hidden dynamics or exact multi-object state prediction
• Judgment–improvement coupling is near zero: $\rho(A,g) = -0.010$ and $\rho(-E,g) = -0.018$ across the full benchmark
• Accurate local scoring does not reliably produce improvement at the next evaluation checkpoint

RQ3: Do Score-Conditioned Summaries Produce Effective Iteration Directions?

Successful summaries translate payoff feedback into executable policies:

• GPT-5.5/Trust: "Always defect when the observed payoff structure consistently favors defection" (ΔNABA = +66.89)
• o3-mini/Tetris: "Jointly inspect current and next blocks, test alternative rotations, prioritize line-clearing" (ΔNABA = +14.42)

Failure summaries remain too generic:

• GPT-4.1/Minesweeper: "Begin from corners, expand zero regions, flag only certain mines" (ΔNABA = −22.09)—generic heuristics don't substitute for exact constraint propagation
• GPT-4o/Snake: "Move toward food while avoiding walls" (ΔNABA = −22.00)—omits precise geometry required for safe navigation

Theoretical and Practical Implications

Theoretical Significance

1. Self-improvement is not automatic: The paper provides systematic evidence that experience alone does not produce improvement—it must be actively tested, judged, and transformed into transferable policies.

2. The judgment–improvement gap: The near-zero correlation between judgment quality and subsequent improvement reveals a critical bottleneck: recognizing successful actions is insufficient; agents must also transform feedback into executable and transferable policies.

3. Task structure determines pathway efficacy: Environments admitting compact causal abstractions (Tetris, Trust, Nullify) benefit from summary memory, while reactive control games (Snake, PvZ, Minesweeper) require state-contingent details preserved in raw history.

Practical Implications

1. Benchmark design: S³Gym provides a unified framework for diagnosing where self-improvement breaks down—whether in experience generation, interpretation, or incorporation.

2. Agent architecture guidance: Future self-improving agents need better:

   • Judgment calibration (especially for hidden-dynamics tasks)
   • Memory selection mechanisms (when to compress vs. retain raw detail)
   • Trajectory filtering for training (to prevent negative transfer)

3. Risk warning for training-based improvement: Parameter training can produce severe negative transfer (PvZ dropped from 23 to 6), suggesting that self-generated trajectories require careful filtering before SFT.

Conclusion

S³Gym turns self-improvement from a broad claim into a measurable agent capability. The benchmark's separation of relaxed exploration from stricter evaluation, combined with support for both context-level and training-level trajectory reuse, enables systematic diagnosis of the experience-to-improvement loop.

Main takeaways:

• Summary memory helps where compact rules transfer; direct history is often better in reactive control games
• Training on self-generated trajectories remains unstable in current implementations
• Future self-improving agents need better judgment calibration, memory selection, and trajectory filtering—not just more interaction data

The findings suggest that the path from experience to reliable self-improvement requires not only the ability to generate and judge experience, but critically, the ability to transform feedback into executable and transferable policies—a capability that current LLMs possess only partially and inconsistently across tasks.