# S3Gym: Can LLMs Turn Self-Testing and Self-Judging into Self-Improvement?

> Self-improvement in LLMs is not automatic; its success depends on task structure, and accurate self-judging alone does not guarantee reliable improvement.

- **Source:** [arXiv](https://arxiv.org/abs/2608.31100)
- **Published:** 2026-09-12
- **Permalink:** https://picx.dev/p/Q7OhiT
- **Whiteboard:** https://picx.dev/p/Q7OhiT/image

## Summary

# S³Gym: Can LLMs Turn Self-Testing and Self-Judging into Self-Improvement?

## Summary (Overview)

- **New Benchmark**: Introduces S³Gym (Self-Testing, Self-Judging, and Self-Improvement Gym), an interactive benchmark that evaluates whether LLMs can improve their own future behavior through environmental experience, rather than treating them as fixed policies.
- **Three Coupled Capabilities**: Decomposes experience-driven learning into Self-Testing (exploring strategies), Self-Judging (evaluating outcomes), and Self-Improvement (converting experience into better future decisions).
- **Unified Evaluation Protocol**: Separates permissive exploration from strict held-out evaluation across seven text-based games (Chess, Minesweeper, Nullify, Tetris, Snake, PvZ, Trust Evolution) with executable environment verifiers.
- **Three Experience Pathways Compared**: Evaluates History ICL (raw trajectory conditioning), Summary Memory (compressed rules), and parameter Training (SFT on self-generated trajectories) under the same interaction budget.
- **Key Finding**: Self-improvement is neither automatic nor uniform—the most effective pathway depends strongly on task structure, and accurate self-judging alone is insufficient for reliable improvement.

## Introduction and Theoretical Foundation

### Background and Motivation

LLMs are increasingly deployed as autonomous agents that interact with external environments through repeated observation–reasoning–action–feedback loops. However, existing agent benchmarks evaluate models as **fixed policies**, answering only *"how capable is the model at the time of evaluation?"* rather than whether the model can use its own past interactions to improve future behavior.

### Theoretical Basis

The paper draws on experiential learning theory:

> "Theories of experiential learning and reflective inquiry suggest that experience becomes useful only when it is actively examined, abstracted, and tested again."

Key theoretical foundations include:
- **Dewey and Kolb**: Transformation of experience into knowledge through reflection and reapplication
- **Popper**: Progress as iterative exposure of conjectures to tests that reveal error
- The principle that *collecting trajectories alone does not guarantee learning*

### The Three Capabilities

The authors decompose experience-driven learning into three interdependent capabilities:
1. **Self-Testing**: The agent explores strategies and gathers diagnostic evidence
2. **Self-Judging**: The agent evaluates actions, outcomes, and their potential reusability
3. **Self-Improvement**: The resulting experience alters future decisions

### Levels of Experience Integration

Experience can be incorporated at different persistence levels:
- **Short-term (in-context)**: Direct conditioning on previous observations, actions, and scores
- **Intermediate (external memory)**: Compression into natural-language rules, as in Reflexion and Voyager
- **Long-term (training)**: Internalization through parameter updates, following STaR, ReST, and Re-ReST

## Methodology

### Formal Framework

One self-improvement cycle contains five stages:

$$
\mathcal{R}_{t}^{(p)} = \mathrm{Explore} \to \mathrm{Judge} \to \mathrm{Consolidate} \to \mathrm{Update}_{p} \to \mathrm{Evaluate}, \tag{1}
$$

where $t$ is the cycle index and $p \in \{\text{History}, \text{Memory}, \text{Training}\}$ denotes the improvement pathway.

### Interaction Protocol

Each episode is initialized as:

$$
O_{x,0} = \mathrm{Reset}(\mathrm{seed}_{x}, c_{x}), \quad H_{x,0} = \varnothing, \tag{2-3}
$$

where $c_x$ denotes the game configuration. At each step, the model jointly produces an action and a self-judged immediate score:

$$
(a_{x,i}, s_{x,i}) = \pi_{\theta_t}(O_{x,i}, H_{x,i}; Z_{t}^{(p)}), \tag{4}
$$

The environment then returns:

$$
(O_{x,i+1}, F_{x,i}, r_{x,i}, d_{x,i}) = \mathrm{Env}(O_{x,i}, a_{x,i}), \tag{5}
$$

**Critical design choice**: The verifier-computed reward $r_{x,i}$ is *not exposed to the agent* during exploration—only the observable feedback $F_{x,i}$ is. This separation allows comparison of self-judgments against ground truth.

### Experience Consolidation

**History Context** (direct serialization):

$$
C_{t+1} = \mathrm{Serialize}(C_{t}, \mathcal{T}_{t}^{\mathrm{exp}}, \mathcal{S}_{t}^{\mathrm{exp}}), \tag{8}
$$

**Summary Memory** (model-compressed experience):

$$
M_{t+1} = \mathrm{Summarize}(M_{t}, \mathcal{T}_{t}^{\mathrm{exp}}, \mathcal{S}_{t}^{\mathrm{exp}}) = (R_{t}^{\mathrm{retain}}, R_{t}^{\mathrm{avoid}}, D_{t}^{\mathrm{next}}), \tag{9-10}
$$

The summary contains strategies to retain, mistakes to avoid, and concrete directions for the next cycle.

### Evaluation Metrics

Three complementary metrics are reported:

$$
\mathrm{Avg}_{m,p,g} = \frac{1}{K+1}\sum_{k=0}^{K} y_{m,p,g,k}, \tag{18}
$$

$$
\operatorname{Max}_{m,p,g} = \max_{0 \leq k \leq K} y_{m,p,g,k}, \tag{19}
$$

$$
\mathrm{AUC}^{+}_{m,p,g} = \int_{0}^{30} \max(0, y_{m,p,g}(x) - y_{m,p,g,0})\,dx, \tag{20}
$$

### Game Configurations

| Game | Exploration | Evaluation | Max Steps |
|------|-------------|------------|-----------|
| Chess | 15×15 board, 12 pieces, 7-step history | Same rules, 22 pieces | 5 |
| Minesweeper | 9×9 board, 10–20 mines, two extra lives | First mine hit terminates | 64 |
| Nullify | 3–7 construction steps | 5–10 construction steps | 50 |
| PvZ | 3×7 battlefield | 5×6 battlefield | 64 |
| Snake | Collisions treated as no-ops | Collisions terminate episode | 64 |
| Tetris | 10×10 board | 8×8 board | 64 |
| Trust Evolution | Biased easier opponent sampling | Uniform opponent sampling | 10 |

## Empirical Validation / Results

### Main Results (Context-Level Pathways)

**Key findings from Table 4:**

- **No universal winner**: Self-improvement potential is jointly determined by base model, experience pathway, and environment
- **History ICL**: Gemini-3.5-Flash leads Chess, Snake, and Trust Evolution; GPT-5.5 leads Minesweeper, Nullify, and Tetris
- **Summary Memory**: GPT-5.5 becomes strongest on Chess, Nullify, and Tetris; Gemini-3.5-Flash remains best on PvZ and Trust Evolution

**Summary Memory vs. History ICL is task-dependent:**
- **Summary helps** when experience compresses into reusable rules: Gemini-2.5-Flash's Minesweeper AUC⁺ rises from 0.000 (ICL) to 7.794 (Summary); GPT-5.5's Chess AUC⁺ rises from 0.474 to 16.840
- **Summary hurts** when state-contingent details matter: GPT-5.5's PvZ AUC⁺ drops from 548.499 to 33.219; Gemini-3.5-Flash drops from 12.280 to 0.000 on Chess

### RQ1: Can Parameter Training Enable Self-Improvement?

Evaluation of Qwen3-8B over 20 training checkpoints reveals:
- **Substantial improvement on Trust Evolution**: Score rises from 0 to max 30, with AUC⁺ = 163.5
- **Limited/transient gains on Chess and Snake**: Occasional successes (Snake reaches 1 at epochs 8, 12, 15) but inconsistent retention
- **No improvement on Minesweeper, Nullify, Tetris**: Scores remain zero throughout
- **Severe negative transfer on PvZ**: Score drops from 23 to 6 persistently—parameter updates can overwrite previously effective behavior

### RQ2: Is Agent Self-Judging Reliable?

**Step-level reliability** (Table 5, 116,117 transitions across 98 runs):

| Game | Agreement | NMAE | Over-conf. | Under-conf. |
|------|-----------|------|------------|-------------|
| Chess | 0.496 | 0.141 | 0.365 | 0.139 |
| Minesweeper | 0.820 | 0.524 | 0.062 | 0.118 |
| Nullify | 0.827 | 0.056 | 0.170 | 0.004 |
| PvZ | 0.881 | 0.882 | 0.032 | 0.087 |
| Snake | 0.879 | 0.121 | 0.110 | 0.011 |
| Tetris | 0.846 | 0.041 | 0.151 | 0.002 |
| Trust | 0.665 | 0.392 | 0.291 | 0.044 |

**Key findings:**
- Self-Judging is only **partially reliable**—high binary agreement often masks poor calibration (PvZ has 0.881 agreement but 0.882 NMAE)
- Chess and Trust are the clearest failures: near-random agreement, high over-confidence—reward depends on hidden dynamics or exact multi-object state prediction
- **Judgment–improvement coupling is near zero**: $\rho(A,g) = -0.010$ and $\rho(-E,g) = -0.018$ across the full benchmark
- Accurate local scoring does **not** reliably produce improvement at the next evaluation checkpoint

### RQ3: Do Score-Conditioned Summaries Produce Effective Iteration Directions?

| Game | Pairs | Summary wins | History wins | Ties | Mean ΔNABA |
|------|-------|-------------|--------------|------|------------|
| Chess | 7 | 3 | 3 | 1 | +3.37 |
| Minesweeper | 7 | 3 | 4 | 0 | −1.12 |
| Nullify | 7 | 4 | 2 | 1 | +2.79 |
| PvZ | 7 | 3 | 4 | 0 | −2.11 |
| Snake | 7 | 3 | 4 | 0 | −0.81 |
| Tetris | 7 | 5 | 1 | 1 | +6.20 |
| Trust | 7 | 4 | 3 | 0 | +8.43 |

**Successful summaries** translate payoff feedback into executable policies:
- GPT-5.5/Trust: "Always defect when the observed payoff structure consistently favors defection" (ΔNABA = +66.89)
- o3-mini/Tetris: "Jointly inspect current and next blocks, test alternative rotations, prioritize line-clearing" (ΔNABA = +14.42)

**Failure summaries** remain too generic:
- GPT-4.1/Minesweeper: "Begin from corners, expand zero regions, flag only certain mines" (ΔNABA = −22.09)—generic heuristics don't substitute for exact constraint propagation
- GPT-4o/Snake: "Move toward food while avoiding walls" (ΔNABA = −22.00)—omits precise geometry required for safe navigation

## Theoretical and Practical Implications

### Theoretical Significance

1. **Self-improvement is not automatic**: The paper provides systematic evidence that experience alone does not produce improvement—it must be actively tested, judged, and transformed into transferable policies.

2. **The judgment–improvement gap**: The near-zero correlation between judgment quality and subsequent improvement reveals a critical bottleneck: *recognizing successful actions is insufficient; agents must also transform feedback into executable and transferable policies*.

3. **Task structure determines pathway efficacy**: Environments admitting compact causal abstractions (Tetris, Trust, Nullify) benefit from summary memory, while reactive control games (Snake, PvZ, Minesweeper) require state-contingent details preserved in raw history.

### Practical Implications

1. **Benchmark design**: S³Gym provides a unified framework for diagnosing where self-improvement breaks down—whether in experience generation, interpretation, or incorporation.

2. **Agent architecture guidance**: Future self-improving agents need better:
   - Judgment calibration (especially for hidden-dynamics tasks)
   - Memory selection mechanisms (when to compress vs. retain raw detail)
   - Trajectory filtering for training (to prevent negative transfer)

3. **Risk warning for training-based improvement**: Parameter training can produce severe negative transfer (PvZ dropped from 23 to 6), suggesting that self-generated trajectories require careful filtering before SFT.

## Conclusion

S³Gym turns self-improvement from a broad claim into a measurable agent capability. The benchmark's separation of relaxed exploration from stricter evaluation, combined with support for both context-level and training-level trajectory reuse, enables systematic diagnosis of the experience-to-improvement loop.

**Main takeaways:**
- Summary memory helps where compact rules transfer; direct history is often better in reactive control games
- Training on self-generated trajectories remains unstable in current implementations
- Future self-improving agents need better judgment calibration, memory selection, and trajectory filtering—not just more interaction data

The findings suggest that the path from experience to reliable self-improvement requires not only the ability to generate and judge experience, but critically, the ability to *transform feedback into executable and transferable policies*—a capability that current LLMs possess only partially and inconsistently across tasks.

---

_Markdown view of https://picx.dev/p/Q7OhiT, served by PicX — AI-generated visual whiteboard summaries of research papers._