Full text not available for this paper
When Can Self-Play Fix a False Belief? Answer Keys, Pass Bands and Replacement Rules in an Execution-Grounded Loop
Authors: Nirdesh Chauhan, Dushyant Rajput, Siddharth Kosaraju (AltSlate Labs LLP)
Summary (Overview)
- Core question: Can self-play loops with code-execution-grounded answer keys actually correct false beliefs in language models, and under what conditions?
- Key finding: Executing the generator's own code produces answer keys that inherit the generator's misreadings, making correction impossible (truth rate never exceeds 0.03). A "decorrelated key" from an unplanted teacher model raises truth rates to 0.23–0.58.
- Three gates govern correction: generation (G1), key (G2), and pass band (G3)—all must pass for a belief to receive direct corrective gradient.
- Replacement rules: Even when corrections occur, models learn shifted rules that are wrong on neighboring items where the old belief was correct (e.g., overcounting after the "at most" fix, reaching 0.18–0.38 of control samples).
- Generator limitations: Planted generators fail to produce the very cases needed for correction (0 of 10,924 tasks with absent limits), and both planted and unplanted generators rewrite task requests into specifications, defeating key decorrelation.
Introduction and Theoretical Foundation
Background and Motivation
Self-play loops where a model writes tasks, derives answer keys, and trains on matching solutions are increasingly common in AI self-improvement. Methods like Absolute Zero, Self-Challenging Agents, and STP rely on code executors or proof checkers for "grounding." The assumption is that external execution escapes the limitations of intrinsic rewards, which merely sharpen the model's initial distribution.
The fundamental problem: An executor computes what code does, not what code should do. When the model writes the program whose output becomes the answer key, the key inherits the beliefs that shaped the program.
Theoretical Framework: Three Gates
For a false belief to receive a direct corrective gradient, a trained task must pass three gates:
- G1 (Generation): The generator writes a task that touches the belief, where
- G2 (Key): The key differs from the belief on that task, ; correction toward truth needs
- G3 (Passage): The solver sometimes produces the key, (out of 8 samples)
When G2 fails because , training reinforces the belief (label-free reward conserves errors). When G3 fails, even a correct key is invisible.
The Four Targeted Beliefs
| ID | Belief | Truth | Touching Cases | Neighbors () |
|---|---|---|---|---|
| R1 | round(x.5) rounds half away from zero | half to even | even .5 ties; truth is one smaller in magnitude | odd .5 ties |
| R2 | // and % truncate toward zero | floor | mixed signs, inexact; truth = belief's remainder or belief | exact divisions with mixed signs |
| S1 | "the -th smallest" is 0-indexed | 1-indexed | every call; truth is one position earlier | none |
| S2 | "at most " means | limit is in the list; truth also counts copies of | absent from list |
R1 and R2 are natural beliefs in the base model; S1 and S2 are planted via fine-tuning.
Methodology
Model and Loop
- Model: Qwen3-4B-Base with raw-text few-shot prompts, bf16 weights, vLLM for generation, LoRA (rank 32, α=64) for training
- Loop: Expert iteration (rejection-sampling fine-tuning, as in STaR and ReST EM)
Each round has five steps:
- Generate: 3,000 task requests (belief topics with probability 0.15 each, otherwise neutral topics)
- Key: Each task keyed according to experimental arm
- Solve: 8 samples per task with chain-of-thought (temperature 0.7)
- Filter: Tasks solved 1–7 times of 8 kept (up to 2 matching solutions each)
- Train: One epoch of SFT (learning rate , batch 16)
Experimental Arms
| Arm | Answer Key | Shares the Belief? |
|---|---|---|
| atrust | Generator's stated output, unchecked | partly |
| agate | Stated output, kept only if execution agrees | no (disagreements dropped) |
| exec | Execution of the program | no |
| ctrl | Execution; neutral topics only, touching tasks dropped | no exposure |
| sexec | Execution of generator's own implementation | yes, by construction |
| sclean | Decorrelation: execution of unplanted base model's implementation | no |
| sctrl | As sexec; neutral topics only | no exposure |
Planting Protocol
- S1 and S2 planted via LoRA SFT with question-answer examples stating the belief and generator-format examples implementing it
- Strong planting: 500 examples/belief/format, 2 epochs (belief rates: S1 1.00, S2 0.92)
- Weak planting: 100 examples, 1 epoch (belief rates: S1 0.99, S2 0.75)
- Every S1 program uses
sorted(xs)[k]; every S2 example draws the limit from the list
Probes
- Diagnostic items: where belief and truth differ (100 per belief)
- Controls/neighbors: where belief and truth agree (50 per belief for R1, R2, S2; none for S1)
- Replacement rule probes: 300 exact .5 ties (150 even, 150 odd); 200 exact divisions with negative operand
Empirical Validation / Results
1. Correlated vs. Decorrelated Keys (S2 Results at Round 3)
| Run | Planted | sexec | sctrl | sclean |
|---|---|---|---|---|
| Strong, seed 0 | 0.03 | 0.00 | 0.02 | 0.58 [0.50, 0.67] |
| Strong, seed 1 | 0.03 | 0.01 | 0.02 | 0.28 [0.20, 0.36] |
| Weak, seed 0 | 0.05 | 0.02 | 0.04 | 0.23 [0.15, 0.30] |
| Weak, seed 1 | 0.05 | 0.03 | 0.04 | 0.29 [0.21, 0.38] |
Table shows truth rate: belief rate: other rate. Only the decorrelated key (sclean) moves the model toward truth in all four runs.
- sexec key encodes the misreading: The generator's executed key equals the count on all 431 checkable S2 tasks at round 0 (strong run), 430/430 (weak), 441/441 (round 2)
- sclean is not perfect: Matches count on 88–94% of tasks; mismatches occur when the base model treats integers as objects with a
.priceattribute - Six rounds of sclean (strong seed 0) reach S2 truth 0.88 [0.83, 0.93], belief 0.03
2. The Pass Band (S1 Results)
S1 barely moves under sclean despite a correct key on 395/428 tasks. The blocker is G3: the solver almost never produces the 1-indexed answer, so few tasks are admitted:
| sclean run | S2 tasks admitted (rounds 0,1,2,...) | S2 truth, round 3 | S1 tasks admitted |
|---|---|---|---|
| Strong, seed 0 | 9, 38, 76, 114, 83, 31 | 0.58 (0.88 at r6) | 1, 3, 14, 15, 12, 28 |
| Strong, seed 1 | 17, 25, 24 | 0.28 | 4, 2, 1 |
| Weak, seed 0 | 24, 52, 73 | 0.23 | 3, 1, 2 |
| Weak, seed 1 | 31, 82, 82 | 0.29 | 4, 5, 5 |
Key insight: The block is "leaky"—admissions grow as the belief loosens. Round-0 admissions did not predict speed (fewest admissions, 9, fixed fastest at 0.58).
3. Execution Keys and Natural Beliefs (Runtime Half)
| Arm, round 3 | R1 belief (seed 0/1) | R2 belief (seed 0/1) |
|---|---|---|
| base | 0.61 / 0.62 | 0.41 / 0.42 |
| exec | 0.04 / 0.04 | 0.10 / 0.06 |
| agate | 0.16 / 0.19 | 0.05 / 0.17 |
| atrust | 0.67 / 0.42 | 0.36 / 0.23 |
| ctrl | 0.37 / 0.56 | 0.33 / 0.39 |
Execution keys sharply reduce both runtime beliefs. An unchecked key (atrust) gives a weaker, seed-dependent fix—not entrenchment.
4. Replacement Rules
S2: The boundary moves to the next value. After the "at most" fix, the model overcounts on control items (limit absent):
- Overcount rises with the fix: truth and overcount correlate at over 27 checkpoints (slope 0.45)
- Strong seed 0: overcount grows 0.07 → 0.38 (round 3) → 0.46 (round 6)
- Most common errors at round 3: +1 (95 times), +2 (38), −1 (19), +4 (10)
- Greedy control accuracy hides the overcount: 37–39/50 at round 3 (weak runs), falling to 23/50 in strong seed 0 by round 6
| Checkpoint | next value = | overcounts = truth + copies of next | ||
|---|---|---|---|---|
| planted (strong) | 0.14 | 0.08 | 0.00 | 0.81 |
| strong s0, r3 | 0.78 | 0.31 | 0.26 | 0.73 |
| strong s0, r6 | 0.86 | 0.35 | 0.50 | 0.77 |
Sources of the rule (cannot be separated):
- One-sided corrective cases (generator wrote only 4 of 10,924 tasks with limit absent)
- Selected reasoning: 11% of trained S2 rationales list a value above the limit and still reach the key; 29% begin by stating the belief
- Planted presupposition: every planting example had the limit in the list
R2: Add the divisor. On exact % with mixed signs (truth 0), models answer the divisor in 29–38% of samples (base: 10%). The rule "add when signs differ" costs % accuracy in seed 0 (exec 0.49, agate 0.54 vs. ctrl 0.80).
R1: Toward zero, in one seed only. Seed 0 exec rounds odd ties toward zero on 22% of samples (base 6%); seed 1 shows no rule (7%).
5. Generation: The Failed Remedy
The neighbor-topic audit revealed a critical failure:
- 2,061 of 2,061 checkable neighbor-topic tasks had the limit in the list—the generator never wrote a true anchor
- The generator moved the request into the specification: as a precondition the call violated ("where does not appear in xs") or as a redefinition ("the cap itself does not count")
- With a violated precondition, the specification no longer fixes the answer; with a redefinition, it fixes the belief's answer
- The key-agreement proxy failed silently: it marked 57–93 tasks per round as anchors, all of which were actually belief-keyed tasks with the limit in the list
The unplanted base model, in contrast, wrote limit-absent tasks in 79/150 (plain topics) and 46/170 (neighbor topics) of checkable calls—never omitting the limit is a planting artifact. But rewriting the request is not: the unplanted model also added preconditions/redefinitions (31 and 36 of 124, respectively).
Theoretical and Practical Implications
Theoretical Contributions
-
Three-gate framework: A systematic account of the direct route by which self-play corrects a belief (generation, key, pass band), with per-task counts for each gate.
-
Decorrelation is necessary but not sufficient: Using a second, independently written implementation as oracle (differential testing) works when versions differ in the belief under study, but fails when the generator rewrites the specification.
-
Replacement rules as underspecification: The corrective data underdetermine the learned rule. A probe containing only diagnostic items rewards every rule that moves away from the belief—models need neighbor probes.
-
The key implements the specification, not the intent: When the specification excludes the limit, every key that follows the specification agrees with the belief.
Practical Recommendations
- Decorrelate the key and audit it on known errors (use a model that does not share the error as key writer)
- Check generated inputs directly—read properties off inputs, not inferred from keys
- Watch the pass band—beliefs held near 100% get almost no direct gradient; exploration (more samples, hints, teacher demonstrations) may be needed
- Score fixes on neighbors—report error distributions where the old belief was right, not just belief rates where it was wrong
Conclusion
Self-play with code execution grounds a model in what code does, not what a task means when the model's misreading has reached its code. It cannot directly correct a belief for which the model almost never produces a correct sample. When correction does occur, it happens from cases that all lie on one side of the belief, with reasoning selected by final answer alone—allowing the model to learn a shifted rule instead of the truth.
Future directions:
- Test whether true neighbor tasks (injected via script or input filtering) remove replacement rules
- Explore key writers from different model families (correlated errors make this uncertain)
- Scale beyond the small loop (3–6 rounds, one 4B model) to production-scale self-play
- Add capability measures before/after planting and training
Key limitation: The remedy was never actually tested—the neighbor-topic runs contained no limit-absent tasks, leaving open whether training on true neighbors would eliminate the replacement rule.
Related papers
- From Repositories to Training Environments: Scaling Long-Horizon Coding Agents with Environment Evolution Loop
RepoFactory's proxy-driven environment evolution loop yields RepoHorizen-35B-A3B, a model improving coding-agent benchmarks by 22.6 points on NL2Repo and 8.5 points on average.
- Evaluating Governed LLM-Driven ML Experimentation: Lifecycle, Provenance, and Failure Metrics
Governed harnesses with append-only ledgers expose process failures invisible to task scores, achieving 100% provenance across 1,445 trials while ungoverned loops silently lose 86% of trials.
- Inside the Loop: A Mechanistic Study of Looped Transformers on Permutation Traversal
Looped transformers solve sequential algorithmic tasks untied models of equal effective depth cannot, by executing one algorithm step per iteration without intermediate supervision.