Full text not available for this paper

When Can Self-Play Fix a False Belief? Answer Keys, Pass Bands and Replacement Rules in an Execution-Grounded Loop

Authors: Nirdesh Chauhan, Dushyant Rajput, Siddharth Kosaraju (AltSlate Labs LLP)

Summary (Overview)

  • Core question: Can self-play loops with code-execution-grounded answer keys actually correct false beliefs in language models, and under what conditions?
  • Key finding: Executing the generator's own code produces answer keys that inherit the generator's misreadings, making correction impossible (truth rate never exceeds 0.03). A "decorrelated key" from an unplanted teacher model raises truth rates to 0.23–0.58.
  • Three gates govern correction: generation (G1), key (G2), and pass band (G3)—all must pass for a belief to receive direct corrective gradient.
  • Replacement rules: Even when corrections occur, models learn shifted rules that are wrong on neighboring items where the old belief was correct (e.g., overcounting after the "at most" fix, reaching 0.18–0.38 of control samples).
  • Generator limitations: Planted generators fail to produce the very cases needed for correction (0 of 10,924 tasks with absent limits), and both planted and unplanted generators rewrite task requests into specifications, defeating key decorrelation.

Introduction and Theoretical Foundation

Background and Motivation

Self-play loops where a model writes tasks, derives answer keys, and trains on matching solutions are increasingly common in AI self-improvement. Methods like Absolute Zero, Self-Challenging Agents, and STP rely on code executors or proof checkers for "grounding." The assumption is that external execution escapes the limitations of intrinsic rewards, which merely sharpen the model's initial distribution.

The fundamental problem: An executor computes what code does, not what code should do. When the model writes the program whose output becomes the answer key, the key inherits the beliefs that shaped the program.

Theoretical Framework: Three Gates

For a false belief to receive a direct corrective gradient, a trained task must pass three gates:

  • G1 (Generation): The generator writes a task that touches the belief, where b(t)≠y∗(t)b(t) \neq y^*(t)
  • G2 (Key): The key differs from the belief on that task, κ(t)≠b(t)\kappa(t) \neq b(t); correction toward truth needs κ(t)=y∗(t)\kappa(t) = y^*(t)
  • G3 (Passage): The solver sometimes produces the key, 1≤h(t)≤71 \leq h(t) \leq 7 (out of 8 samples)

When G2 fails because κ(t)=b(t)\kappa(t) = b(t), training reinforces the belief (label-free reward conserves errors). When G3 fails, even a correct key is invisible.

The Four Targeted Beliefs

IDBeliefTruthTouching CasesNeighbors (b=y∗b = y^*)
R1round(x.5) rounds half away from zerohalf to eveneven .5 ties; truth is one smaller in magnitudeodd .5 ties
R2// and % truncate toward zerofloormixed signs, inexact; truth = belief's remainder +b+ b or belief −1- 1exact divisions with mixed signs
S1"the kk-th smallest" is 0-indexed1-indexedevery call; truth is one position earliernone
S2"at most nn" means <n< n≤n\leq nlimit nn is in the list; truth also counts copies of nnnn absent from list

R1 and R2 are natural beliefs in the base model; S1 and S2 are planted via fine-tuning.

Methodology

Model and Loop

  • Model: Qwen3-4B-Base with raw-text few-shot prompts, bf16 weights, vLLM for generation, LoRA (rank 32, α=64) for training
  • Loop: Expert iteration (rejection-sampling fine-tuning, as in STaR and ReST EM)

Each round has five steps:

  1. Generate: 3,000 task requests (belief topics with probability 0.15 each, otherwise neutral topics)
  2. Key: Each task keyed according to experimental arm
  3. Solve: 8 samples per task with chain-of-thought (temperature 0.7)
  4. Filter: Tasks solved 1–7 times of 8 kept (up to 2 matching solutions each)
  5. Train: One epoch of SFT (learning rate 10−410^{-4}, batch 16)

Experimental Arms

ArmAnswer KeyShares the Belief?
atrustGenerator's stated output, uncheckedpartly
agateStated output, kept only if execution agreesno (disagreements dropped)
execExecution of the programno
ctrlExecution; neutral topics only, touching tasks droppedno exposure
sexecExecution of generator's own implementationyes, by construction
scleanDecorrelation: execution of unplanted base model's implementationno
sctrlAs sexec; neutral topics onlyno exposure

Planting Protocol

  • S1 and S2 planted via LoRA SFT with question-answer examples stating the belief and generator-format examples implementing it
  • Strong planting: 500 examples/belief/format, 2 epochs (belief rates: S1 1.00, S2 0.92)
  • Weak planting: 100 examples, 1 epoch (belief rates: S1 0.99, S2 0.75)
  • Every S1 program uses sorted(xs)[k]; every S2 example draws the limit from the list

Probes

  • Diagnostic items: where belief and truth differ (100 per belief)
  • Controls/neighbors: where belief and truth agree (50 per belief for R1, R2, S2; none for S1)
  • Replacement rule probes: 300 exact .5 ties (150 even, 150 odd); 200 exact divisions with negative operand

Empirical Validation / Results

1. Correlated vs. Decorrelated Keys (S2 Results at Round 3)

RunPlantedsexecsctrlsclean
Strong, seed 00.030.000.020.58 [0.50, 0.67]
Strong, seed 10.030.010.020.28 [0.20, 0.36]
Weak, seed 00.050.020.040.23 [0.15, 0.30]
Weak, seed 10.050.030.040.29 [0.21, 0.38]

Table shows truth rate: belief rate: other rate. Only the decorrelated key (sclean) moves the model toward truth in all four runs.

  • sexec key encodes the misreading: The generator's executed key equals the << count on all 431 checkable S2 tasks at round 0 (strong run), 430/430 (weak), 441/441 (round 2)
  • sclean is not perfect: Matches ≤\leq count on 88–94% of tasks; mismatches occur when the base model treats integers as objects with a .price attribute
  • Six rounds of sclean (strong seed 0) reach S2 truth 0.88 [0.83, 0.93], belief 0.03

2. The Pass Band (S1 Results)

S1 barely moves under sclean despite a correct key on 395/428 tasks. The blocker is G3: the solver almost never produces the 1-indexed answer, so few tasks are admitted:

sclean runS2 tasks admitted (rounds 0,1,2,...)S2 truth, round 3S1 tasks admitted
Strong, seed 09, 38, 76, 114, 83, 310.58 (0.88 at r6)1, 3, 14, 15, 12, 28
Strong, seed 117, 25, 240.284, 2, 1
Weak, seed 024, 52, 730.233, 1, 2
Weak, seed 131, 82, 820.294, 5, 5

Key insight: The block is "leaky"—admissions grow as the belief loosens. Round-0 admissions did not predict speed (fewest admissions, 9, fixed fastest at 0.58).

3. Execution Keys and Natural Beliefs (Runtime Half)

Arm, round 3R1 belief (seed 0/1)R2 belief (seed 0/1)
base0.61 / 0.620.41 / 0.42
exec0.04 / 0.040.10 / 0.06
agate0.16 / 0.190.05 / 0.17
atrust0.67 / 0.420.36 / 0.23
ctrl0.37 / 0.560.33 / 0.39

Execution keys sharply reduce both runtime beliefs. An unchecked key (atrust) gives a weaker, seed-dependent fix—not entrenchment.

4. Replacement Rules

S2: The boundary moves to the next value. After the "at most" fix, the model overcounts on control items (limit absent):

  • Overcount rises with the fix: truth and overcount correlate at r=0.95r = 0.95 over 27 checkpoints (slope 0.45)
  • Strong seed 0: overcount grows 0.07 → 0.38 (round 3) → 0.46 (round 6)
  • Most common errors at round 3: +1 (95 times), +2 (38), −1 (19), +4 (10)
  • Greedy control accuracy hides the overcount: 37–39/50 at round 3 (weak runs), falling to 23/50 in strong seed 0 by round 6
Checkpointnext value = n+1n+1n+2n+2≥n+3\geq n+3overcounts = truth + copies of next
planted (strong)0.140.080.000.81
strong s0, r30.780.310.260.73
strong s0, r60.860.350.500.77

Sources of the rule (cannot be separated):

  1. One-sided corrective cases (generator wrote only 4 of 10,924 tasks with limit absent)
  2. Selected reasoning: 11% of trained S2 rationales list a value above the limit and still reach the key; 29% begin by stating the belief
  3. Planted presupposition: every planting example had the limit in the list

R2: Add the divisor. On exact % with mixed signs (truth 0), models answer the divisor bb in 29–38% of samples (base: 10%). The rule "add bb when signs differ" costs % accuracy in seed 0 (exec 0.49, agate 0.54 vs. ctrl 0.80).

R1: Toward zero, in one seed only. Seed 0 exec rounds odd ties toward zero on 22% of samples (base 6%); seed 1 shows no rule (7%).

5. Generation: The Failed Remedy

The neighbor-topic audit revealed a critical failure:

  • 2,061 of 2,061 checkable neighbor-topic tasks had the limit in the list—the generator never wrote a true anchor
  • The generator moved the request into the specification: as a precondition the call violated ("where nn does not appear in xs") or as a redefinition ("the cap itself does not count")
  • With a violated precondition, the specification no longer fixes the answer; with a redefinition, it fixes the belief's answer
  • The key-agreement proxy failed silently: it marked 57–93 tasks per round as anchors, all of which were actually belief-keyed tasks with the limit in the list

The unplanted base model, in contrast, wrote limit-absent tasks in 79/150 (plain topics) and 46/170 (neighbor topics) of checkable calls—never omitting the limit is a planting artifact. But rewriting the request is not: the unplanted model also added preconditions/redefinitions (31 and 36 of 124, respectively).

Theoretical and Practical Implications

Theoretical Contributions

  1. Three-gate framework: A systematic account of the direct route by which self-play corrects a belief (generation, key, pass band), with per-task counts for each gate.

  2. Decorrelation is necessary but not sufficient: Using a second, independently written implementation as oracle (differential testing) works when versions differ in the belief under study, but fails when the generator rewrites the specification.

  3. Replacement rules as underspecification: The corrective data underdetermine the learned rule. A probe containing only diagnostic items rewards every rule that moves away from the belief—models need neighbor probes.

  4. The key implements the specification, not the intent: When the specification excludes the limit, every key that follows the specification agrees with the belief.

Practical Recommendations

  1. Decorrelate the key and audit it on known errors (use a model that does not share the error as key writer)
  2. Check generated inputs directly—read properties off inputs, not inferred from keys
  3. Watch the pass band—beliefs held near 100% get almost no direct gradient; exploration (more samples, hints, teacher demonstrations) may be needed
  4. Score fixes on neighbors—report error distributions where the old belief was right, not just belief rates where it was wrong

Conclusion

Self-play with code execution grounds a model in what code does, not what a task means when the model's misreading has reached its code. It cannot directly correct a belief for which the model almost never produces a correct sample. When correction does occur, it happens from cases that all lie on one side of the belief, with reasoning selected by final answer alone—allowing the model to learn a shifted rule instead of the truth.

Future directions:

  • Test whether true neighbor tasks (injected via script or input filtering) remove replacement rules
  • Explore key writers from different model families (correlated errors make this uncertain)
  • Scale beyond the small loop (3–6 rounds, one 4B model) to production-scale self-play
  • Add capability measures before/after planting and training

Key limitation: The remedy was never actually tested—the neighbor-topic runs contained no limit-absent tasks, leaving open whether training on true neighbors would eliminate the replacement rule.

Related papers