Recursive Self-ImprovementIssue 13Oct 31 – Nov 7, 2026

Self-Improvement Measurement Enters Statistical and Theoretical Foundations: Noise Bands, Adaptive Testing, and Computability Boundaries

Highlights

The main thread this issue is that self-improvement measurement is beginning to develop 'statistical and theoretical foundations': claimed gains are re-examined under three constraints—frozen rescoring noise bands, error budgets for adaptive testing, and computability boundaries for gate-proposal self-reference. These three lines advance the previous issue's survey on 'product claims vs. process claims' from audit protocols to provable measurement contracts.

On the measurement side, Trace Evolution Bench turns the often-overlooked 'frozen rescoring variance' into an independent measurement object: using four controls—locked held-out ids, validation-gated acceptance/rollback, four frozen no_evolve rescans, and per-seed paired excess—it separates the portion of endpoint Gain attributable to evaluation noise. DeepSeek's scores still move 7.5–12.5 percentage points across seeds with unchanged versions; seven units exceed their own noise band, but GLM/MiniMax do not reproduce, and no method exceeds its own band across all seeds. Which Self-Improvements Should We Trust? is the first to introduce the statistical theory of adaptive data analysis and multiple testing into the closed-loop evaluation layer: the Reuse framework returns only promotion decisions, explicitly allocates error budgets across all possible decision histories, controls three types of false promotion sources (within-round selection, repeated testing, and adaptive dependence), and provides theoretical guarantees for simultaneous validity and cumulative improvement lower bounds. On the theory side, The Self-Improvement Problem elevates the empirical assertion that 'evaluator hacking is inevitable' to a formal theorem: any fully computable gate cannot simultaneously satisfy reliability and non-conservatism under unbounded semantic behavior and self-referential candidates, and randomization, hiding, and stateful gates cannot circumvent this.

Component decomposition continues to advance. Evolving in Thought Space splits test-time training into two layers—guidance (policy) and executor (implementation)—updating only a small guidance model to propose high-level policy changes while freezing a strong executor model to implement full code. Control groups show that directly training the full solution generator is far worse than training the guidance (Polyomino 41.63 vs. 91.89). Test-Time Memory Evolution makes external memory evolution a direct alternative to weight-based test-time training: under the same discovery framework, it wins on all five math tasks, reduces FLOPs by 82.8–84.6%, and adding TTT to TTME actually worsens performance. Beneath the Surface decomposes four harness evolution methods into select/analyze/mutate/gate operators and empirically tests each operator; all sixteen evolution agents both gain and lose task coverage, and high-scoring parents do not yield the largest subsequent improvements. Scientist World turns the 'paper → executable environment' conversion into a human-out-of-the-loop scalable pipeline and introduces Frontier-Normalized Improvement as a cross-paper normalized improvement metric.

On the community side, METR's independent evaluation of Anthropic is the first third-party audit of a lab-disclosed self-improvement rate (AECI 1.9x acceleration), but most key evidence was redacted by the lab for confidentiality—this itself is the latest footnote to the practice of 'treating lab disclosures as data sources.' The karpathy/autoresearch integrity log discussion enumerates specific mechanisms by which AI-scientist closed-loop reports fabricate val_bpb improvements (skipping optimizers, short-circuiting metrics, unfingerprinted validation sets) and provides an approximately 30-line binding log fix; The Incumbent Did the Work systematically organizes 27 circle-stacking records, noting that multiple 2026 records differ by about 2.5e-7, one is amplified within validator tolerance, and questions the value of additional mechanisms like archives and test-time weight updates; ThinkingBox provides a large-scale failure taxonomy from 79,853 failed runs—about two-thirds of failures 'look complete' in transcription signals, and tool-use failures account for 79.9%.

Community and Dynamics

METR's independent evaluation of Anthropic advances the practice of 'treating lab disclosures as data sources' from unilateral self-reporting to third-party audit: METR, with high-privilege access, evaluates Anthropic's AECI acceleration (estimated 1.9x, 95% CI 1.5–2.8x) and attempts to attribute the acceleration to AI use. However, the report itself admits that most of the strongest evidence was redacted by the lab for confidentiality, leaving the public to rely on METR's judgment rather than the evidence itself—this draws a practical boundary for 'lab disclosure as measurement data': independent audits can verify conclusions but may not be able to make evidence public.

The karpathy/autoresearch integrity log discussion is a practical sample of closed-loop constraints: it verifies four paths that move val_bpb without model improvement (skipping the optimizer while still consuming the budget, short-circuiting evaluation calls, unfingerprinted validation sets, and results only printed without binding to code/data/model) and provides an approximately 30-line stdlib integrity log that binds reported metrics to code hashes, validation shards, and true optimization steps. The discussion further distinguishes four levels of assurance—process receipt, signed receipt, external witness, and hardware endorsement—and the distinction that 'integrity, authenticity, and execution truth are three different guarantees' has direct value for designing closed-loop governance layers.

The Incumbent Did the Work is a systematic calibration of record attribution in evolutionary program search: it compiles 27 published circle-stacking results into a table, showing that multiple 2026 records differ by about 2.5e-7, one is amplified within validator tolerance, and argues that the 2024 baseline of 'carrying the best program forward' is already near-winning on most benchmarks, while the value of additional mechanisms like archives, diversity objectives, and test-time weight updates is not demonstrated by these benchmarks. Its seven-step protocol for 'attributing reported gains to harness components' aligns with the honest accounting valued in this direction, similar to the Discovery Loop record attribution analysis in Issue 12.

ThinkingBox provides a large-scale failure taxonomy from 79,853 failures out of 121,680 valid runs: 67.24% of failures 'look complete' in transcription signals (have completion signals, call state-changing tools, and final tool responses show no explicit errors), but 98.95% of failures have final databases that do not match expectations; tool-use failures account for 79.9%. It advances 'invalid run determination' from anecdotal reports to a statistically scalable scale and notes that training and evaluation must use the same interpretation code to parse model outputs—otherwise, the reward measures a different agent.

Open Questions

  1. Trace Evolution Bench shows that seven DeepSeek units exceed their own noise band, but GLM/MiniMax do not reproduce, and no method exceeds its own band across all seeds—should 'frozen no_evolve rescans + per-seed paired excess' become a mandatory control for any claimed gain? When the noise band itself is as wide as 17.5 percentage points, which benchmarks' endpoint Gain has already lost meaning?

  2. The Reuse framework provides theoretical guarantees for simultaneous validity and cumulative improvement lower bounds under pass/fail and bounded scores, but its error budget thins exponentially with rounds and candidates—in long-horizon, large-candidate-pool closed loops, will this budget be diluted to the point where no candidate can actually be promoted? Can the 'decision-only feedback' assumption be relaxed to non-binary signals like human voting or continuous rewards?

  3. The Self-Improvement Problem proves that fully computable gates cannot be both reliable and non-conservative under unbounded semantics, but provides recovery guarantees under bounded evaluation and finite proposal budgets—where does the 'bounded' boundary fall in practice? Can the construction where agent drift causes true scores to decline round by round be converted into a precomputable drift threshold?

  4. Both Guidance-TTT and TTME demonstrate that 'decomposition' beats holistic black-box self-improvement, but they point to opposite conclusions: the former concentrates test-time learning in the policy layer (guidance), while the latter stores experience in external memory rather than weights. Under what task/budget/backbone combinations does the boundary of 'should experience be stored in context or weights' flip? Can these two frameworks complement each other in the same closed loop rather than being mutually exclusive?

Papers in this issue

  1. Trace Evolution Bench: Measuring Whether Agents Improve from Their Own Environment Traces

    Trace Evolution Bench isolates attributable self-evolution gains from evaluation noise via frozen rescoring and seed-matched excess, revealing endpoint gains alone overstate progress.

    Editor's note

    Should read. It explicitly makes 'frozen rescoring variance' a control baseline for self-improvement measurement: using four controls—locked held-out ids, validation-gated acceptance/rollback, four frozen no_evolve rescans, and per-seed paired excess—it separates the portion of endpoint Gain attributable to evaluation noise. Compared to Issue 9's Winner's Curse analytical modeling of selection noise, the increment is a reproducible protocol that specifically reports the magnitude of 'scores moving while versions are unchanged'—DeepSeek moves 7.5–12.5 percentage points across seeds with unchanged versions, seven units exceed their own noise band, but GLM/MiniMax do not reproduce, and no method exceeds its own band across all seeds. It is a methodological warning for any closed-loop evaluation relying on endpoint Gain.

  2. Which Self-Improvements Should We Trust? Reliable Self-Improvement When Agents Reuse Their Benchmarks

    REUSE provides certified evaluation for recursive self-improvement, guaranteeing with probability 1−α that every promoted modification is a true population improvement, eliminating false promotions without sacrificing performance.

    Editor's note

    Should read. It is the first to systematically introduce the statistical theory of adaptive data analysis and multiple testing into the evaluation layer of self-improvement loops: the Reuse framework returns only promotion decisions, explicitly allocates error budgets across all possible decision histories, controls three types of false promotion sources (within-round selection, repeated testing, and adaptive dependence), and provides theoretical guarantees for simultaneous validity and cumulative improvement lower bounds. Compared to Issue 9's Winner's Curse analytical modeling of shared selection error, the increment is incorporating 'adaptive dependence' itself as a provable error source into a unified framework, with an operational algorithm and a report of false promotion counts over 30 seeds. It has direct governance implications for any self-improvement system relying on fixed benchmarks.

  3. The Self-Improvement Problem

    No terminating gate can be both sound and complete for self-improvement when proposals can self-reference, but explicit timeouts and finite budgets restore reliable guarantees.

    Editor's note

    Should read. It elevates the empirical assertion that 'evaluator hacking is inevitable' to a verifiable formal boundary: any fully computable gate (certifier or searcher) cannot simultaneously satisfy reliability and non-conservatism under unbounded semantic behavior and self-referential candidates, and randomization, hiding, and stateful gates cannot circumvent this; it also proves recovery guarantees under bounded evaluation and finite proposal budgets, but agent drift can still cause true scores to decline round by round. Compared to the empirical approaches of Issue 9's CATCH and Issue 7's Reward Hacking Challenges, the increment is making 'gate-proposal self-reference' a formal theorem and providing drift constructions. Purely theoretical with no runtime data, but as a theoretical anchor for closed-loop constraints, it has direct value for designing governance layers.

  4. Evolving in Thought Space: Training a Small Model at Test Time Unlocks Better Discoveries (Guidance-TTT)

    Guidance-TTT decouples strategic adaptation from solution generation via a small trained guidance model, achieving state-of-the-art results across four scientific discovery domains at 58% lower cost.

    Editor's note

    Should read. It advances the role separation in test-time training to a new level: only a small guidance model (8B/14B) is updated to propose high-level policy changes, while a frozen strong executor model implements full code, enabling 'thought space' learning rather than directly generating complete solutions. Compared to Issue 10's Athanor weight self-training, the increment is explicitly distinguishing policy from implementation and using control groups to show that directly training the full solution generator is far worse than training the guidance (Polyomino 41.63 vs. 91.89). It also reports failure modes (execution failures causing reward ambiguity, search getting stuck in local families) and invalid run determination (compilation failures, correctness failures, HTTP 402), aligning with a preference for reproducible closed loops.

  5. Can Test-Time Memory Evolution Outperform Test-Time Training for Scientific Discovery?

    Test-Time Memory Evolution outperforms Test-Time Training in scientific discovery across all tasks while cutting computational cost by up to 84.6% in FLOPs.

    Editor's note

    Should read. It makes 'external memory evolution' a direct alternative to weight-based test-time training (TTT): TTME distills search experience into two complementary modules—Guideline Memory and Strategy Memory—guiding discovery under fixed weights, and in the same discovery framework (TTT-Discover, ThetaEvolve) it goes head-to-head with TTT—winning on all five math tasks, reducing FLOPs by 82.8–84.6% and runtime by 47.7–58.4%. Compared to Issue 10's Not All Experience Belongs in the Weights component routing, the increment is explicitly framing memory evolution as a substitute for weight adaptation and providing a clean ablation: adding TTT to TTME actually worsens performance. This negative result is new evidence for the 'should experience be stored in context or weights' debate.

  6. SelfImprove-Bench: Evaluating Generalization in Self-Improving Agents

    SelfImprove-Bench reveals that recursively self-improving agents show inconsistent generalization across difficulty levels and task types, with rankings reversing despite similar final scores.

    Editor's note

    Should read. It provides a five-level generalization ladder (G0–G4) that separates same-instance replay, unseen instances, harder difficulty, same-type tasks, and cross-type tasks level by level, with semantic process analysis attributing gains to modification surfaces and functional objectives. Compared to Issue 7's EvoPathBench three-capability separation, the increment is a unified five-level ladder with revision/repetition rates and modification surface quantification across three RSI methods, and the new measurement of 'ranking reversal across ladders' (HyperAgents wins on training difficulty but lags on harder instances) as a generalization inconsistency. The limitation is that it uses game-type tasks rather than real unsolved problems, but the protocol is transferable.

  7. Beneath the Surface: What We Overlook When We Talk About Agent Evolution

    Benchmark scores conceal that evolved agents gain newly covered tasks while losing previously covered ones, making aggregate metrics insufficient for evaluating agent evolution.

    Editor's note

    Should read. It is a mechanism-level audit of four harness evolution methods (AHE, HyperAgents, A-Evolve, GEPA): across two harnesses and two task domains, all sixteen evolution agents both gain and lose task coverage, reliability gains often come with higher deployment costs, and high-scoring parents do not yield the largest subsequent improvements. Compared to Issue 7's Rethinking the Evaluation of Harness Evolution budget protocol negative results, the increment is explicitly decomposing evolution into select/analyze/mutate/gate operators and empirically testing each operator (parent selection, component isolation, admission replay), then turning these diagnostics into EvoAudit/AGE operator revisions. It advances aggregate score criticism to component-level intervention.

  8. Scientist World: Grow a Research World for AI-Scientists

    Scientist World converts 1,000 research papers into execution-grounded environments where frontier AI-Scientists outperform human methods in up to 84% of tasks, while training smaller models to match that performance.

    Editor's note

    Should read. It is the first to make the 'paper → executable environment' conversion a human-out-of-the-loop scalable pipeline: 1,000 environments, 239,506 GPU hours, 730 agent rounds per paper, and introduces Frontier-Normalized Improvement (FNI) as a cross-paper normalized improvement metric. Compared to Issue 11's Automating ML for Science 20-benchmark testbed, the increment is making 'environment construction itself' a scalable automation object and providing three uses simultaneously: benchmark, trace dataset, and gym; its 'improve vs. innovate two-mode capability separation' and 'effort barely explains score variance (R²≤0.02)' are new measurement findings. Limitations: it does not report invalid run determination criteria, and the 74% baseline of surpassing human authors is 'reproduced paper methods' rather than human researchers themselves.