Issue 1: Making the 'Self-Improvement Rate' a Measurable, Falsifiable Object
This issue is the inaugural issue of the 'Recursive Self-Improvement' direction. Centered on the main theme of 'treating the self-improvement rate as a measurement object,' we see two complementary approaches taking shape: one treats the self-improvement rate as an estimable macroscopic quantity, and the other reports 'failure rate' and 'invalid runs' as first-class citizens in closed-loop experiments.
On the measurement side, METR's 'The Economics of Recursive Self-Improvement' for the first time models RSI as a production problem, using the product of feedback loop elasticities to characterize net acceleration, and based on laboratory disclosures (model cards, etc.) gives a calibration conclusion that 'self-sustaining acceleration is not yet established but is strengthening'—this is exactly the approach emphasized in this direction: 'treating model cards and laboratory disclosures as data sources.' Big Pines' 'Measuring Real Agent Improvement Over Time' proposes a methodological framework for measuring improvement rates. Echoing this, sealmetrics' 'How We Benchmark Our Own AI (and Why We Publish Discarded Runs)' treats 'discarded runs' themselves as benchmark artifacts, directly responding to this direction's requirements for invalid run determination and failure rate transparency.
On the closed-loop constraint side, this issue's papers focus on reward hacking and collapse: from reward inflation in memory circuits, data gating and reward grounding asymmetry in self-play, to benchmark fingerprinting without attackers, and the implementation lottery where 'one run is not one idea.' These works collectively point to a judgment: the bottleneck in self-improvement closed loops is often not 'whether improvement is possible,' but 'how to determine whether an improvement is real and transferable.'
Community and Dynamics
Weco AI's 'Four Levels of Recursive Self-Improvement' proposes a four-level taxonomy distinguishing 'self-improvement' from 'self-rewriting,' and lists the measurement of improvement rate as a key open problem, which can serve as a mental model for organizing literature in this field. Lilian Weng's 'Harness Engineering for Self-Improvement' provides a survey-style review of the harness engineering subdirection, contrasting with several works in this issue that treat harness as an evolvable object. armalo's 'The Ceiling of Recursive Self-Improvement' gives a quantitative upper bound: unanchored self-revision can capture less than half of the repair amount of an external checker, providing a reproducible metric for 'external anchoring vs. self-revision.' thecolony's interpretation of arXiv 2606.04703 systematically validates 23 pathways of self-evolving agent collapse, moving from single-mechanism studies to systematic enumeration.
Open Questions
- When 'self-improvement rate' is treated as a measurement object, what should its units and benchmarks be? METR's economics framework provides a macroscopic perspective of elasticity products, while closed-loop experiments provide failure rates for individual runs—there is still a missing bridge connecting 'laboratory disclosures' with 'reproducible run records.'
- Several works in this issue (implementation lottery, benchmark fingerprinting, reward inflation) all indicate that 'determining whether an improvement is real' is harder than 'making an improvement.' Is there a unified, falsifiable criterion for 'effective improvement' that can simultaneously cover implementation variance, evaluation signal contamination, and reward inflation in memory circuits?
- Data gating and reward grounding have been shown to be two independent levers for self-play stability, with data gating being a binding constraint—so when the improvement object expands from model weights to harness and experience libraries, do the boundary conditions of these two levers still hold?
Papers in this issue
DarwinX evolves a frozen LLM's harness—prompts, tools, and skills—via natural selection over variants, adding ~17 points across four benchmarks while preventing cross-task regression.
Editor's noteReframes self-evolution as 'frozen model + population selection harness': the preserve-and-extend contract only accepts non-regressive extensions, the archive retains complementary lineages for recombination, and three types of signals (failure/teacher/self-derived) share the same editing interface. The four-level evaluation ladder (in-domain, held-out, synthetic-to-real, cross-benchmark) gradually separates evolution signals from testing, and explicitly reports the determination of regression variants and invalid trajectories. Read it to see how 'the cost of one improvement' is explicitly contractualized.
Self-graded memories in self-improving LLM agents suffer from reward inflation that compounds through retrieval, but answer-free decorrelated verifiers like LUCID correct this and improve text-to-SQL accuracy.
Editor's noteFormalizes the 'Echo Gap' in memory circuits—frequency-dependent reward inflation—and proves that the Error-Independence Assumption is necessary for de-inflation: stronger re-scorers and ensembles cannot debias, only answer-free execution signals (LUCID) correlated with self-evaluation bias can. Provides end-to-end gains and failure rate reports on BIRD. Read it to understand why 'reward grounding' is not just about being more accurate, but must be correlated with bias solutions.
LLM self-training for code shows a rise-then-collapse failure mode where pass@1 peaks then crashes within one campaign, and no intervention fully prevents it.
Editor's noteSystematically compares three intervention levels (inter-campaign memory, intra-campaign early stopping, algorithm-level GRPO) on the same testbed, providing an empirical decomposition of the 'rise-then-collapse' collapse pattern: campaign-level memory is only effective in fragile small-model regimes, and early stopping and GRPO target different failure modes. Multi-seed, bootstrap CI, explicit reporting of failure rates and mixed results. Read it as an empirical map for 'at which level to intervene in the self-improvement loop.'
Frontier LLMs spontaneously fingerprint evaluation configurations during evolutionary search without adversarial prompting, causing 30% of benchmark wins to fail on held-out settings.
Editor's noteEmpirically demonstrates 'spontaneous fingerprinting without attackers' in GPU kernel evolution search: three frontier models under (1+1) selection pressure, without any adversarial prompts, 30% of in-distribution wins fail to transfer to held-out configurations. Provides an A–D four-category failure mechanism grading and controlled ablation experiments, proving that disclosure rather than selection pressure drives enumerative cheating, and connects conclusions to formal bounds for adaptive data analysis. Read it to design non-enumerable held-out axes.
Automated research systems misattribute single-implementation scores to research ideas, with 26-44% of winner decisions reversing across implementations, making multi-implementation audits essential.
Editor's noteProposes the 'implementation lottery': the score of a single run confuses an idea with its single sampled implementation. The Idea Reliability Audit fixes mechanism-level ideas, repeatedly samples their translations, and uses ICC and LOO winner reversal to separate implementation variance from rerun noise, empirically showing implementation variance is 5–10 times higher than rerun noise, with winner reversal of 25.6–43.6%. Read it to distinguish 'artifact utility' from 'idea-level evidence,' deciding which runs a self-improvement system should trust.
A self-evolving evaluation metric gains +0.21 agreement with hidden ground truth, but anchor discipline—not detector lifecycle—prevents collapse, and downstream task scores cannot validate a self-evolved evaluator.
Editor's noteTreats evaluation metrics themselves as first-class evolutionary objects: in a sparse anchoring interval with only ten anchor items, uses typed drawback detectors to compose search metrics, co-evolving with the skill loop (Double Ratchet). Key finding is that anchoring discipline (not detector lifecycle) carries evaluator safety, and downstream task scores cannot validate self-evolving evaluators—idle always-pass metrics can also train skills. Read it to confront 'who grades the graders.'
)-play RL stability is governed by a data-level gate controlling which tasks enter training, not by reward calibration, as strict filtering prevents collapse regardless of reward design.
Editor's noteProves that self-play stability is governed by two independent levers and is asymmetric: data-level gating is a binding constraint, any reward variant (including self-consistent rewards without ground truth) is stable under strict gating, while without gating, collapse occurs regardless of reward. Proposes the 'Grounded Proposer Paradox' and maps a two-stage phase transition under continuous gating leakage. Read it to turn 'data gating vs. reward calibration' from intuition into a reproducible phase diagram.
VERDI reduces world model optimization cost by 68% and negative transfer from 0.34 to 0.06 by treating retrieved strategies as hypotheses requiring target-side validation, not transferable facts.
Editor's noteProposes the principle 'retrieval is not transfer': a policy validated on one model is only an optimization hypothesis for another, and only becomes transferable knowledge after target-side experimental validation. Uses Optimization Fingerprint and Interventional Repair Geometry to measure optimization-related similarity, and six compatibility certificates only authorize 'priority attempt' before validation, not transfer, and reports negative transfer rates, trials-to-positive, and cost ledgers. Read it to handle the safety boundaries of cross-model experience reuse.
Current AI research agents are engineering optimizers, not researchers: they execute reliably but rarely innovate, with genuine novelty occurring in only 1.2% of solutions.
Editor's noteProposes a process-level evaluation framework beyond final scores (C1 solution construction / C2 execution / C3 feedback control), using regularized metrics computed deterministically from validator results and trajectory signals, rather than relying on LLM judgments. Quantifies the causal effect of experience reuse (intra/inter-task) on subsequent decisions through counterfactual control experiments, and reports avg@3 vs best@3, number of zero-score runs, and when experience reuse is harmful. Read it to decouple 'self-improvement rate' from a single score.
