Self-Improvement Rate Enters the Era of Laboratory Disclosure
Highlights of This Issue
The main theme of this issue is the advancement of measuring the 'self-improvement rate' from paper-level methodology to laboratory-level disclosure. Anthropic released the first R&D Automation Index that turns the 'self-improvement rate' into a public measurement framework: using Epoch AI's AL0-AL5 automation grading, it performs weighted aggregation on approximately 15,000 model R&D tasks, providing comparable figures such as 'Claude leads 26% of AI R&D, with over 90% reaching collaboration level,' and turning 'laboratory disclosure as a data source' from a slogan into a reproducible protocol. This directly responds to the focus of this direction on 'treating model cards and laboratory disclosures as data sources'—but note its self-evaluation nature: using its own models to evaluate its own systems, and currently only the abstract is visible.
The AI scientist closed loop has significantly scaled up this issue. ScientistTwo makes 'simulated peer review-rebuttal' a core component of the closed loop, reporting success and failure rates on 107 top-conference paper tasks; Qiushi Engine completed a long-horizon autonomous research on the structural proof of the lower bound 21 for the tensor rank of 3×3 matrix multiplication, from numerical experiments to Lean formalization. Together, they advance 'what the closed loop can produce' from paper-level to 'publishable papers + formal mathematical proofs' level.
Measurement methodology continues to advance. RSIBench-Data isolates 'data-centric research capability' from end-to-end post-training, quantifying the 'discovery-reliability gap'—78.26% of searches end with final attempts below their peak; Evolution or Illusion? turns the budget allocation of 'width vs depth' into a reproducible measurement object. Component-decomposed harness evolution also has new samples: Dream-RSI uses historical trees as replay simulators for zero-cost offline policy improvement, and ModularRSI uses a benchmark-disjoint protocol for the first time to independently evolve five modules.
Community and Developments
The most important community event this issue is Anthropic's R&D Automation Index disclosure. It turns the 'self-improvement rate' into a public index comparable across months and discusses barriers to third-party verification and cross-laboratory comparability—this is pushing 'laboratory disclosure as a data source' toward operability. On the independent audit side, Progressiverobot's breakdown categorizes Anthropic's 40 self-improvement rate claims into 'measured numbers' and 'unquantified policy proposals,' providing community-level verification; backfield.net's disclosure gap analysis points out that frontier releases 'score models but are blind to harness/run aspects,' citing Nemotron and GPT-5.6's reasoning effort curves as rare transparent counterexamples.
There are also noteworthy samples in the practical side of closed-loop constraints. The pass^k metric treats 'all attempts' as a first-class metric for autonomous overnight runs, quantifying '4/5 workflows only pass occasionally' and warning that 'clean termination may mask erroneous outputs'—this directly aligns with this direction's focus on criteria for invalid runs. On the tool side, sia provides a self-improvement loop CLI that persists per-generation agent_execution.json logs and improvement diff reasons, with a real-time run visualization panel, serving as reproducible scaffolding for putting 'report failure rates rather than only the best attempt' into engineering practice.
Open Questions
- Anthropic's index uses 'its own models to evaluate its own systems,' and currently only the abstract is visible—can third-party verification close this self-evaluation gap? This is directly related to the critical stance needed when 'treating laboratory disclosures as data sources.'
- Evolution or Illusion? proves that the optimal width-depth split varies with policy, task, and budget—can the optimal split be predicted a priori rather than measured post hoc? The authors acknowledge this as an open problem.
- ScientistTwo's review-rebuttal loop is simulated: does simulated review feedback improve 'reviewer satisfaction' or 'actual research progress'? How does this differ from the 'gaming the evaluator' risk in reward hacking?
- RSIBench-Data shows that 78.26% of searches end with final attempts below their peak—why does search degrade after reaching a peak? Can mechanisms be designed to stop or roll back at the peak rather than continue deteriorating?
Papers in this issue
ScientistTwo, a fully autonomous multi-agent framework, outperforms human state-of-the-art on 86 of 107 research problems, producing publication-quality papers with verified code without human intervention.
Editor's noteShould read. It advances the AI scientist closed loop to 'simulated peer review-rebuttal' as a core component: the Meta-Review Agent drives deep idea refinement based on review feedback, and the Rebuttal Agent proactively designs and executes supplementary experiments to respond to reviewer comments, forming a closed loop. Compared to the 'engineering meets standards but research judgment does not' case in Issue 5's shadow evaluation, this provides larger-scale (107 top-conference paper tasks) closed-loop evidence, reporting success rates, average relative improvements, and failure rates (21/107 unsuccessful). Limitation: both review and rebuttal are simulated, not real peer review, so one must be wary of the separation between 'reviewer satisfaction' and 'actual research progress.'
RSIBench-Data isolates data-centric research from training infrastructure, revealing a discovery-reliability gap where agents improve initial attempts but regress in 78% of continued feedback-driven searches.
Editor's noteShould read. It isolates 'data-centric research capability' from end-to-end post-training benchmarks, fixing the training/serving/evaluation stack, and specifically measures the agent's ability to propose hypotheses, synthesize data, validate, and revise strategies in a closed loop. Compared to S3Gym's coupled self-test-self-evaluate-self-improve measurement in Issue 5, the increment is making 'data strategy evolution' an independent measurement object and reporting the 'discovery-reliability gap': 58.33% of settings iteratively improve the first candidate, but 78.26% of searches end with final attempts below their peak—this is a rare quantification of whether 'improvement is monotonic.' It aligns with the preference for reproducible closed loops and failure rate reporting.
Dream-RSI repurposes completed discovery histories as replay simulators, enabling zero-cost evaluation of exploration policies and cutting discovery compute by up to 162x without sacrificing quality.
Editor's noteShould read. It reframes 'completed discovery history' as a replayable simulator, turning meta-level improvements to exploration strategies from expensive online rollouts into zero-execution-cost offline 'dreaming' evaluation—this is an increment over Issue 5's Evolution Fine-Tuning (cross-task meta-learning during training): elevating history from a static context to an interactive simulated environment, explicitly separating online exploration from offline policy improvement. It reports evolutionary behavior where exploration effort first decreases then increases across recursive rounds, and distinguishes invalid run criteria such as fail_class, n_valid==0, and repairable. Limitation: replay evaluation depends on historical tree coverage, potentially underestimating performance on unseen branches.
ModularRSI achieves generalizable recursive harness self-improvement by contrastively analyzing trajectories and evolving five independent harness modules, outperforming joint evolution and transferring across unseen tasks, domains, and foundation models.
Editor's noteShould read. It is the first to systematically use 2000 evolution tasks completely disjoint from downstream evaluation in harness self-improvement, and localizes defects using same-task success/failure trajectory comparisons + cross-task evidence aggregation + independent evolution of five modules (Agent Loop, Tool Use, Observation Management, Context Management, Task Completion Detection). Compared to HarnessDev's two-stage creation+evolution in Issue 4, the increment is separating evolution data from evaluation data and explicitly reporting that modular independent evolution outperforms joint/non-modular evolution (TerminalBench Acc 47.57→52.43, vs non-modular 46.44, joint below baseline), with cross-model transfer validating generality. It reports invalid run criteria such as infra_only exclusion.
Single-point budget evaluations of LLM evolutionary search yield unreliable rankings, as optimal seed-iteration splits and even winner identities invert with budget.
Editor's noteShould read. It turns 'budget allocation' itself into a measurement object: on a full grid of 40 seeds × 200 iterations, it replays recorded trajectories with exact order statistics, computes the expected optimal score for each (k,t) split, and proves that the optimal width-depth split varies with policy, task, and total budget, and that single-seed evaluation can reverse policy rankings. Compared to Issue 2's Phantom Gains transfer-level audit, the increment is making the 'width vs depth' budget allocation a reproducible measurement protocol and providing probabilistic evidence of ranking reversals. Its 'replay recorded trajectories + exact order statistics' protocol has zero cost, and it reports how validator vulnerabilities are exposed by multiple seeds. Limitation: single model, single engine, and does not predict the optimal split (authors acknowledge this as an open problem).
The tensor rank of 3x3 matrix multiplication over the binary field is at least 21, proven via a novel saturation argument and fully verified in Lean.
Editor's noteShould read. It advances the measurement object of 'self-improvement rate' to a specific mathematical problem: Qiushi Engine autonomously completes a structural proof of the lower bound 21 for the tensor rank of 3×3 matrix multiplication over F2, recording the full research trajectory, failure corrections, and Lean formalization. Compared to Issue 5's Evolution Fine-Tuning cross-task meta-learning, the increment is a 'long-horizon autonomous research closed loop on a single unsolved mathematical problem'—a complete trajectory from numerical experiments, quotient constructions, to structural proof, with explicit reporting of early errors (e.g., misjudgment of paired singular values) and corrections. The mathematical proof itself is not core to this direction, but as a complete case of an autonomous research closed loop, it is highly valuable as a reference.
SoL-Pi's auto-research loop discovers four reusable harness mechanisms that cut coding agent token traffic by up to 49% and API cost by ~33% while maintaining performance.
Editor's noteShould read. It shifts the measurement object of 'self-improvement rate' from capability gains to token efficiency, proposing a broad-to-deep funnel-style harness search: approximately 150 hypothesis directions, approximately 500 executable environments, 3000+ runs, with dual filtering by capability gating (fixed tolerance) and efficiency gating, and strict isolation of held-out validation. Compared to Issue 5's Recursive Harness Self-Improvement prompt-level harness specification, the increment is making 'token efficiency' itself a measurable optimization target and reporting independent and combined ablations of four retention mechanisms (Action Fusion, Online Context Compact, ObservationPack, Evidence-Preserving Reducer), as well as cross-model (GPT-5.6 Sol → Opus 5) transfer evidence. Limitation: the Efficiency point trades about 6% capability for 44.7-49.0% token reduction, and invalid run criteria are not reported.
