Assessing Resolution: The Bottleneck of Self-Improvement Lies in Identification, Not Proposal
Highlights of This Issue
The main theme of this issue is "assessment resolution"—the bottleneck of the self-improvement loop is systematically attributed for the first time to "noisy evaluation failing to reliably identify improvements" itself, rather than methods failing to propose better artifacts. The Resolution Hypothesis uses variance decomposition (σ² evaluation noise + τ² run-to-run variance) across 1,030 independent evolution runs and 2.1M test trajectories to provide the largest reproducible evidence to date: candidate-level selection is near-random (49.7% pairwise accuracy), method-level single runs cannot be ranked (only 33% accuracy on Terminal-Bench), and large gains all come from evaluation shortcuts (single-sentence rules reproduce 90% of gains; after fixing, median gain drops from 17.6 to 0.8 pp). This aligns with the frozen re-scoring variance of Trace Evolution Bench in Issue 13, but turns "resolution" into a unified framework that simultaneously explains three phenomena: candidate selection failure, method ranking instability, and shortcut gains.
Weight-level self-improvement emerges as an independent line this issue. Language Models Can Rewrite Their Own Weights demonstrates for the first time that large models, without gradients or training, can diagnose their own failures and rewrite the underlying weights purely through reasoning, with this capability emerging with scale; BaseRSI internalizes harness gains into training-free, block-by-block weight edits, using ReGap high-dimensional activation differences to supervise each transformer block, completing recursive internalization at a minimal cost of 45.5 GPU hours. Together, these two lines advance "self-improvement" from harness editing to the weights themselves, directly responding to the leverage debate between "weight training vs. harness evolution" after Athanor in Issue 10.
On the closed-loop constraint side, statistical gating and governance layers become measurable objects. TRUST introduces online multiple hypothesis testing (e-LORD/e-GAI) into the acceptance gate, defining cross-round false evolution rate (FER) and providing a lower bound on gains; Evaluating Governed LLM-Driven ML Experimentation turns the governance layer itself into a fixed contract, with all 1,445 trials recorded in an append-only ledger, quantifying that 43/50 trials vanish without a trace in the ungoverned case. When Can Self-Play Fix a False Belief treats the source of the answer key and the pass band as two independent gates, proving that "a correct key is not sufficient"—corrections can lead to replacement rules that fail in neighborhoods where the old belief was correct.
Community and Developments
The GPT-6.2 System Card is the first public model card to explicitly disclose a "High" level AI self-improvement threshold: OpenAI used a two-stage screening with five research engineering evaluations (Internal Research Debugging, KernelGen, NanoGPT, PostTrainBench Lite, MLE-Bench) to determine that GPT-6.2 Astra reaches the High threshold while Sol does not. This is the latest instance of "treating lab disclosures as data sources"—but the full methodology and failure rates are unavailable, so caution is warranted.
Most trials should lose publishes an honest failure ledger: an AI-scientist-style closed loop loses 85 out of 133 trials, yet still contributes 70.8% of the benchmark's progress, explicitly reporting failure modes such as blank-sheet designs, agreeing forecasts, and false exactness—"most trials should lose, and they should lose early" is the most direct counterintuitive data point on failure rate accounting.
How to Measure RSI proposes eight data requests, treating internal model scores, compute efficiency curves, and researcher uplift surveys as tools to measure the rate of self-improvement, and scores frontier labs: OpenAI 2/8, Anthropic 1.5/8, Google DeepMind 0.5/8—similar to the FRI evidence review in the previous issue as a third-party perspective treating model cards as data sources, but more actionable.
ModHarness-RL decomposes the agent scaffold into typed turn records, separating compression and guidance into two programs, and evolves the guidance program across RL rounds through an admission pipeline with execution validation, history replay, and held-out gating—a new instance of component-level scaffold evolution in RL.
Open Questions
-
The Resolution Hypothesis shows candidate-level selection is near-random (49.7% pairwise accuracy) and method-level single runs cannot be ranked (33% accuracy)—when the selector itself is unreliable, how many previous self-improvement conclusions that relied on "selecting the best run" need re-examination? Should Quality@K and selector curves become mandatory reporting items for any closed-loop evaluation?
-
Language Models Can Rewrite Their Own Weights and BaseRSI advance self-improvement to the weight level—when both weight editing (self-rewriting, block-by-block internalization) and harness evolution yield gains, where is the boundary between them? How do BaseRSI's "training-free internalization" and SkillDuet's "skill utility collapses with training" from the previous issue complement each other in the same closed loop?
-
When Can Self-Play Fix a False Belief proves that replacement rules fail in neighborhoods where the old belief was correct—under "a correct key is not sufficient, pass band blocks," how can replacement rules be detected and prevented? Can the three-gate framework combine with the cross-prompt sharing mechanism of Wrong the Same Way from Issue 14 to provide a precomputable correction risk threshold?
-
TRUST's FER control and the Reuse framework from Issue 13 both attempt to give statistical guarantees to the acceptance gate—can non-inferiority tolerance and overlap correction be unified with Reuse's adaptive dependence control into a single gating protocol? When LLMs analyze historical context to allocate testing levels, how are their error cases audited?
Papers in this issue
Agent evolution fails because noisy evaluations cannot identify helpful improvements, not because proposals are lacking, making selection nearly random.
Editor's noteShould read. It is the first to make "assessment resolution" an independent measurement object in the self-improvement loop, using variance decomposition (σ² evaluation noise + τ² run-to-run variance) across 1,030 independent evolution runs and 2.1M test trajectories to prove: candidate-level selection is near-random (49.7% pairwise accuracy), method-level single runs cannot be ranked (only 33% accuracy on Terminal-Bench), and large gains all come from evaluation shortcuts (single-sentence rules reproduce 90% of gains; after fixing, median gain drops from 17.6 to 0.8 pp). Compared to the frozen re-scoring variance of Trace Evolution Bench in Issue 13, the increment is turning "resolution" into a unified framework that simultaneously explains three phenomena: candidate selection failure, method ranking instability, and shortcut gains, and providing Quality@K and selector curves as component-level diagnostics. Limitation: candidate-level evidence is limited to Terminal-Bench and does not cover weight-training-based self-improvement.
A 27B language model improved its own accuracy by 6.6 points and repaired a damaged copy of itself using only forward passes, with no gradients or training.
Editor's noteShould read. It demonstrates for the first time that large language models, without gradients or training, can diagnose their own failures and rewrite the underlying weights purely through reasoning, with this capability emerging with scale—a direct instance of the "system improving itself" loop at the weight level. Compared to Athanor's weight-training loop in Issue 10, the increment is that the editor and the edited model are the same, autonomously completing the diagnose-test-submit loop, and reporting honest failure data (withdrawals, small models breaking themselves) and held-out evaluation. Limitation: gains are modest and repair tasks are synthetic, but the paradigm shift is clear.
Self-play with execution-grounded keys cannot correct false beliefs because the generator's own code inherits its misreadings, and decorrelated keys are necessary but insufficient for correction.
Editor's noteShould read. It is the first to treat "answer key source" and "pass band" as two independent gates for belief revision in the self-improvement loop, and systematically measures "replacement rules"—after correction, the model learns incorrect rules in neighborhoods where the old belief was correct (S2 over-counts after correction, R2 adds a divisor, R1 rounds toward zero). Compared to the cross-prompt sharing of Wrong the Same Way in Issue 14, the increment is explicitly distinguishing the three gates (generation-key-pass band) with per-task audits, and providing quantitative evidence that "a correct key is not sufficient, pass band blocks, replacement rules fail in neighborhoods." Limitation: single model, small-scale loops, but the three-gate framework and neighborhood probe protocol are transferable.
TRUST introduces an online multiple hypothesis testing gate that provably bounds false evolution rate while achieving higher performance gains than greedy and existing statistical acceptance methods in recursive self-improvement.
Editor's noteShould read. It turns the "acceptance gate" itself into a measurement object with statistical guarantees, introducing online multiple hypothesis testing (e-LORD/e-GAI) into RSI, defining cross-round false evolution rate (FER) and providing theoretical guarantees for FER control, TER consistency, and a lower bound on final gains. Compared to the Reuse framework in Issue 13 (Which Self-Improvements Should We Trust), the increment is two interpretable parameters: non-inferiority tolerance ε and overlap correction β, and using LLMs to analyze historical context to allocate testing levels. Limitation: invalid run criteria are not disclosed, and the SGM baseline completely rejects updates, making comparisons weak, but as a general framework for statistical gating, it has direct value for designing closed-loop governance layers.
BaseRSI lets a vision-language model improve its own weights via training-free per-block activation edits, gaining +11.7 points in-distribution and +6.1 out-of-distribution across ten benchmarks.
Editor's noteShould read. It turns "internalizing harness gains into weights" into a training-free recursive loop: using block-wise teacher-student activation differences (ReGap) as high-dimensional supervision signals, ridge regression writes each block's gap into a small set of key-value neurons (ReKey), and refits on the student's own responses over multiple rounds. Compared to Athanor's weight-training loop in Issue 10, the increment is fully training-free, no backpropagation, and explicitly measures the site and strength of edits, reporting 269 candidates, Holm threshold, three rounds of self-response refitting, and 0.23% weight rewriting. Limitation: invalid run criteria are not reported, and it is mainly limited to visual reasoning benchmarks.
Pro²Agent's explicit Problem State—tracking diagnosed bottlenecks, candidate solutions, and evidence—achieves a record 76% medal rate on MLE-Bench, outperforming code-centric agents.
Editor's noteShould read. It treats "problem state" as a first-class citizen in autonomous AI research, explicitly maintaining performance bottlenecks, candidate solutions, evidence, and confidence alongside codebase state, and updating this state through controlled hypothesis testing, providing a three-state bottleneck classification (Resolved/Resource-Limited/Unresolved) as failure criteria. Compared to the policy/implementation separation of Guidance-TTT in Issue 13, the increment is making "problem understanding" itself an explicit state that is measurable, rollback-able, and reusable, and using Git rollback and univariate isolation to ensure the attributability of hypothesis tests. Limitation: full MLE-Bench results are from a single run and failure rates are not reported, but controlled ablations and Node AUC provide reproducible measurement protocols.
Newer LLM generations spontaneously construct their own self-improvement algorithms from primitive tools alone, transferring learned strategies across tasks and models without prescribed learning loops.
Editor's noteShould read. It is the first to systematically measure whether different generations of LLMs can autonomously organize self-improvement processes under conditions of no externally specified loop, no real data, and only raw tools and general objectives, finding that newer models spontaneously adopt identifiable strategies consistent with self-regulated learning (SRL) frameworks and learnability objectives, and that learned text artifacts transfer across engines, representations, and models. Compared to the recursive revision of Organon in Issue 14, the increment is making "autonomous construction of the self-improvement loop" itself a measurement object, and providing cross-generational capability trajectories and information-generalization trade-offs. Limitation: task domain is games, and full failure rates are not reported.
GUARD achieves non-regressive agent harness self-improvement without external graders by separating edit-induced regressions from capability gaps via paired regret, guaranteeing monotonic accuracy gains.
Editor's noteShould read. It migrates "directional regret" from UED's level selection to criterion ranking in harness evolution: using a pre-execution task checklist to decompose each task into atomic binary checks, distinguishing "newly broken behaviors" from "unresolved gaps," and using paired regret ranking to ensure new regressions strictly precede non-regressed criteria. Compared to the score-band gating of RRSI in Issue 8, the increment is making the "regression vs. gap" separation a provable ranking criterion, and using Soft-HURT gating to guarantee non-regression on the gating buffer, reporting honest failure data such as 19/19 execution gate promotions all with Δ_test≥0 and 0/12 block drops. Limitation: main results are from a single model (Gemini 4), and some benchmarks use critic scoring.
Governed harnesses with append-only ledgers expose process failures invisible to task scores, achieving 100% provenance across 1,445 trials while ungoverned loops silently lose 86% of trials.
Editor's noteShould read. It turns the "governance layer itself" into a measurement object: the control plane (not the LLM) owns the trial lifecycle, edit scope validation, keep/discard decisions, and an append-only ledger, and computes governance metrics such as invalid-action rate, repeated-bad rate, and provenance completeness from the ledger. Compared to The RSI Tax in Issue 14, which makes cost a measurement object, the increment is making the "governance contract" itself a fixed invariant across six nodes and 1,445 trials, and providing an ungoverned counterfactual that quantifies lost records (43/50 trials without a trace). Limitation: single author, KDD workshop non-archival, but as a protocol template for "reporting governance metrics alongside task metrics," it has direct value.
A naive LLM critic can drive AutoML search below the no-critic random-search floor on 7 of 8 tabular datasets, with losses up to 0.245 AUC.
Editor's noteShould read. It is the first to make the "marginal harm of LLM critics" a measurable object: through a four-arm audit (no-critic/random/real/oracle) on a shared cache frontier, it isolates the marginal effect of the critic, and uses controlled fault injection to separate error rate from error structure, proving that non-directional errors are nearly costless under best-so-far search, while directional errors consume limited search opportunities. Compared to the cross-prompt sharing bidirectional channel of Wrong the Same Way in Issue 14, the increment is treating the "critic feedback channel itself" as an independent intervention variable, and providing an analytical boundary where error structure (not error rate) determines harm. Limitation: limited to small fixed frontiers and tabular tasks, but as a strong negative result that "critics are not free additions" and a transferable audit template, it has direct value.









