Full text not available for this paper
Summary
- Core contribution: Proposes evaluating LLM-driven ML experimentation systems on their process (governance metrics) in addition to their outcome (task score), via a "governed harness" with a control plane that owns trial lifecycle, scope validation, and keep/discard decisions.
- Key mechanism: An append-only ledger records one immutable, typed record per trial (kept, discarded, or failed_invalid), guaranteeing 100% provenance completeness across all 1,445 trials on six ML tasks.
- Main findings: The ledger exposed failures invisible to task scores—a no-memory arm with 120 total failures, a bounded-proposal adapter that ran out, and a reset failure detected via reproducibility hashes—and a matched counterfactual showed an ungoverned loop silently lost 43 of 50 trials.
- Memory ablation: Memory's effect on repeated-bad proposals is task-dependent; summary memory gave the lowest repeated-bad rate in 2/3 ResNet replicates but reversed on the synthetic task.
- Artifact release: All 1,445 trial ledgers are released so every reported number can be recomputed without re-running training.
Introduction and Theoretical Foundation
The paper addresses a fundamental gap in evaluating autonomous ML experimentation agents: scores say nothing about process. An agent reporting a 0.05 AUC improvement may have edited protected files, silently dropped crashed runs, or repeated the same bad idea many times. For scientific ML, auditability—"a traceable and reproducible record of every decision and its evidence"—matters as much as the score.
The theoretical framing follows the view of an agent as a model plus a harness forming a compound AI system. Three roles are separated:
- Manager (typically an LLM): proposes one bounded change per trial
- Worker: applies the patch and runs training
- Control plane: owns trial lifecycle, scope validation, metric parsing, keep/discard decisions, and ledger appends
Five auditability desiderata (Table 1):
| # | Desideratum | Operational meaning |
|---|---|---|
| D1 | Bounded edit scope | Agent cannot change protected benchmark/data/eval files |
| D2 | Complete attempt accounting | Every attempt recorded, including invalid proposals and crashes |
| D3 | Independent adjudication | Acceptance/rejection by fixed control-plane rules, not the agent |
| D4 | Provenance linkage | Each attempt links to patch, config, run, metric, decision evidence |
| D5 | Recomputability | Governance claims recomputable from ledger without re-running training |
The key distinction from prior work (AIDE, MLAgentBench, MLE-bench, MLflow, W&B): existing systems either couple search state with decision-making (AIDE), ask only "did the agent improve the task?" (MLAgentBench), or record runs post-hoc without governing them (trackers). This paper inverts the design—the control-plane contract is fixed and externally visible; managers and workers are replaceable.
Methodology
System Architecture
The governed harness (Figure 2) separates replaceable components (manager, worker, node) from the fixed control-plane contract. The control plane owns: scope validation, lifecycle state machine, keep/discard decisions, provenance ID assignment, and memory context updates.
Trial Lifecycle State Machine
Each trial follows a fixed lifecycle (Table 4):
| From state | To state | Condition |
|---|---|---|
| pending | executing | Pending guard written; worker starts |
| executing | kept | Metric parses; beats current best; patch diff retained |
| executing | discarded | Metric parses; does not beat current best |
| executing | failed_invalid | Scope violation, runtime error, missing metric, no-op patch, or precondition failure |
| executing | failed_invalid | Mid-trial crash via stale pending guard at next campaign start |
Lifecycle invariant: For every budget slot entering executing, exactly one terminal record is appended before the next slot opens. Enforcement has two paths: normal (pending guard written/deleted) and crash-recovery (stale guard raises PendingTrialError, a failed_invalid record is appended).
Benchmark Nodes
Six nodes share the same contract, differing only in NodeSpec YAML (Table 3):
| Node | Task type | Worker | Metric | Direction |
|---|---|---|---|---|
| resnet_trigger | Waveform classification | Claw | val_auc | max |
| lr_synthetic | Logistic regression | Local | val_score | max |
| openml_credit_g | Tabular classification | Local | val_auc | max |
| openml_bank_mkt | Tabular classification | Local | val_auc | max |
| mlagentbench_vec. | NumPy convolution | Local | speed_score | max |
| autoresearch_linux | LM pretraining | Claw | val_bpb | min |
Governance Metric Suite (Table 5)
Given trials:
- Acceptance rate (AR):
- Invalid rate (IR):
- Repeated-bad rate (RBR): repeated-bad proposals /
- Provenance completeness: trials with all five IDs /
- Artifact-reference coverage: patch/log refs and hashes
- Failure taxonomy: counts per FailureCategory
Memory Ablation Design
Three memory modes: no prior-trial context, append-only summary, and rationale-augmented summary. The behavioral hypothesis tested:
Metric Validation
Structural relabeling agreement: Cohen's (95% CI ) on a stratified sample of 50 failed_invalid trials.
Empirical Validation / Results
Cross-Node Governance Summary (Table 7)
| Node | Trials | AR% | IR% | Lifecycle outcomes |
|---|---|---|---|---|
| resnet_trigger (case study) | 20 | 20.0 | 55.0 | kept, disc., failed_inv. |
| resnet_trigger (L40S, all arms) | 225 | 46.7 | 0.4 | kept, disc., failed_inv. |
| lr_synthetic | 450 | 8.4 | 46.2 | kept, disc., failed_inv. |
| openml_credit_g | 230 | 13.5 | 26.1 | kept, disc., failed_inv. |
| openml_bank_marketing | 230 | 13.0 | 32.2 | kept, disc., failed_inv. |
| mlagentbench_vec. | 50 | 28.0 | 48.0 | kept, disc., failed_inv. |
| autoresearch (none arm) | 120 | 0.0 | 100.0 | failed_inv. only |
| autoresearch (summary) | 120 | 11.7 | 40.8 | kept, disc., failed_inv. |
Provenance completeness: 100% on every row.
Governed vs. Ungoverned Counterfactual (Table 9)
| Node | N | Governed | Ungoverned | Drops |
|---|---|---|---|---|
| autoresearch_linux (IR=100%) | 30 | 1.00 | 0.00 | 30 (100%) |
| openml_bank_mkt (IR=55%) | 20 | 1.00 | 0.35 | 13 (65%) |
Combined: 43/50 ungoverned trials absent vs. 0/50 governed.
Large-Scale Memory Replication (225 trials, NVIDIA L40S)
Across 5 clean seeds with verified reset integrity (shared node_state_hash at trial 1):
- Mean best AUC: none < summary ≈ rationale (0.899 vs. 0.939 and 0.941)
- Cross-seed variability: vs. (4–5× smaller)
- Memory arms diverge from no-memory by trial 5, hold gap through trial 15
- Provenance completeness: 100% across all 225 trials
Real LM Training Node (autoresearch_linux)
- None arm: IR = 1.00 across all 120 trials (4 seeds)
- Summary arm: 3/4 seeds produce valid improving trials (mean val_bpb reduction ); seed s2 suffered total worker failure
- Across 240 trials including 151 runtime-error crashes: provenance completeness 240/240 (100%)
Theoretical and Practical Implications
Theoretical Implications
-
Process-based evaluation paradigm: The paper argues that autonomous ML experimentation should be evaluated on governance metrics (provenance, lifecycle classification, scope enforcement, failure taxonomy) alongside task metrics—a shift from outcome-only benchmarking.
-
Structural vs. empirical guarantees: The lifecycle invariant holds "by construction of the control-plane code, not because nothing crashed." This distinguishes structural guarantees from empirical observations—a key methodological contribution.
-
Replaceability principle: Separating proposal (manager), execution (worker), and authority (control plane) means the LLM manager is a replaceable component, enabling controlled experiments on manager behavior without changing governance.
Practical Implications
-
Auditability for scientific ML: Every decision can be traced to its evidence (patch, log, metric, decision), enabling reproducible scientific experimentation.
-
Failure visibility: The ledger exposes failures that task scores hide—total failures, bounded-proposal exhaustion, and reset failures detectable via reproducibility hashes.
-
Memory diagnostics: Repeated-bad rate serves as a per-node diagnostic for memory sensitivity, but the task-dependence warns against general claims about memory formats.
-
Reproducibility: Released ledgers allow recomputation of all reported numbers without re-running training (no GPU required).
Conclusion
The paper demonstrates that "autonomous ML experimentation should be evaluated on governance metrics—provenance, lifecycle classification, scope enforcement, and failure taxonomy—not only task metrics." Across 1,445 governed trials on six ML nodes, the control plane classifies every outcome without silent corruption, and provenance is complete by construction.
Key limitations acknowledged by the authors:
- Short budgets and limited node diversity
- Structural relabeling is a consistency check, not a human inter-rater study
- ResNet AUC gains fall within baseline seed noise
- Human-auditor utility not yet tested
Future directions: A human-auditor utility study (do reviewers with governed ledgers detect seeded errors faster?), two-rater validation, and holdout-node replication.
Evidence strength summary (Table 10): Demonstrated properties include control-plane classification, 100% provenance, guard failure visibility, and protocol transfer across node types. Observed-but-not-claimed findings include memory ordering on AUC (5 clean seeds) and smaller cross-seed spread with memory. The claim that "rationale reduces repeated-bad rate" is explicitly not claimed due to non-monotonic results.
Related papers
- Do Halting Heads Really Save Compute in Looped Language Models?
Halting heads in looped language models save far less compute than reported because mixture-trained heads rarely stop tokens and batching overhead eliminates most remaining gains.
- GUARD: Stabilizing Recursive Self-Improvement with Regret Guidance
GUARD achieves non-regressive agent harness self-improvement without external graders by separating edit-induced regressions from capability gaps via paired regret, guaranteeing monotonic accuracy gains.
- Rethinking Fixed-Horizon Latent Reasoning
Adaptive latent inference that stops reasoning when progress stalls improves accuracy by up to 7.2 points while cutting computation by 29%, as completing fixed latent horizons can degrade performance.