Full text not available for this paper

Summary

  • Core contribution: Proposes evaluating LLM-driven ML experimentation systems on their process (governance metrics) in addition to their outcome (task score), via a "governed harness" with a control plane that owns trial lifecycle, scope validation, and keep/discard decisions.
  • Key mechanism: An append-only ledger records one immutable, typed record per trial (kept, discarded, or failed_invalid), guaranteeing 100% provenance completeness across all 1,445 trials on six ML tasks.
  • Main findings: The ledger exposed failures invisible to task scores—a no-memory arm with 120 total failures, a bounded-proposal adapter that ran out, and a reset failure detected via reproducibility hashes—and a matched counterfactual showed an ungoverned loop silently lost 43 of 50 trials.
  • Memory ablation: Memory's effect on repeated-bad proposals is task-dependent; summary memory gave the lowest repeated-bad rate in 2/3 ResNet replicates but reversed on the synthetic task.
  • Artifact release: All 1,445 trial ledgers are released so every reported number can be recomputed without re-running training.

Introduction and Theoretical Foundation

The paper addresses a fundamental gap in evaluating autonomous ML experimentation agents: scores say nothing about process. An agent reporting a 0.05 AUC improvement may have edited protected files, silently dropped crashed runs, or repeated the same bad idea many times. For scientific ML, auditability—"a traceable and reproducible record of every decision and its evidence"—matters as much as the score.

The theoretical framing follows the view of an agent as a model plus a harness forming a compound AI system. Three roles are separated:

  • Manager (typically an LLM): proposes one bounded change per trial
  • Worker: applies the patch and runs training
  • Control plane: owns trial lifecycle, scope validation, metric parsing, keep/discard decisions, and ledger appends

Five auditability desiderata (Table 1):

#DesideratumOperational meaning
D1Bounded edit scopeAgent cannot change protected benchmark/data/eval files
D2Complete attempt accountingEvery attempt recorded, including invalid proposals and crashes
D3Independent adjudicationAcceptance/rejection by fixed control-plane rules, not the agent
D4Provenance linkageEach attempt links to patch, config, run, metric, decision evidence
D5RecomputabilityGovernance claims recomputable from ledger without re-running training

The key distinction from prior work (AIDE, MLAgentBench, MLE-bench, MLflow, W&B): existing systems either couple search state with decision-making (AIDE), ask only "did the agent improve the task?" (MLAgentBench), or record runs post-hoc without governing them (trackers). This paper inverts the design—the control-plane contract is fixed and externally visible; managers and workers are replaceable.

Methodology

System Architecture

The governed harness (Figure 2) separates replaceable components (manager, worker, node) from the fixed control-plane contract. The control plane owns: scope validation, lifecycle state machine, keep/discard decisions, provenance ID assignment, and memory context updates.

Trial Lifecycle State Machine

Each trial follows a fixed lifecycle (Table 4):

From stateTo stateCondition
pendingexecutingPending guard written; worker starts
executingkeptMetric parses; beats current best; patch diff retained
executingdiscardedMetric parses; does not beat current best
executingfailed_invalidScope violation, runtime error, missing metric, no-op patch, or precondition failure
executingfailed_invalidMid-trial crash via stale pending guard at next campaign start

Lifecycle invariant: For every budget slot entering executing, exactly one terminal record is appended before the next slot opens. Enforcement has two paths: normal (pending guard written/deleted) and crash-recovery (stale guard raises PendingTrialError, a failed_invalid record is appended).

Benchmark Nodes

Six nodes share the same contract, differing only in NodeSpec YAML (Table 3):

NodeTask typeWorkerMetricDirection
resnet_triggerWaveform classificationClawval_aucmax
lr_syntheticLogistic regressionLocalval_scoremax
openml_credit_gTabular classificationLocalval_aucmax
openml_bank_mktTabular classificationLocalval_aucmax
mlagentbench_vec.NumPy convolutionLocalspeed_scoremax
autoresearch_linuxLM pretrainingClawval_bpbmin

Governance Metric Suite (Table 5)

Given NN trials:

  • Acceptance rate (AR): ∣kept∣/N|\text{kept}| / N
  • Invalid rate (IR): ∣failed_invalid∣/N|\mathrm{failed\_invalid}| / N
  • Repeated-bad rate (RBR): repeated-bad proposals / NN
  • Provenance completeness: trials with all five IDs / NN
  • Artifact-reference coverage: patch/log refs and hashes
  • Failure taxonomy: counts per FailureCategory

Memory Ablation Design

Three memory modes: no prior-trial context, append-only summary, and rationale-augmented summary. The behavioral hypothesis tested:

RBR(none)>RBR(summary)>RBR(rationale)\text{RBR}(\text{none}) > \text{RBR}(\text{summary}) > \text{RBR}(\text{rationale})

Metric Validation

Structural relabeling agreement: Cohen's κ=0.830\kappa = 0.830 (95% CI [0.703,0.958][0.703, 0.958]) on a stratified sample of 50 failed_invalid trials.

Empirical Validation / Results

Cross-Node Governance Summary (Table 7)

NodeTrialsAR%IR%Lifecycle outcomes
resnet_trigger (case study)2020.055.0kept, disc., failed_inv.
resnet_trigger (L40S, all arms)22546.70.4kept, disc., failed_inv.
lr_synthetic4508.446.2kept, disc., failed_inv.
openml_credit_g23013.526.1kept, disc., failed_inv.
openml_bank_marketing23013.032.2kept, disc., failed_inv.
mlagentbench_vec.5028.048.0kept, disc., failed_inv.
autoresearch (none arm)1200.0100.0failed_inv. only
autoresearch (summary)12011.740.8kept, disc., failed_inv.

Provenance completeness: 100% on every row.

Governed vs. Ungoverned Counterfactual (Table 9)

NodeNGovernedUngovernedDrops
autoresearch_linux (IR=100%)301.000.0030 (100%)
openml_bank_mkt (IR=55%)201.000.3513 (65%)

Combined: 43/50 ungoverned trials absent vs. 0/50 governed.

Large-Scale Memory Replication (225 trials, NVIDIA L40S)

Across 5 clean seeds with verified reset integrity (shared node_state_hash at trial 1):

  • Mean best AUC: none < summary ≈ rationale (0.899 vs. 0.939 and 0.941)
  • Cross-seed variability: σnone=0.009\sigma_{\text{none}} = 0.009 vs. σsummary=σrationale=0.002\sigma_{\text{summary}} = \sigma_{\text{rationale}} = 0.002 (4–5× smaller)
  • Memory arms diverge from no-memory by trial 5, hold gap through trial 15
  • Provenance completeness: 100% across all 225 trials

Real LM Training Node (autoresearch_linux)

  • None arm: IR = 1.00 across all 120 trials (4 seeds)
  • Summary arm: 3/4 seeds produce valid improving trials (mean val_bpb reduction −0.062-0.062); seed s2 suffered total worker failure
  • Across 240 trials including 151 runtime-error crashes: provenance completeness 240/240 (100%)

Theoretical and Practical Implications

Theoretical Implications

  1. Process-based evaluation paradigm: The paper argues that autonomous ML experimentation should be evaluated on governance metrics (provenance, lifecycle classification, scope enforcement, failure taxonomy) alongside task metrics—a shift from outcome-only benchmarking.

  2. Structural vs. empirical guarantees: The lifecycle invariant holds "by construction of the control-plane code, not because nothing crashed." This distinguishes structural guarantees from empirical observations—a key methodological contribution.

  3. Replaceability principle: Separating proposal (manager), execution (worker), and authority (control plane) means the LLM manager is a replaceable component, enabling controlled experiments on manager behavior without changing governance.

Practical Implications

  1. Auditability for scientific ML: Every decision can be traced to its evidence (patch, log, metric, decision), enabling reproducible scientific experimentation.

  2. Failure visibility: The ledger exposes failures that task scores hide—total failures, bounded-proposal exhaustion, and reset failures detectable via reproducibility hashes.

  3. Memory diagnostics: Repeated-bad rate serves as a per-node diagnostic for memory sensitivity, but the task-dependence warns against general claims about memory formats.

  4. Reproducibility: Released ledgers allow recomputation of all reported numbers without re-running training (no GPU required).

Conclusion

The paper demonstrates that "autonomous ML experimentation should be evaluated on governance metrics—provenance, lifecycle classification, scope enforcement, and failure taxonomy—not only task metrics." Across 1,445 governed trials on six ML nodes, the control plane classifies every outcome without silent corruption, and provenance is complete by construction.

Key limitations acknowledged by the authors:

  • Short budgets and limited node diversity
  • Structural relabeling is a consistency check, not a human inter-rater study
  • ResNet AUC gains fall within baseline seed noise
  • Human-auditor utility not yet tested

Future directions: A human-auditor utility study (do reviewers with governed ledgers detect seeded errors faster?), two-rater validation, and holdout-node replication.

Evidence strength summary (Table 10): Demonstrated properties include control-plane classification, 100% provenance, guard failure visibility, and protocol transfer across node types. Observed-but-not-claimed findings include memory ordering on AUC (5 clean seeds) and smaller cross-seed spread with memory. The claim that "rationale reduces repeated-bad rate" is explicitly not claimed due to non-monotonic results.

Related papers