Full text not available for this paper

Summary (Overview)

  • BuggyMLE is a scalable framework for generating machine learning engineering (MLE) tasks via controlled bug injection into working ML repositories, producing over 6,600 bug-injected repository variants and ~260K lines of injected code from 222 source repositories.
  • Code Complexity Index (CCI) is a novel difficulty metric based on ten structural features (repository scale, dependency depth, patch size) that predicts average solve rates across eight public coding benchmarks with R2=0.951R^2 = 0.951, identifying patch size as the top predictor.
  • BuggyMLEBench-Lite is the first difficulty-graded, repository-level variant of MLE-Bench Lite, spanning 88 oracle-verified tasks across four controlled difficulty tiers (1–60 bugs per repository) over 22 multi-file MLE repositories. Across 14 evaluated models, the top solve rate drops from 95% to 9% from the easiest to hardest tier.
  • Curriculum learning with BuggyMLE trains a Qwen3.5-9B model that rivals frontier models on BuggyMLEBench-Lite and transfers beyond ML: SWE-bench Lite resolve rates improve from 24% to 33%, and LiveCodeBench v6 pass@1 from 69% to 71%.

Introduction and Theoretical Foundation

Background and Motivation

The paper addresses a critical bottleneck in AI-for-Research: enabling AI systems to autonomously improve AI models requires agents capable of machine learning engineering (MLE)—navigating real codebases, running training pipelines, and iteratively improving metrics. Existing MLE benchmarks face three recurring limitations:

  1. Single-file scope: Most benchmarks grade single-file solutions, sidestepping the multi-file repositories characterizing real-world MLE projects.
  2. Saturation: Many existing suites are saturated, with frontier models outperforming human reference solutions.
  3. Costly curation: Manual curation requires vetted datasets, execution harnesses, and human-verified reference outcomes.

Theoretical Foundation

The central research questions are: What makes a repository-level MLE task challenging for frontier models? How can this be measured? How can difficulty be systematically controlled?

The paper introduces CCI, a principled metric quantifying benchmark difficulty using only structural code properties. The core insight is that code structure, rather than model identity, drives coding benchmark difficulty. CCI uses a logistic regression model to predict solve rates from three feature groups, confirming the intuition that as required solution complexity increases, solve rates decline.

The framework draws on mutation-based task generation (following SERA and SWE-smith) but differs by targeting the ML domain specifically, where evaluation requires running training pipelines and measuring task-specific performance metrics.


Methodology

3.1 Code Complexity Index (CCI)

Raw features are grouped into three conceptual dimensions:

Grepo={lines_of_code,file_count,max_dir_depth}G_{repo} = \{lines\_of\_code, file\_count, max\_dir\_depth\} Gdep={dependency_count,max_dep_chain}G_{dep} = \{dependency\_count, max\_dep\_chain\} Gpatch={files_changed,hunks,lines_added,lines_removed,total_lines_delta}G_{patch} = \{files\_changed, hunks, lines\_added, lines\_removed, total\_lines\_delta\}

Preprocessing: Features are log-transformed, z-scored, and grouped:

zi=log⁡(1+fi)−μiσi,zˉG=1∣G∣∑i∈Gziz_i = \frac{\log(1 + f_i) - \mu_i}{\sigma_i}, \quad \bar{z}_G = \frac{1}{|G|} \sum_{i \in G} z_i

Modeling: Logistic regression of solve rate πm,b\pi_{m,b} on group scores:

logit(πm,b)=β0+βrepozˉrepo,b+βdepzˉdep,b+βpatchzˉpatch,blogit(\pi_{m,b}) = \beta_0 + \beta_{repo}\bar{z}_{repo,b} + \beta_{dep}\bar{z}_{dep,b} + \beta_{patch}\bar{z}_{patch,b}

CCI derivation (sign-flipped weighted sum, min-max scaled):

CCIb=−(βrepozˉrepo,b+βdepzˉdep,b+βpatchzˉpatch,b)CCI_b = -\left(\beta_{repo}\bar{z}_{repo,b} + \beta_{dep}\bar{z}_{dep,b} + \beta_{patch}\bar{z}_{patch,b}\right)

Key finding (Remark 1): Patch complexity dominates benchmark difficulty—its coefficient is the largest in magnitude, about three times that of dependency complexity.

3.2 BuggyMLE Pipeline

Four-stage configuration-driven pipeline:

  1. Injector: LLM agent explores a clean ML repository in a Docker sandbox, introducing breaking changes across the pipeline (data loading, preprocessing, model architecture, training, evaluation) subject to automated verification.
  2. Oracle: Ablation procedure isolates impactful mutations from inert decoys via two passes: (a) cumulative reversion to find a minimal reverting set; (b) isolated application of each mutation to measure its individual effect.
  3. Solver: Debugging agent receives the buggy codebase and tries to diagnose and repair bugs without clean reference code.
  4. Evaluator: Repaired solutions graded on quantitative performance recovery and oracle-guided code review.

Evaluation criteria for full repair:

  • Metric recovery: recovery≥0.99recovery \geq 0.99 where recovery=mfixed/mbaselinerecovery = m_{fixed}/m_{baseline}
  • Routing integrity: Entry point still executes mutated modules (prevents bypass)
  • Per-bug verification: LLM judge confirms each bug site resolved genuinely

3.3 Curriculum Learning

The All Difficulties recipe trains Qwen3.5-9B with LoRA adapters in two phases:

  1. SFT phase: 2,724 evaluator-verified repair trajectories across all difficulty bands from five frontier teacher models, supplemented via RFT on 411 verified rollouts from a teacher-tuned Qwen3.8-27B.
  2. RL phase: GRPO across a three-stage curriculum (Easy: 1–5 hunks; Medium: 10–12; Hard: 30–34) on 40 repositories, with rewards incorporating all evaluation criteria.

3.4 BuggyMLEBench-Lite

Four complexity tiers (1 Bug, 5 Bugs, 10 Bugs, 50 Bugs) across 22 MLE repositories. The 50 Bugs tier expands each repository to roughly twice its original size before injecting 23–60 bugs. Oracle verification confirms reverting all mutations exactly restores baseline grading scores.


Empirical Validation / Results

RQ1: Performance Scaling with Difficulty

Solve rates decline sharply across tiers (monotonically for 13 of 14 models), with top solve rate dropping from 95% to 9%. Model separation is clearest on the 10 Bugs tier, where the 9B All Difficulties agent outperforms Claude Opus 5.5 in end-to-end solves.

Table 2 highlights (Full Repair %, selected models):

Model1 Bug5 Bugs10 Bugs50 Bugs
Claude Opus 5.595.531.813.69.1
Claude Sonnet 5.590.927.331.80.0
Claude Haiku 5.577.318.20.00.0
Qwen3.5-9B (base)13.60.00.00.0
All Difficulties (9B)72.722.718.20.0

RQ2–RQ4: Curriculum Design Insights

  • Harder training tasks improve easier-task performance when embedded in a full curriculum (All Difficulties: 72.7% vs. Easy Only: 50.0% on 1 Bug).
  • Exclusive hard-task training hurts easy tasks due to distribution shift (Hard Only: 22.7% vs. 50.0% on 1 Bug).
  • Easy tasks are essential foundations: Easy Only solves 0% on 5 and 10 Bugs, but including easy stages doubles the 10 Bugs solve rate.
  • SFT vs. RL contributions: SFT establishes reliable tool use; RL improves fault localization and exploration efficiency.

RQ5: Out-of-Distribution Transfer

Table 3 highlights (OOD generalization):

ModelSWE-bench Lite Resolved (%)LiveCodeBench v6 Pass@1 (%)
Qwen3.5-9B (base)24.369.3
SFT Only18.361.3
Easy Only17.068.3
Hard Only22.069.8
All Difficulties33.071.3

The All Difficulties model shows significant improvement on SWE-bench Lite (+8.7† percentage points) and LiveCodeBench v6 (+2.0†), demonstrating that RL recovers and improves general capabilities beyond the SFT checkpoint.

RQ6–RQ7: Bug Realism and Observability

  • Realism: LLM judge rates ~50% of bugs at 4–5/5 realism; expert judged 27/30 sampled bugs as plausible. Higher realism correlates with lower resolution rates.
  • Observability: Silent bugs (affecting only test predictions) are resolved at substantially lower rates than visible bugs (degrading validation metrics) across all models.
  • Modality: Data modality has minimal impact on frontier agents; output-formatting and submission defects are hardest for most models.

Theoretical and Practical Implications

Theoretical Contributions

  1. CCI as a predictive difficulty metric: Demonstrates that structural code properties alone explain 95.1% of variance in benchmark solve rates, independent of model identity. This provides a principled basis for benchmark design and difficulty calibration.

  2. Difficulty control via mutation budget: Establishes that task difficulty can be systematically controlled by scaling the number of injected bugs, enabling construction of tasks at any target difficulty without changing the underlying repository.

  3. Curriculum learning insights: Shows that difficulty-graded curricula (easy-to-hard) are superior to single-difficulty training, and that RL recovers general capabilities degraded by domain-specialized SFT.

Practical Implications

  • Benchmark design: BuggyMLEBench-Lite provides an unsaturated benchmark at high difficulty tiers, addressing the saturation problem plaguing existing suites.
  • Scalable training data: The framework generates dense, verifiable rewards for SFT and RL, enabling training of capable agents without human-curated data.
  • Agent training: Demonstrates that training on repository-level ML debugging transfers to general software engineering tasks (SWE-bench Lite) and algorithmic coding (LiveCodeBench v6).

Conclusion

The paper establishes that code structure, not model identity, drives coding benchmark difficulty. The CCI captures this relationship using ten structural features and four parameters, accurately tracking solve rates across eight diverse public benchmarks. BuggyMLE operationalizes this insight by scaling repository difficulty via controllable mutation budgets, yielding BuggyMLEBench-Lite—a repository-level benchmark remaining unsaturated at its highest tiers. Training a 9B agent on an easy-to-hard BuggyMLE curriculum allows it to rival frontier models on 10 Bugs and generalize to broader code repair.

Limitations:

  • CCI validation requires broader coverage, including independently generated synthetic sets (BuggyMLE tasks were excluded to prevent circularity).
  • CCI relies purely on structural features; it does not capture semantic complexity and is less predictive on standalone-program benchmarks.
  • Transfer to from-scratch ML solution generation yields mixed results, suggesting the need for direct optimization on solution quality (e.g., leaderboard-percentile rewards).

Future directions include broader CCI validation, integrating semantic complexity measures, and extending training to improve from-scratch ML generation capabilities.

Related papers