Full text not available for this paper
Summary (Overview)
- Core Contribution: Pro²Agent introduces the Problem State, an explicit representation of what problems limit AI research performance, which candidate solutions likely address them, and the supporting evidence/confidence—shifting the primary state of autonomous AI research from a codebase state to a problem-centered state.
- State-of-the-Art Results: Achieves 76.0% medal rate on the complete MLE-Bench with a 48-hour budget and 70.7% with the standard 24-hour budget, the highest reported on the complete benchmark.
- Transfers Across Domains: Achieves competitive performance on PostTrainBench (30.69 weighted average), comparable to Claude Code powered by Opus 4.7, despite using the lightweight Gemini 3.5 Flash as the implementation model.
- Stronger Initial Solutions: On MLE-Bench Lite, Pro²Agent's first complete solution reaches a 57.6% Any Medal Rate (vs. 28.8% for MARS), with a normalized bronze gap of −0.57 (vs. 0.96 for MARS), meaning the average first solution starts inside the medal region.
- Efficient Convergence: Reaches medal-level performance within 5 high-quality scientific iterations, while MARS typically evaluates more than 70 repository variants without consistently reaching comparable scores.
Introduction and Theoretical Foundation
Background and Motivation
Large language model (LLM) agents can now autonomously conduct AI research by writing code and running experiments. However, existing agents typically formulate model development as search or optimization over candidate code solutions, iteratively optimizing a codebase while ignoring the problem structure behind the task. This formulation works on simpler tasks but struggles on difficult problems because:
- The agent never explicitly represents its understanding of why the task remains difficult
- The agent may repeatedly propose locally reasonable code changes while staying within the wrong solution region
- Experimental results are used only to accept/reject code changes, without updating a higher-level explanation of the problem
Key Insight
"A codebase state is necessary for autonomous AI research, but it provides only a partial view of the research process. It records what has been implemented, how the current solution performs, and which modifications have already been attempted. However, it does not explicitly capture the agent's evolving understanding of why the task remains difficult."
Theoretical Foundation
The core theoretical contribution is the formalization of the Problem State as a nested structure:
where:
- = the -th identified performance-limiting problem
- = supporting evidence from diagnostic profiling
- = the -th candidate solution for problem
- = dynamic belief confidence score
- = solution evidence set from literature or empirical traces
This contrasts with traditional code-space optimization:
where is the space of valid code implementations and is a performance metric.
Methodology
Three-Phase Framework
Phase 1: Problem State Initialization
- Dynamic runtime profiling: Interactive execution probes inside a sandbox extract processing latencies, memory footprints, and domain-specific structural signals (including visual diagnosis for multimodal tasks)
- Bottleneck-conditioned literature review: Maps identified physical constraints to targeted web/academic search queries, retrieving state-of-the-art architectures and augmentations
- Modular baseline construction: Builds under strict security and profiling constraints, with validation evaluation decoupled as an HTTP grading service to prevent target leakage
Phase 2: Scientific Exploration Phase
Organized as a resource-bounded best-first search over a tree of hypothesis branches. Key components:
Hypothesis Prioritization (frequency-penalized score):
where is the belief score and counts historical exploration attempts targeting problem . This penalizes repetitive failures and forces broader search.
Belief Update Logic:
- Supported Hypothesis ():
- Falsified/Unstable Hypothesis ( or crash):
Single-Variable Control: Strictly enforced at both code and architecture levels, with a Reviewer Agent rejecting any updates introducing confounding variables.
Phase 3: Engineering Optimization Phase
Models hyperparameter tuning as Monte Carlo Tree Search (MCTS) with:
Audit Firewall (pipeline invariance):
Cost-Aware Reward Function:
where is the validation performance gain, is execution duration in seconds, and is a calibrated penalty weight.
Tripartite Multi-Agent Loop
Three specialized roles ensure reliable implementation:
| Agent | Role | Model |
|---|---|---|
| PI Agent () | Supervisor maintaining Problem State, formulating goals, inspecting trajectories every 10 steps | Gemini 3.1 Pro |
| Implementation Agent () | Engineer in sandboxed environment, ReAct loop with file/diagnostic tools | Gemini 3.5 Flash |
| Reviewer Agent () | Code auditor enforcing single-variable isolation (Phase 2) and guarding core modules (Phase 3) | Gemini 3.5 Flash |
Additional components include a dynamic budgeting protocol with an automated monitoring sentinel and an Experience Bank () for persistent long-term memory with retrieve_experience and write_experience operations.
Empirical Validation / Results
MLE-Bench (Complete 75-Task Benchmark)
Table 1: Performance comparison on complete MLE-Bench
| Agent | Backbone LLM | Compute | Valid | Above Median | Bronze | Silver | Gold | Any Medal |
|---|---|---|---|---|---|---|---|---|
| ML-Master | DeepSeek-R1 | A100, 12h | 93.3 | 44.9 | 4.4 | 7.6 | 17.3 | 29.3 |
| R&D-Agent | GPT-5 | V100, 12h | 53.3 | 40.4 | 6.7 | 12.0 | 16.4 | 35.1 |
| InternAgent | DeepSeek-R1 | A800, 12h | 96.4 | 48.4 | 7.1 | 10.7 | 18.7 | 36.4 |
| Famou-Agent | Gemini 2.5 Pro | A800, 24h | 96.9 | 51.6 | 8.4 | 12.4 | 22.7 | 43.6 |
| Kapso | Gemini 3 Pro Preview | –, 24h | – | – | – | – | – | 50.7 |
| ML-Master 2.0 | DeepSeek-V3.2-Speciale | 2×4090, 24h | 95.6 | 63.1 | 11.1 | 25.8 | 19.6 | 56.4 |
| MARS | Gemini 3 Pro Preview | A100, 24h | 98.7 | 65.8 | 9.3 | 15.6 | 31.1 | 56.0 |
| MARS+ | Gemini 3 Pro Preview | 2×H100, 24h | 100.0 | 74.2 | 12.4 | 16.4 | 33.8 | 62.7 |
| AIBuildAI | Claude Opus 4.6 | A100, 24h | 100.0 | 71.1 | – | – | 25.8 | 63.1 |
| Famou-Agent 2.0 | Gemini 3 Pro Preview | A800, 24h | – | – | – | – | – | 64.4 |
| Iris | Claude Opus 4.6 | –, 12h | – | – | – | – | – | 64.9 |
| MLEvolve | Gemini 3.1 Pro Preview | H200, 12h | 100.0 | 76.0 | – | – | 34.7 | 65.3 |
| ScienceFlow | DeepSeek-V4-Flash-Preview | ≤2 GPUs, 24h | – | – | – | – | – | 70.2 |
| AIBuildAI-2 | Claude Opus 4.7 | A100, 24h | – | – | – | – | – | 70.7 |
| Pro²Agent | Gemini 3.5 Flash‡ | A100, 24h | 100.0 | 84.0 | 25.3 | 21.3 | 24.0 | 70.7 |
| Pro²Agent | Gemini 3.5 Flash‡ | A100, 48h | 100.0 | 84.0 | 29.3 | 21.3 | 25.3 | 76.0 |
PostTrainBench (28 Configurations)
Table 2: Performance on complete PostTrainBench
| Method | Avg. | AIME 2025 | ArenaHard | BFCL | GPQA | GSM8K | HealthBench | HumanEval |
|---|---|---|---|---|---|---|---|---|
| Official Instruct Models | 51.14 | 29.17 | 70.21 | 85.00 | 36.21 | 87.00 | 43.32 | 71.49 |
| Fable 5 (Claude Code) | 41.79±1.7 | 13.33±2.4 | 61.46±10.8 | 73.00±32.9 | 28.35±2.7 | 83.53±1.9 | 40.12±9.4 | 58.38±13.8 |
| GPT 5.6 (Sol, Codex) | 36.23±0.2 | 7.50±3.5 | 26.40±13.5 | 94.38±1.2 | 30.13±1.2 | 69.48±2.7 | 27.39±3.4 | 63.95±2.7 |
| Opus 4.7 (Claude Code) | 28.56±1.7 | 6.39±3.2 | 19.59±18.1 | 76.91±28.0 | 27.48±4.0 | 60.72±17.0 | 17.81±8.7 | 42.07±22.9 |
| Gemini 3.1 Pro (OpenCode) | 21.99±1.8 | 3.89±1.9 | 7.42±5.4 | 63.59±27.7 | 18.53±8.3 | 45.51±22.3 | 16.33±9.8 | 40.19±8.4 |
| Pro²Agent (Gemini 3.5 Flash)‡ | 30.69±3.1 | 5.75±4.4 | 28.59±12.2 | 78.60±17.8 | 26.45±3.3 | 64.47±5.1 | 19.22±6.9 | 50.63±16.6 |
Controlled Studies on MLE-Bench Lite
Table 3: Initial solution quality and node efficiency
| Method | Above Median | Any Medal | Gold | ↓ | Build Time (min) | Medal@3 | Medal@5 | Node AUC ↑ |
|---|---|---|---|---|---|---|---|---|
| Direct Coding Loop | 25.8±1.5 | 12.1±3.0 | 4.5±0.0 | 1.56±0.09 | 2.1±1.2 | 21.2±3.0 | 30.3±3.0 | −0.60±0.07 |
| Search-Augmented | 39.4±3.0 | 28.8±4.0 | 12.1±1.5 | 1.23±0.07 | 8.4±2.3 | 30.3±1.5 | 36.4±5.5 | −0.44±0.12 |
| MARS | 50.0±2.6 | 28.8±4.0 | 12.1±1.5 | 0.96±0.20 | 14.8±2.6 | 34.8±3.0 | 39.4±5.5 | −0.39±0.09 |
| Pro²Agent | 81.8±2.6 | 57.6±1.5 | 27.3±0.0 | −0.57±0.05 | 11.2±4.4 | 63.6±2.6 | 66.7±1.5 | 0.58±0.17 |
The normalized bronze gap is defined as:
where and denote the bronze threshold and leaderboard median, and indicates medal-level performance.
Ablation Study
Table 4: Ablation of the Problem State
| Variant | Initial Medal | Final Medal | Node AUC ↑ |
|---|---|---|---|
| Codebase State | 10.6±1.5 | 56.1±3.0 | 0.06 |
| Prob. State w/o Grounding | 47.0±1.5 | 77.3±2.6 | 0.27 |
| Static Prob. State | 56.1±1.5 | 83.3±3.0 | 0.45 |
| Full Pro²Agent | 57.6±1.5 | 89.4±1.5 | 0.66 |
Three-State Bottleneck Taxonomy
Across diverse trajectories, Pro²Agent's Problem State provides a structured taxonomy of research outcomes:
- Resolved Bottleneck: A diagnosed bottleneck is experimentally mitigated enough that it is no longer the dominant limitation (e.g.,
bms-molecular-translation,gemma3-4b-gsm8k) - Resource-Limited Bottleneck: The bottleneck is correctly diagnosed and supported by literature, but the intervention cannot be trained to convergence within available compute (e.g.,
rsna-breast-cancer-detection) - Unresolved Bottleneck: Experiments narrow the likely failure source but do not establish an effective intervention (e.g.,
qwen3-1.7b-aime)
Theoretical and Practical Implications
Theoretical Implications
- Problem State as First-Class State: The paper demonstrates that representing understanding of why a task is difficult, rather than just what code has been tried, fundamentally changes how autonomous research agents explore solution spaces.
- Experimentation as Hypothesis Testing: Instead of using experiments only to accept/reject code changes, Pro²Agent uses experiments to test and revise its understanding of the problem—turning local code improvements into a directed research process.
- Evidence-Grounded Belief Revision: The explicit separation of problems, candidate solutions, and evidence enables principled belief updates that adapt to new experimental outcomes.
Practical Implications
- Stronger Initial Solutions: Grounding design choices in explicit problem understanding rather than LLM preferences produces substantially better first solutions (57.6% vs. 28.8% medal rate on first solution).
- Compute Efficiency: Hypothesis-guided iteration concentrates compute on scientifically distinct decisions rather than many local code modifications, reaching medal-level performance within 5 iterations vs. 70+ for MARS.
- Model Efficiency: The tripartite architecture uses a lightweight model (Gemini 3.5 Flash) for implementation while reserving a stronger model (Gemini 3.1 Pro) for high-level diagnosis—achieving results comparable to systems using frontier models like Claude Opus 4.7.
- Transparency and Reusability: Each run produces an inspectable record of diagnosed bottlenecks, tested hypotheses, and outcomes, enabling human researchers to audit and reuse the research process.
Conclusion
Pro²Agent introduces a paradigm shift in autonomous AI research: treating the agent's understanding of the problem as a first-class research state. Instead of representing progress only through the current codebase, evaluation scores, and modification history, Pro²Agent maintains a Problem State that records performance-limiting problems, candidate solutions, supporting evidence, and confidence. This changes experimentation from direct code selection into a process for testing explanations of why performance is limited.
Key findings:
- On MLE-Bench: 76.0% medal rate (48h) and 70.7% (24h), the highest reported on the complete benchmark
- On PostTrainBench: Comparable to Claude Code with Opus 4.7 while using a lighter implementation model
- Ablations confirm that evidence grounding and continual Problem State updates contribute beyond static planning
Future directions:
- Scaling Problem Probing to longer horizons and parallel hypothesis testing
- Exploring other model allocations in the tripartite architecture
- Richer belief revision mechanisms (probabilistic, noise-aware updates)
- Reusable research knowledge across tasks, enabling agents that accumulate scientific understanding rather than only code
"These findings suggest that progress in autonomous AI research depends not only on stronger language models or larger code-search budgets, but also on maintaining an explicit, evidence-grounded, and revisable understanding of the problem."
Related papers
- BuggyMLE: Scaling Machine Learning Engineering Difficulty with Bug Injection
BuggyMLE generates 6,600+ bug-injected ML repositories and a structural complexity metric (CCI) predicting solve rates with R²=0.951, enabling difficulty-graded benchmarks where a 9B model rivals frontier agents.
- Do Halting Heads Really Save Compute in Looped Language Models?
Halting heads in looped language models save far less compute than reported because mixture-trained heads rarely stop tokens and batching overhead eliminates most remaining gains.
- When Terminal-Agent Training Stalls: Demystifying Data Generation and Verification Challenges
Structural validity alone does not guarantee benchmark validity; task difficulty calibration is critical, as adding hard tasks restores informative RL training failures.