# TraceML: An Empirical Analysis of Human-Agent Planning in Machine Learning Development

> TraceML, a version-level trajectory dataset pairing human and agent Kaggle work, reveals that agents and humans develop solutions through fundamentally different process patterns, not just outcome gaps.

- **Source:** [arXiv](https://arxiv.org/abs/2608.26086)
- **Published:** 2026-09-05
- **Permalink:** https://picx.dev/p/dq20UX
- **Whiteboard:** https://picx.dev/p/dq20UX/image

## Summary

## Summary (Overview)

- **TraceML** is a novel version-level trajectory dataset that pairs human and agent work on the same Kaggle competitions, enabling process-level comparison of machine learning development rather than outcome-only benchmarking.
- The dataset contains **4,465 human trajectories** across 134 competitions and **207 agent trajectories** (Codex and MLEvolve) across 7 paired competitions, with every code version labeled for action, intent, edit size, and score effect.
- Key finding: **agents and humans develop ML solutions differently, and not as one clean gap** — Codex collapses into a submission-tuning loop (re-weighting ensembles), while MLEvolve mutates models in place without pivoting at human rates or reopening abandoned work.
- A **planning harness** (a ~1,000-token prompt) distilled from human practice moves specific behaviors toward the human profile and improves scores in 5/7 competitions, but the effort profile stays "agent-shaped" — instruction closes only the part of the gap that reduces to instructions.
- The corpus, schema, labelers, and extraction pipeline are released at https://huggingface.co/datasets/jerryyan/TraceML.

## Introduction and Theoretical Foundation

The paper addresses a fundamental limitation of outcome-based benchmarks for LLM agents in machine learning development. While LLMs write correct code for isolated problems, autonomous ML development requires agents to load and clean data, choose and train models, read validation signal, and revise approaches over many hours. On Kaggle-style tasks, agents make steady progress but still finish below strong human competitors.

**The core problem**: Outcome-based benchmarks grade the final submission without seeing the sequence of edits behind it, so two runs with the same score look identical even when one experimented carefully and the other tuned blindly. This makes it impossible to diagnose *why* agents fall short.

**The theoretical foundation** is that the gap between human and agent performance is a question about **process**, not just outcomes. TraceML represents every run as an ordered sequence of code versions, each with:
- Leaderboard score
- Timestamp
- Labels for what the code contains
- Transitions labeled for action, intent, edit size, and score effect

The same schema covers human Kaggle submissions and agent runs, so a human and agent working the same competition can be read side by side.

## Methodology

### Human Trajectory Collection (4-stage pipeline)
1. **Ingestion and alignment**: Extract public saved versions for 134 competitions, deduplicate by hash, join to author tier and public leaderboard score.
2. **Lineage reconstruction**: Recover development order as a DAG over notebook histories, fork relationships, and code-similarity links.
3. **Pruning**: Drop post-deadline edits, shallow/unscored lineages, and score-fishing resubmissions.
4. **Normalization**: Write surviving trajectories into the version-level format.

### Agent Trajectory Collection
- **Codex** (codex-cli 0.146.0): single edit-run-observe loop, tracked via sidecar Git commits.
- **MLEvolve**: evolutionary search agent growing a search tree, linearized into root-to-leaf branches.
- Both run on the same gpt-5.4-mini backend, same task prompt, same 12-hour wall-clock budget — leaving search topology as the difference.

### Process-Level Annotation Schema
- **Version state**: 8 coarse ML-pipeline stages (e.g., feature engineering, ensembling) with a 136-tag fine vocabulary.
- **Transitions** (each labeled from the diff and grader):
  - **Action**: multi-label set of operations
  - **Intent**: purpose of the edit
  - **Magnitude**: how much code is rewritten
  - **Score-Effect**: improve / plateau / regress

### Labeler Training
Hand-labeling 151,088 versions is infeasible, so the schema is distilled into open-weight labelers: a teacher (gpt-5.4-mini) emits schema-constrained labels, which train two Qwen3-1.7B students (one for version state, one for transition action). Reliability is checked via schema compliance, agreement with an independent annotator, and behavioral checks.

## Empirical Validation / Results

### Action Profiles
- **Coarse grain**: PCA of per-trajectory coarse-action distributions shows partial separation — MLEvolve sits 0.09–0.12 bits (JSD) from every human cohort; Codex is about as close to top humans as human cohorts are to each other.
- **Fine grain**: Humans spread edits across the vocabulary; each scaffold settles into one band:
  - Codex re-weights ensembles, stacks models, tweaks post-processing at several times the human rate.
  - MLEvolve mutates the model in place (averaging seeds, editing layers, changing epochs).
  - Both miss checkpoint swaps, pretrained source changes, and verification re-runs by an order of magnitude.

### Pivot Rates
| Cohort | Pivot rate (fraction of transitions) |
|---|---|
| Humans | 25% |
| Codex | 9% |
| MLEvolve | 58% |

Frequent pivots don't mean good ones: matched humans average +0.089 on the 3 steps after a pivot, MLEvolve averages −0.008 (gains and losses cancel).

### Return Behavior
- Top humans return to earlier work on 9% of eligible versions; 78% of those returns end above the version they returned to.
- Codex returns once across all runs; MLEvolve never returns. The agents lack **memory**, not recovery — Codex recovers from setbacks at rates above top humans but never reopens abandoned lines.

### Ensembling
- All three cohorts ensemble, but 78% of Codex's ensemble edits re-weight a member set it never grows; MLEvolve mostly averages seeds; top humans add new members.
- Within human runs, an ensemble step that adds/changes a member raises the chance the next version improves by **+6.4 points**; one that only re-weights lowers it by **−5.8 points**. For Codex, neither kind moves anything.

### Planning Harness Results
- **Behaviors that moved**: re-weighting fell roughly fivefold; small early edits rose from near zero to above the top human rate; attention shifted toward ensembling.
- **Where it missed**: the plain hold-out went to zero (overshoot — a ban gives direction but no destination); K-fold averaging and persisted OOF predictions were already at human rates (saturation).
- **Scores**: 5/7 competitions improved, 2 within noise, none regressed. Removing planning content while keeping injection schedule lands at or below baseline everywhere.

## Theoretical and Practical Implications

The paper reframes the human-agent gap from a single outcome number into **named behaviors** that can be targeted:

1. **Memory as a design problem**: Agents never return to abandoned work because they lack their own history in a searchable form. This suggests building retrieval over a run's earlier states — a concrete engineering target, not a model-capability problem.

2. **Control as a design problem**: Both scaffolds miss the human pivot rate from opposite sides, yet each follows one policy throughout. What's needed is a *controller* that reads where the run stands, rather than a stronger base model.

3. **The probe insight**: A prompt is useful less as a fix and more as a probe — it marks the boundary between what instruction can reach and what it cannot. Practices transfer when the instruction names a level the agent hasn't already passed; they fail when the instruction is a direction with no destination (prohibitions) or when the agent already stands beyond the human value.

4. **Benchmark design**: The human corpus is fixed while agents keep changing, so it can serve as a reference that new systems are read against as they appear, rather than a benchmark that ages with them.

## Conclusion

TraceML pairs human and agent ML development on the same competitions under one version-level schema, enabling comparison through the work behind a submission rather than the submission alone. The paired data shows agents and experts developing differently — not as one clean gap, but as two scaffolds missing the human profile from opposite sides: one tuning without changing direction (Codex) and the other changing direction without consolidating (MLEvolve).

A planning prompt built from these diagnostics moves the behaviors that reduce to a yes/no check and leaves the rest — which is where instruction stops and agent design begins. The corpus, schema, labelers, and extraction pipeline are released so new agents can be read against human practice as they appear.

**Future directions** center on building memory (retrieval over a run's earlier states) and control (a controller that adapts policy based on run state), both now cheap to test with the released pipeline.

---

_Markdown view of https://picx.dev/p/dq20UX, served by PicX — AI-generated visual whiteboard summaries of research papers._