MetaCurate: Evolving Curation Filters for High-Quality Agentic Coding Data

Summary (Overview)

• MetaCurate is a training-free framework that automatically evolves trajectory curation filters for supervised fine-tuning (SFT) of coding agents, using pairwise LLM judgments as feedback instead of expensive model training loops.
• The framework jointly optimizes a filter's deterministic rules, rubric criteria, and score aggregation policy through an agentic search loop, starting from either a blank or handwritten seed filter.
• MetaCurate achieves the highest mean SWE-bench Verified scores among compared methods: 64.0% on the DSv4Pro corpus and 56.5% on the Nemotron v3 pool, outperforming hand-authored rule-based filtering, LLM-based filtering, and a training-loop baseline (CurateEvo) whose search costs 3.5× more.
• The evolved filter generalizes beyond its selection set, with selection and held-out test concordance improving by nearly identical amounts (+0.067 and +0.066, respectively).
• The approach transfers to larger models: Qwen 3.5 27B trained on MetaCurate-selected data reaches 74.3% resolved rate, outperforming all baselines without repeating filter search.

Introduction and Theoretical Foundation

Background

Large language models have made rapid progress in coding, with coding agents now solving tasks through repeated interaction with environments (ReAct-style interleaving of reasoning and action). Improving a model for agentic coding involves two steps: obtaining trajectories and choosing which to learn from.

The Core Problem

The most common selection rule—keeping only trajectories whose final patch passes tests (rejection sampling)—is insufficient because:

• A resolved patch may follow inefficient reasoning or unnecessary tool use
• A failed run may still contain useful intermediate steps (sound diagnosis, useful tests)

In the authors' DSv4Pro dataset, all attempts receive the same outcome label on 90.1% of issues, so outcome alone cannot distinguish attempts. Furthermore, among audited pairs with different outcome labels, the SFT-suitability judge prefers the failed trajectory in 28.3% of comparisons.

Limitations of Existing Approaches

1. Hand-authored rule-based filters: Limited to behaviors specified in advance; require substantial human expertise to design.
2. LLM-based filters: Quality depends on prompt and rubric; scores can be unstable and model-dependent.
3. Training-based validation (Train-and-Validate): Provides direct feedback but requires a training run per candidate—expensive (CurateEvo's three candidate evaluations required 110.2 B200-hours, approximately $659).

Key Insight

Pairwise comparisons give the judge a reference for relative quality, which can be more discriminating than rating either trajectory on an absolute scale.

The authors propose using pairwise LLM judgments as a cheap, targeted feedback signal to evolve a pointwise filter that scales to full-pool scoring at deployment.

Methodology

Filter Representation

A filter is an executable scorer comprising three files:

• rules.py: Deterministic rules (e.g., flagging malformed tool calls)
• rubric.md: LLM-evaluated criteria (qualitative assessments)
• policy.py: Score aggregation policy

Concordance Objective

Each filter $F$ scores a trajectory $x$ individually, producing $s_F(x)$. For a judged pair $(x \succ y)$, let $\kappa_F(x,y)$ equal 1 if $s_F(x) > s_F(y)$, $\frac{1}{2}$ for a tie, and 0 otherwise. Concordance with a preference set $P$ is:

$$C_F(P) = \frac{1}{|P|} \sum_{(x \succ y) \in P} \kappa_F(x, y). \tag{1}$$

The objective is to maximize $C_F(P_{\mathrm{sel}})$—concordance on fixed selection pairs—subject to constraints on filter size, criterion redundancy, label use, and permitted edits.

Evolution Loop (Algorithm 1)

Algorithm 1 METACURATE loop over filters
1: Input: exploration set D_exp, selection pairs P_sel, proposer P, iterations N
2: Initialize: current filter F ← F_0
3: Initialize: run directory D ← ∅
4: for t = 1 ... N do
5:    Proposer P queries D                    ▷ prior filters, scores, feedback
6:    repeat
7:    Proposer edits F and probes the draft on D_exp
8:    Store probe results in D
9:    until probe budget is spent
10:   Proposer submits k candidate filters {F_1, ..., F_k}
11:   Precheck candidates and revise rejected candidates once
12:   for F_i ∈ {F_1, ..., F_k} do
13:   if F_i passes validation then
14:   E_i ← EVALUATE(F_i, D_exp, P_sel)
15:   D ← D ∪ {(F_i, E_i)}
16:   F ← PROMOTE(F, D)                        ▷ selection concordance, gated
17: return SFT subset selected by F

Key Components

Actionable failure feedback: The proposer receives misordered exploration pairs, judge rationales, criterion score distributions (flagging low variation), and pairwise Spearman correlations (flagging redundant criteria).

Probing: The proposer tests draft rules on exploration trajectories to inspect firing rates and scores draft rubric criteria before submission. Probing tools reject trajectories outside $D_{\mathrm{exp}}$, keeping the selection set inaccessible.

Order-consistent judgments: Every judged pair is presented in both AB and BA order to account for position bias; a preference is retained only when both presentations identify the same winner.

Noise-aware promotion: Apparent gains are screened using discordant-pair counts, and selected candidates are confirmed with fresh scoring draws. Candidates introducing new criterion-health defects face a stricter threshold.

Pointwise full-pool scoring: After search, the final filter independently scores every trajectory without pairwise judging or LLM comparisons.

Empirical Validation / Results

Setup

• DSv4Pro corpus: 4,286 trajectories generated with DeepSeek V4 Pro using mini-swe-agent from 1,415 GitHub issues
• Nemotron v3 pool: 17,220 trajectories from the mini-swe-agent subset of Nemotron v3 SWE-SFT
• Training model: Qwen 3.5 9B (default budget: 1,000 trajectories)
• MetaCurate models: GPT-5.6 Sol Pro (proposer), GPT-5.6 Luna (scorer and pairwise judge)

Main Comparison

Table 1: Downstream performance of Qwen 3.5 9B across curation methods (SWE-bench Verified resolved rates, %)

Key findings:

• MetaCurate (blank seed) achieves the highest mean scores on both pools (64.0% and 56.5%)
• Outperforms CurateEvo by 1.3 and 2.9 points respectively, without model training during search
• On DSv4Pro, the selected subset matches full-pool training (63.1%) with only 1,000 of 4,286 trajectories

Model Scale Transfer

Table 2: Transfer to Qwen 3.5 27B (same training subsets as 9B)

MetaCurate's benefit extends to 27B without repeating filter search.

Iteration Series

Table 3: Iteration series on DSv4Pro (SWE-bench Verified scores, mean ± SD over three evaluations)

• Selection and test concordance improve by nearly the same amount (+0.067 and +0.066), showing no substantial selection-set overfitting
• Downstream performance tracks concordance gains

Ablation Studies

• Seed comparison: Blank seed reaches concordance comparable to handwritten seed (0.912 vs. 0.916) and yields higher downstream scores on both pools
• Proposer strength: Sol Pro achieves higher concordance than Luna as proposer with the same solver and judge
• Judge agreement: Pairwise agreement between different LLM judges ranges from 0.90–0.96, motivating use of the cheaper Luna ($0.004/call vs.$0.14 for Sol Pro)
• Order flips: Preferences are more stable for trajectories farther apart in rank, supporting dual-order judging

Reduced Budget

Table 4: 200-trajectory budget on DSv4Pro

MetaCurate achieves the highest mean even with a smaller 200-trajectory training budget.

Theoretical and Practical Implications

Theoretical Contributions

1. Pairwise-to-pointwise transfer: Demonstrates that pairwise comparative feedback can effectively supervise the optimization of a pointwise scoring procedure, avoiding the need for calibrated absolute scores.

2. Training-free curation search: Shows that curation filters can be iteratively improved through cheap pairwise judgments rather than expensive training–evaluation loops, reducing search costs by 3.5× compared to training-based baselines.

3. Interpretable curation: The executable filter format (rules, rubric, policy) provides transparency into what trajectory properties are being selected for, unlike black-box LLM scoring.

Practical Implications

1. Cost efficiency: Pairwise judging costs approximately $0.004 per call for Luna, enabling 45 candidate evaluations versus only 3 for the training-loop baseline at similar cost.

2. Scalability: The final pointwise filter scores the full pool linearly, requiring no comparison partner or pairwise judgments at deployment time.

3. No task-specific expertise required: The blank-seed variant achieves the highest scores without handwritten criteria, democratizing high-quality data curation.

4. Transferability: Filters evolved on one pool (DSv4Pro) transfer to another (Nemotron v3) and to larger models (27B) without re-search.

Conclusion

MetaCurate introduces a training-free framework for evolving trajectory curation filters through pairwise LLM feedback. By iteratively revising deterministic rules, rubric criteria, and score aggregation, it produces an executable filter that scores trajectories independently at scale. Across two coding-trajectory datasets, the evolved filter selects subsets yielding the highest observed mean downstream scores among compared methods, demonstrating that pairwise judgments can effectively guide pointwise data selection.

Limitations and Future Work

• The current work focuses on whether evolved filters improve downstream selection rather than interpreting the curation principles that emerge during search
• Future directions include:
  • Analyzing how executable filters change over time (rules/criteria introduced, removed, or strengthened)
  • Relating filter changes to downstream training outcomes for a quantitative account of trajectory properties useful for SFT
  • Controlled analyses of individual rules to reveal whether filter-preferred behaviors are reflected in the trained agent
  • Turning MetaCurate into a tool for extracting empirical principles that inform future trajectory collection, generation, and curation