WEAVE: Weaving Real-World Code Evolution into Dependency-Structured Tasks for Benchmarking and Training Code Agents

Summary (Overview)

• WEAVE is a framework that transforms verified PR (Pull Request) histories into executable, dependency-structured tasks for benchmarking and training code agents, addressing the mismatch between real-world multi-PR development and existing single-issue benchmarks.
• The framework introduces repository-calibrated typed change loci to organize related PRs into bounded dependency DAGs (Directed Acyclic Graphs), each instantiated as a unified executable task with a base state, specification, and test oracle.
• A novel metric called Graph Credit provides deterministic, test-grounded evaluation of dependency-consistent partial progress, complementing binary resolution and flat scoring while exposing "phantom credit" in structure-agnostic evaluation.
• The framework yields 3,200 training tasks and 300 held-out benchmark tasks spanning six programming languages, with empirical validation across task construction, agent evaluation, supervised fine-tuning, and reinforcement learning.
• Results show WEAVE produces coherent executable tasks with higher dependency recovery quality (AP 37.7% vs. 26.6% best baseline), enables more faithful progress measurement, and provides effective supervision and verifiable rewards for training code agents.

Introduction and Theoretical Foundation

Background and Motivation

Real-world software engineering rarely consists of isolated changes. Features, migrations, and systematic fixes often unfold across multiple pull requests (PRs) requiring coordinated modifications across a repository. However, most executable repository-level benchmarks (e.g., SWE-bench) retain a single issue or PR as the basic task unit, which only captures isolated bug fixes and localized features.

The missing abstraction is not merely longer context or larger patches, but the dependency structure connecting multiple changes into a coherent engineering objective. Two coupled challenges remain:

1. Task coherence: Aggregation signals like temporal proximity, explicit cross-references, and coarse overlap are imperfect proxies—temporal proximity can merge unrelated changes, explicit references are sparse, and coarse overlap is dominated by frequently modified utilities or configuration files.

2. Structured partial progress evaluation: Binary resolution ignores progress on unsolved tasks, while flat test-level aggregation overlooks dependencies and may over-credit downstream behavior built on incomplete upstream changes.

Theoretical Foundation

The paper formalizes a WEAVE instance as:

$$\mathcal {I} = \left(S _ {\mathrm{base}}, q, G, \mathcal {T} _ {\mathrm{F2P}}, \mathcal {T} _ {\mathrm{P2P}}, A\right),\tag{1}$$

where $S _ { \mathrm { b a s e } }$ is the repository state presented to the agent, $q$ is a unified task specification, $G \overset { \vartriangle } { = } ( V , E )$ is the dependency DAG, $\mathcal { T } _ { \mathrm { F 2 P } }$ and $\mathcal { T } _ { \mathrm { P 2 P } }$ are functional and regression test sets, and $A$ denotes test-to-node attribution. The agent observes only $S _ { \mathrm { b a s e } }$ and $q$; individual PR descriptions, identifiers, historical decomposition, and the graph itself are not exposed.

Methodology

Repository-Calibrated DAG Construction

Typed change loci: For each PR $p$, WEAVE extracts typed change loci:

$$\mathcal {L} (p) = \mathcal {L} _ {\text { ast }} (p) \cup \mathcal {L} _ {\text { sym }} (p) \cup \mathcal {L} _ {\text { cfg }} (p) \cup \mathcal {L} _ {\text { decl }} (p) \cup \mathcal {L} _ {\text { dep }} (p).\tag{2}$$

These capture AST declarations, lightweight symbols, structured configuration/schema entries, and dependency keys—representing PRs by concrete program entities rather than files. Test files, generated/vendored code, and binary diffs are excluded.

Repository-calibrated coupling: Each locus $\ell$ receives a repository-local inverse-frequency weight:

$$w _ {R} (\ell) = \log \left(\frac {N _ {R} + 1}{\mathrm{df} _ {R} (\ell) + 1}\right) + 1.\tag{3}$$

where $N_R = |\mathcal{P}_R|$ and $\mathrm{df}_R(\ell)$ is the number of PRs touching locus $\ell$. The coupling score between PRs $p_i$ and $p_j$ is:

$$s _ {R} (p _ {i}, p _ {j}) = \frac {\sum_ {\ell \in \mathcal {L} (p _ {i}) \cap \mathcal {L} (p _ {j})} w _ {R} (\ell)}{\sum_ {\ell \in \mathcal {L} (p _ {i}) \cup \mathcal {L} (p _ {j})} w _ {R} (\ell)}.\tag{4}$$

Edge filtering and bounded DAG assembly: Candidate relations are constructed within the same repository:

$$E _ {R} ^ {\mathrm{cand}} = \{(p _ {i}, p _ {j}) | t _ {i} < t _ {j}, t _ {j} - t _ {i} \leq \Delta , s _ {R} (p _ {i}, p _ {j}) \geq \theta \}\tag{5}$$

where $\Delta$ is the search radius and $\theta$ the minimum coupling threshold. Relations from reverts, backports, release replays, and near-duplicate patches are removed. Components of at most $K$ nodes are retained.

Executable Task Instantiation

For each candidate graph $G$, WEAVE:

1. Starts from repository state $S_{\mathrm{gold}}$ containing all constituent PRs
2. Reverts their changes to construct counterfactual base state $S_{\mathrm{base}}$
3. Consolidates PR objectives into a single task specification $q$
4. Validates the oracle satisfies:

$$\mathcal {T} _ {\mathrm{F2P}} (S _ {\mathrm{base}}) = \mathrm{fail}, \qquad \mathcal {T} _ {\mathrm{F2P}} (S _ {\mathrm{gold}}) = \mathrm{pass},\tag{6}$$

Node attribution assigns each F2P test to a PR node:

$$A: \mathcal {T} _ {\mathrm{F2P}} \to V \cup \{\text { ambiguous }, \text { unattributed } \}.\tag{7}$$

Graph Credit Metric

For each evaluable node $v$ with attributed tests $\mathcal{T}_v$, local completion is:

$$\ell_ {v} = \frac {1}{| \mathcal {T} _ {v} |} \sum_ {t \in \mathcal {T} _ {v}} \mathbb {I} [ t \mathrm{passes} ].\tag{8}$$

Effective credit is defined recursively to enforce dependency consistency:

$$c _ {v} = \left\{ \begin{array}{l l} \ell_ {v}, & \text { pa } _ {G _ {A}} (v) = \varnothing , \\ \min \left\{\ell_ {v}, \min _ {u \in \text { pa } _ {G _ {A}} (v)} c _ {u} \right\}, & \text { otherwise }. \end{array} \right.\tag{9}$$

Graph Credit is the mean effective credit over evaluable nodes:

$$\mathrm{GraphCredit} (G _ {A}) = \frac {1}{| V _ {A} |} \sum_ {v \in V _ {A}} c _ {v}.\tag{10}$$

Reinforcement Learning Reward

The combined reward combines flat progress, dependency-conditioned credit, resolution, and regression preservation:

$$\begin{array}{c} R _ {\text {WEAVE}} = \lambda_ {\text {f2p}} \text {FlatF2P} + \lambda_ {\text {graph}} \text {GraphCredit} (G _ {A}) \\ + \lambda_ {\text {solve}} \text {Resolved} - \lambda_ {\text {reg}} \text {Reg} _ {\text {P2P}}. \end{array}\tag{11}$$

Empirical Validation / Results

RQ1: Construction Validity

Evaluated on 30 repositories with 1,198 validated atomic PRs and 219 gold dependency edges (Fleiss $\kappa = 0.791$):

WEAVE achieves the highest AP (37.7%), outperforming File Overlap (26.6%) by 11.1 points with significance after Holm–Bonferroni correction ($p \leq 0.009$). Notably, WEAVE achieves both the highest precision AND recall, escaping the precision–recall trade-off of single-signal baselines.

RQ2: Benchmark Performance

Agent performance on WEAVE-Bench (8 models × 2 scaffolds):

Key findings:

• Best configuration (Claude Opus 4.7 + Cline, 48.7% Resolved) outperforms worst (MiniMax-2.7 + Cline, 9.7%) by >5×
• Flat-F2P exceeds Graph Credit by 10.5 pp on average (range 7.2–16.8 pp), exposing substantial phantom credit
• Cline configurations exhibit larger OverCredit (mean 12.2 pp) than OpenCode (mean 8.6 pp)
• Replacing true DAG with node-permuted random graphs reduces the gap to <2 pp, verifying signal comes from real dependency structure

RQ3: SFT Utility and Token Efficiency

Under matched training conditions, WEAVE consistently outperforms alternatives across model scales:

• WEAVE-Bench: WEAVE improves Resolved over Atomic-PR by 3.0/3.7 pp at 4B/35B, with larger Graph Credit gains (5.4/6.2 pp)
• Transfer to SWE-bench Verified: +3.6/2.8 pp over Atomic-PR, with consistent gains on SWE-bench Pro and Multilingual
• Temporal-Window provides only modest improvements, confirming temporal proximity alone is weaker supervision than verified dependency structure
• External corpora (SWE-rebench V2, daVinci-Agency) yield substantially smaller gains, suggesting training utility depends on task organization, not data volume

RQ4: Graph-Conditioned Reinforcement Learning

Using GRPO with matched rollout budgets (8 rollouts per task):

• Qwen3.5-4B: $R_{\text{WEAVE}}$ improves Resolved by 3.7 pp over SFT and 2.0 pp over flat F2P
• Qwen3.5-35B-A3B: gains of 4.3 and 2.6 pp respectively, showing graph-conditioned reward scales with model capacity
• Graph Credit separation: +5.6/+6.2 pp over SFT at 4B/35B
• Transfer to SWE-bench Verified: +3.2/+4.2 pp over SFT
• Binary Resolved reward yields marginal gains (≤0.7 pp), consistent with sparse-signal limitations
• All settings maintain P2P regression below 2.3%

Theoretical and Practical Implications

Theoretical Contributions

1. Unified task abstraction: WEAVE demonstrates that dependency DAGs from typed change loci provide a principled foundation connecting task construction, evaluation, and training—a unified representation addressing the fragmentation of existing benchmarks.

2. Dependency-aware credit assignment: Graph Credit formalizes the concept of "phantom credit"—downstream test passes without upstream resolution—and provides a deterministic, test-grounded correction. The recursive minimum operator in Eq. (9) provides a principled mechanism for dependency-consistent credit.

3. Repository-calibrated evidence weighting: The inverse-frequency weighting (Eq. 3) demonstrates that repository-local calibration is essential for distinguishing genuine dependencies from hub-file artifacts, escaping the precision–recall trade-off inherent in single-signal approaches.

Practical Implications

1. Benchmarking: WEAVE-Bench (300 held-out tasks) provides a more realistic evaluation for long-horizon code agents, with frontier models achieving only 48.7% Resolved—qualitatively harder than single-PR benchmarks.

2. Training: WEAVE's dependency-structured supervision is more token-efficient and effective than atomic or temporal-window alternatives, and graph-conditioned rewards provide dense, verifiable signals for RL.

3. Diagnostics: OverCredit (Flat-F2P − Graph Credit) serves as a structural diagnostic exposing whether agents resolve upstream dependencies faithfully or exploit downstream shortcuts.

Conclusion

This paper addresses the fundamental limitation of current software engineering agent evaluation: the lack of a task abstraction capturing causal dependencies among multiple repository changes. WEAVE recovers dependency DAGs from verified PR histories through repository-calibrated typed-locus coupling, unifying task construction, dependency-conditioned partial credit (Graph Credit), and structured reward design.

Experiments across construction quality, agent benchmarking, supervised training, and reinforcement learning demonstrate WEAVE's effectiveness as both a challenging evaluation workbench and an effective training resource. The framework enables consistent evaluation and optimization of long-horizon code agents, with future directions including:

• Scaling to larger and more diverse repository ecosystems
• Exploring more complex dependency structures beyond DAGs
• Investigating additional training paradigms enabled by dense, verifiable rewards
• Extending the framework to other software engineering activities beyond PR-based changes