# WEAVE: Weaving Real-World Code Evolution into Dependency-Structured Tasks for Benchmarking and Training Code Agents

> WEAVE turns real multi-PR histories into dependency-structured code agent tasks, improving dependency recovery and training while exposing phantom credit in flat evaluation.

- **Source:** [arXiv](https://arxiv.org/abs/2610.22039)
- **Published:** 2026-10-17
- **Permalink:** https://picx.dev/p/EqT5MV
- **Whiteboard:** https://picx.dev/p/EqT5MV/image

## Summary

# WEAVE: Weaving Real-World Code Evolution into Dependency-Structured Tasks for Benchmarking and Training Code Agents

## Summary (Overview)

- **WEAVE** is a framework that transforms verified PR (Pull Request) histories into executable, dependency-structured tasks for benchmarking and training code agents, addressing the mismatch between real-world multi-PR development and existing single-issue benchmarks.
- The framework introduces **repository-calibrated typed change loci** to organize related PRs into bounded dependency DAGs (Directed Acyclic Graphs), each instantiated as a unified executable task with a base state, specification, and test oracle.
- A novel metric called **Graph Credit** provides deterministic, test-grounded evaluation of dependency-consistent partial progress, complementing binary resolution and flat scoring while exposing "phantom credit" in structure-agnostic evaluation.
- The framework yields **3,200 training tasks and 300 held-out benchmark tasks** spanning six programming languages, with empirical validation across task construction, agent evaluation, supervised fine-tuning, and reinforcement learning.
- Results show WEAVE produces coherent executable tasks with higher dependency recovery quality (AP 37.7% vs. 26.6% best baseline), enables more faithful progress measurement, and provides effective supervision and verifiable rewards for training code agents.

## Introduction and Theoretical Foundation

### Background and Motivation

Real-world software engineering rarely consists of isolated changes. Features, migrations, and systematic fixes often unfold across **multiple pull requests (PRs)** requiring coordinated modifications across a repository. However, most executable repository-level benchmarks (e.g., SWE-bench) retain a **single issue or PR** as the basic task unit, which only captures isolated bug fixes and localized features.

The missing abstraction is not merely longer context or larger patches, but the **dependency structure** connecting multiple changes into a coherent engineering objective. Two coupled challenges remain:

1. **Task coherence**: Aggregation signals like temporal proximity, explicit cross-references, and coarse overlap are imperfect proxies—temporal proximity can merge unrelated changes, explicit references are sparse, and coarse overlap is dominated by frequently modified utilities or configuration files.

2. **Structured partial progress evaluation**: Binary resolution ignores progress on unsolved tasks, while flat test-level aggregation overlooks dependencies and may over-credit downstream behavior built on incomplete upstream changes.

### Theoretical Foundation

The paper formalizes a WEAVE instance as:

$$
\mathcal {I} = \left(S _ {\mathrm{base}}, q, G, \mathcal {T} _ {\mathrm{F2P}}, \mathcal {T} _ {\mathrm{P2P}}, A\right),\tag{1}
$$

where $S _ { \mathrm { b a s e } }$ is the repository state presented to the agent, $q$ is a unified task specification, $G \overset { \vartriangle } { = } ( V , E )$ is the dependency DAG, $\mathcal { T } _ { \mathrm { F 2 P } }$ and $\mathcal { T } _ { \mathrm { P 2 P } }$ are functional and regression test sets, and $A$ denotes test-to-node attribution. The agent observes only $S _ { \mathrm { b a s e } }$ and $q$; individual PR descriptions, identifiers, historical decomposition, and the graph itself are not exposed.

## Methodology

### Repository-Calibrated DAG Construction

**Typed change loci**: For each PR $p$, WEAVE extracts typed change loci:

$$
\mathcal {L} (p) = \mathcal {L} _ {\text { ast }} (p) \cup \mathcal {L} _ {\text { sym }} (p) \cup \mathcal {L} _ {\text { cfg }} (p) \cup \mathcal {L} _ {\text { decl }} (p) \cup \mathcal {L} _ {\text { dep }} (p).\tag{2}
$$

These capture AST declarations, lightweight symbols, structured configuration/schema entries, and dependency keys—representing PRs by concrete program entities rather than files. Test files, generated/vendored code, and binary diffs are excluded.

**Repository-calibrated coupling**: Each locus $\ell$ receives a repository-local inverse-frequency weight:

$$
w _ {R} (\ell) = \log \left(\frac {N _ {R} + 1}{\mathrm{df} _ {R} (\ell) + 1}\right) + 1.\tag{3}
$$

where $N_R = |\mathcal{P}_R|$ and $\mathrm{df}_R(\ell)$ is the number of PRs touching locus $\ell$. The coupling score between PRs $p_i$ and $p_j$ is:

$$
s _ {R} (p _ {i}, p _ {j}) = \frac {\sum_ {\ell \in \mathcal {L} (p _ {i}) \cap \mathcal {L} (p _ {j})} w _ {R} (\ell)}{\sum_ {\ell \in \mathcal {L} (p _ {i}) \cup \mathcal {L} (p _ {j})} w _ {R} (\ell)}.\tag{4}
$$

**Edge filtering and bounded DAG assembly**: Candidate relations are constructed within the same repository:

$$
E _ {R} ^ {\mathrm{cand}} = \{(p _ {i}, p _ {j}) | t _ {i} < t _ {j}, t _ {j} - t _ {i} \leq \Delta , s _ {R} (p _ {i}, p _ {j}) \geq \theta \}\tag{5}
$$

where $\Delta$ is the search radius and $\theta$ the minimum coupling threshold. Relations from reverts, backports, release replays, and near-duplicate patches are removed. Components of at most $K$ nodes are retained.

### Executable Task Instantiation

For each candidate graph $G$, WEAVE:
1. Starts from repository state $S_{\mathrm{gold}}$ containing all constituent PRs
2. Reverts their changes to construct counterfactual base state $S_{\mathrm{base}}$
3. Consolidates PR objectives into a single task specification $q$
4. Validates the oracle satisfies:

$$
\mathcal {T} _ {\mathrm{F2P}} (S _ {\mathrm{base}}) = \mathrm{fail}, \qquad \mathcal {T} _ {\mathrm{F2P}} (S _ {\mathrm{gold}}) = \mathrm{pass},\tag{6}
$$

**Node attribution** assigns each F2P test to a PR node:

$$
A: \mathcal {T} _ {\mathrm{F2P}} \to V \cup \{\text { ambiguous }, \text { unattributed } \}.\tag{7}
$$

### Graph Credit Metric

For each evaluable node $v$ with attributed tests $\mathcal{T}_v$, local completion is:

$$
\ell_ {v} = \frac {1}{| \mathcal {T} _ {v} |} \sum_ {t \in \mathcal {T} _ {v}} \mathbb {I} [ t \mathrm{passes} ].\tag{8}
$$

Effective credit is defined recursively to enforce dependency consistency:

$$
c _ {v} = \left\{ \begin{array}{l l} \ell_ {v}, & \text { pa } _ {G _ {A}} (v) = \varnothing , \\ \min \left\{\ell_ {v}, \min _ {u \in \text { pa } _ {G _ {A}} (v)} c _ {u} \right\}, & \text { otherwise }. \end{array} \right.\tag{9}
$$

Graph Credit is the mean effective credit over evaluable nodes:

$$
\mathrm{GraphCredit} (G _ {A}) = \frac {1}{| V _ {A} |} \sum_ {v \in V _ {A}} c _ {v}.\tag{10}
$$

### Reinforcement Learning Reward

The combined reward combines flat progress, dependency-conditioned credit, resolution, and regression preservation:

$$
\begin{array}{c} R _ {\text {WEAVE}} = \lambda_ {\text {f2p}} \text {FlatF2P} + \lambda_ {\text {graph}} \text {GraphCredit} (G _ {A}) \\ + \lambda_ {\text {solve}} \text {Resolved} - \lambda_ {\text {reg}} \text {Reg} _ {\text {P2P}}. \end{array}\tag{11}
$$

## Empirical Validation / Results

### RQ1: Construction Validity

Evaluated on 30 repositories with 1,198 validated atomic PRs and 219 gold dependency edges (Fleiss $\kappa = 0.791$):

| Method | AP↑ | Coverage↑ | P@50↑ |
|--------|-----|-----------|-------|
| Temporal Window | 5.1 | 73.1 | 2.0 |
| Traceability Links | 17.2 | 12.8 | 28.0 |
| File Overlap | 26.6 | 73.5 | 10.0 |
| Symbol Overlap | 24.0 | 69.0 | 30.0 |
| **WEAVE (Ours)** | **37.7**** | **84.5** | **38.0** |

WEAVE achieves the highest AP (37.7%), outperforming File Overlap (26.6%) by 11.1 points with significance after Holm–Bonferroni correction ($p \leq 0.009$). Notably, WEAVE achieves both the highest precision AND recall, escaping the precision–recall trade-off of single-signal baselines.

### RQ2: Benchmark Performance

Agent performance on WEAVE-Bench (8 models × 2 scaffolds):

| Scaffold | Model | Resolved↑ | Flat-F2P↑ | GraphCr.↑ | OverCr.↓ |
|----------|-------|-----------|-----------|-----------|----------|
| OpenCode | MiniMax-2.7 | 12.3 | 24.2 | 13.4 | 10.8 |
| OpenCode | DeepSeek-V4-Flash | 21.0 | 21.8 | 14.6 | 7.2 |
| OpenCode | Kimi-K2.5 | 14.0 | 12.4 | 5.1 | 7.3 |
| OpenCode | Kimi-K2.6 | 29.3 | 30.1 | 21.4 | 8.7 |
| OpenCode | GLM-5.1 | 28.7 | 24.1 | 14.5 | 9.6 |
| OpenCode | GLM-5.2 | 34.0 | 32.5 | 23.8 | 8.7 |
| OpenCode | Claude Opus 4.6 | 30.3 | 24.9 | 15.8 | 9.1 |
| OpenCode | Claude Opus 4.7 | 41.0 | 45.6 | 37.2 | 8.4 |
| Cline | MiniMax-2.7 | 9.7 | 19.1 | 10.5 | 8.6 |
| Cline | DeepSeek-V4-Flash | 34.3 | 58.1 | 41.3 | 16.8 |
| Cline | Kimi-K2.5 | 18.7 | 27.5 | 16.8 | 10.7 |
| Cline | Kimi-K2.6 | 37.3 | 69.1 | 57.2 | 11.9 |
| Cline | GLM-5.1 | 21.0 | 18.8 | 11.6 | 7.2 |
| Cline | GLM-5.2 | 43.0 | 59.5 | 44.1 | 15.4 |
| Cline | Claude Opus 4.6 | 36.3 | 61.3 | 47.6 | 13.7 |
| Cline | Claude Opus 4.7 | **48.7** | 64.4 | 50.8 | 13.6 |

Key findings:
- Best configuration (Claude Opus 4.7 + Cline, 48.7% Resolved) outperforms worst (MiniMax-2.7 + Cline, 9.7%) by >5×
- Flat-F2P exceeds Graph Credit by 10.5 pp on average (range 7.2–16.8 pp), exposing substantial **phantom credit**
- Cline configurations exhibit larger OverCredit (mean 12.2 pp) than OpenCode (mean 8.6 pp)
- Replacing true DAG with node-permuted random graphs reduces the gap to <2 pp, verifying signal comes from real dependency structure

### RQ3: SFT Utility and Token Efficiency

Under matched training conditions, WEAVE consistently outperforms alternatives across model scales:

- **WEAVE-Bench**: WEAVE improves Resolved over Atomic-PR by 3.0/3.7 pp at 4B/35B, with larger Graph Credit gains (5.4/6.2 pp)
- **Transfer to SWE-bench Verified**: +3.6/2.8 pp over Atomic-PR, with consistent gains on SWE-bench Pro and Multilingual
- Temporal-Window provides only modest improvements, confirming temporal proximity alone is weaker supervision than verified dependency structure
- External corpora (SWE-rebench V2, daVinci-Agency) yield substantially smaller gains, suggesting training utility depends on task organization, not data volume

### RQ4: Graph-Conditioned Reinforcement Learning

Using GRPO with matched rollout budgets (8 rollouts per task):

- **Qwen3.5-4B**: $R_{\text{WEAVE}}$ improves Resolved by 3.7 pp over SFT and 2.0 pp over flat F2P
- **Qwen3.5-35B-A3B**: gains of 4.3 and 2.6 pp respectively, showing graph-conditioned reward scales with model capacity
- Graph Credit separation: +5.6/+6.2 pp over SFT at 4B/35B
- Transfer to SWE-bench Verified: +3.2/+4.2 pp over SFT
- Binary Resolved reward yields marginal gains (≤0.7 pp), consistent with sparse-signal limitations
- All settings maintain P2P regression below 2.3%

## Theoretical and Practical Implications

### Theoretical Contributions

1. **Unified task abstraction**: WEAVE demonstrates that dependency DAGs from typed change loci provide a principled foundation connecting task construction, evaluation, and training—a unified representation addressing the fragmentation of existing benchmarks.

2. **Dependency-aware credit assignment**: Graph Credit formalizes the concept of "phantom credit"—downstream test passes without upstream resolution—and provides a deterministic, test-grounded correction. The recursive minimum operator in Eq. (9) provides a principled mechanism for dependency-consistent credit.

3. **Repository-calibrated evidence weighting**: The inverse-frequency weighting (Eq. 3) demonstrates that repository-local calibration is essential for distinguishing genuine dependencies from hub-file artifacts, escaping the precision–recall trade-off inherent in single-signal approaches.

### Practical Implications

1. **Benchmarking**: WEAVE-Bench (300 held-out tasks) provides a more realistic evaluation for long-horizon code agents, with frontier models achieving only 48.7% Resolved—qualitatively harder than single-PR benchmarks.

2. **Training**: WEAVE's dependency-structured supervision is more token-efficient and effective than atomic or temporal-window alternatives, and graph-conditioned rewards provide dense, verifiable signals for RL.

3. **Diagnostics**: OverCredit (Flat-F2P − Graph Credit) serves as a structural diagnostic exposing whether agents resolve upstream dependencies faithfully or exploit downstream shortcuts.

## Conclusion

This paper addresses the fundamental limitation of current software engineering agent evaluation: the lack of a task abstraction capturing causal dependencies among multiple repository changes. **WEAVE** recovers dependency DAGs from verified PR histories through repository-calibrated typed-locus coupling, unifying task construction, dependency-conditioned partial credit (Graph Credit), and structured reward design.

Experiments across construction quality, agent benchmarking, supervised training, and reinforcement learning demonstrate WEAVE's effectiveness as both a challenging evaluation workbench and an effective training resource. The framework enables consistent evaluation and optimization of long-horizon code agents, with future directions including:

- Scaling to larger and more diverse repository ecosystems
- Exploring more complex dependency structures beyond DAGs
- Investigating additional training paradigms enabled by dense, verifiable rewards
- Extending the framework to other software engineering activities beyond PR-based changes

---

_Markdown view of https://picx.dev/p/EqT5MV, served by PicX — AI-generated visual whiteboard summaries of research papers._