SWE-Prime: Fewer Trajectories, Better Performance

Summary (Overview)

• Core contribution: SWE-Prime is a multi-granularity, two-stage supervised fine-tuning (SFT) data selection method for coding agents that progressively filters training data at both the trajectory level and the semantic segment level.
• Key finding: Training on only 10% of the resolved trajectories selected by SWE-Prime outperforms training on the full resolved dataset, yielding relative performance gains of up to 12.2% on SWE-Bench Pro and 24.2% on SWE-Bench Verified.
• Methodological innovation: SWE-Prime introduces semantic segments (groups of consecutive steps sharing a common intent) as behaviorally coherent units for fine-grained supervision-quality assessment, rather than evaluating isolated steps.
• Multi-dimensional quality assessment: The method evaluates trajectories on process quality (workflow pattern, tool success rate, redundancy, git-hacking prevention), result quality (patch scope), and data representativeness (cluster-based selection), then filters segments by contribution, learnability, and behavioral risk.
• Validation breadth: Experiments across three base models (Qwen3-30B-A3B-Instruct-2507, GLM-4.7-Flash, Qwen3-Coder-30B-A3B-Instruct) and two benchmarks demonstrate consistent improvements, including across Python, JavaScript, TypeScript, and Go subsets.

Introduction and Theoretical Foundation

Background

Software issue resolution in real-world repositories is a long-horizon task requiring coding agents to understand repository context, iteratively locate relevant code, implement changes, and validate patches. Recent work has constructed large-scale software engineering datasets for SFT, such as:

• SWE-Gym: Derives executable tasks from real-world GitHub issues
• SWE-smith and R2E-Gym: Use synthetic data generation to scale task and trajectory collection
• SWE-Factory: Automates environment construction and task verification across multiple programming languages

These approaches perform SFT on successful trajectories identified through execution-based verification.

Key Insight: Success ≠ Quality

The authors identify that task success alone does not guarantee high-quality supervision. Successful trajectories may contain ineffective, redundant, or risky behaviors. The paper analyzes this problem from three perspectives:

Outcome View: Existing work adopts an outcome-based view, retaining all successful trajectories.

Trajectory View: A successful trajectory may still be unsuitable for SFT due to:

1. Problem-solving pattern: Poor workflows (e.g., modifying code without prior exploration vs. observe-edit-verify patterns)
2. Tool call success rate: Frequent tool failures introduce unreliable tool-use patterns
3. Continuous redundancy: Repeated actions without new information
4. Git hacking: Leakage-based shortcuts using future repair information from version history
5. Result quality: Overly broad patches modifying unnecessary files increase regression risks

Segment View: Even high-quality trajectories may contain segments of limited value (e.g., steps that execute successfully but don't contribute to the final fix).

Theoretical Foundation

The work draws on prior findings that:

• Task success alone is insufficient for identifying high-quality supervision (Uesato et al., 2022; Lightman et al., 2023)
• High-quality SFT data should be compact and low-noise, providing demonstrations models can reliably learn from
• Standard problem-solving processes help models learn systematic behaviors (Chen, Ma, and Jiang, 2026)
• Frequent tool failures introduce unreliable patterns (Yang et al., 2025a)
• Long-context repetition introduces undesirable supervision (Xiao et al., 2025)
• Leakage-based shortcuts introduce cheating behaviors (Song et al., 2026; Tao et al., 2026)

Methodology

Overview of SWE-Prime

SWE-Prime is a two-stage method:

• Stage 1: Trajectory-level selection based on process quality, result quality, and data representativeness
• Stage 2: Segment-level selection identifying high-value semantic segments within retained trajectories

During SFT, complete trajectories are retained as context, but only selected segments contribute to the loss.

Stage 1: Trajectory-Level Selection

Process Quality Assessment

Four complementary signals:

1. Observe-Edit-Verify Problem-solving Pattern: Sets $S_{\mathrm{workflow}} = 1$ if a trajectory inspects the repository before its first edit AND validates the patch through a test or check after its final edit; otherwise 0.

2. Tool Call Success Rate: The tool reliability score $S_{\mathrm{tool}}$ is the percentile rank of the trajectory's tool-call success rate within the trajectory pool.

3. Continuous Redundancy Control: $S_{\mathrm{redundancy}} = 0$ if any adjacent pair of tool calls has matching tool names AND arguments; otherwise 1.

4. Git Hacking Prevention: $S_{\mathrm{git}} = 0$ when Git-history operations reveal reference-patch content before code modification; otherwise 1.

Result Quality Evaluation

Using reference patch to estimate appropriate modification scope:

$$S_{\mathrm{file}} = \min\left(\frac{F_{\mathrm{gold}}}{F_{\mathrm{model}}}, 1\right), \qquad S_{\mathrm{line}} = \min\left(\frac{L_{\mathrm{gold}}}{L_{\mathrm{model}}}, 1\right)$$

where $F_{\mathrm{model}}$ and $F_{\mathrm{gold}}$ are the numbers of files modified by the generated and reference patches, and $L_{\mathrm{model}}$ and $L_{\mathrm{gold}}$ are their changed line counts.

$$S_{\mathrm{result}} = S_{\mathrm{file}} \times S_{\mathrm{line}}$$

Multiplication ensures a high score only when both file and line scopes are focused.

Combined Trajectory Score

$$S_{\mathrm{traj}} = \frac{1}{5}\left(S_{\mathrm{workflow}} + S_{\mathrm{tool}} + S_{\mathrm{redundancy}} + S_{\mathrm{git}} + S_{\mathrm{result}}\right)$$

Data Representativeness Assessment

• Issue descriptions are encoded using Qwen3-Embedding-8B
• HDBSCAN clusters semantically similar issues
• Trajectories are ranked by $S_{\mathrm{traj}}$ within each cluster
• Highest-ranked candidates are selected under the target budget (10%)

Stage 2: Segment-Level Selection

Segment Chunking

Individual steps are too narrow for behavior-level assessment. SWE-Prime groups consecutive steps sharing a common intent into semantic segments.

Boundary-aware chunking: Fixed-size sliding windows may split ongoing behaviors. SWE-Prime:

• Finalizes all segments except the last in each window
• Carries the potentially incomplete last segment into the next window
• Provides the LLM with a compact summary of task objective, established facts, and identified risks from preceding windows

Segment Scoring

Each segment $s_{i,j}$ receives a quality score $q_{i,j} \in [0, 10]$ based on three criteria:

1. Contribution to the Final Fix: Advances issue localization, context gathering, modification, verification, or error recovery
2. Local Learnability: Provides a clear evidence–action–feedback pattern based on available information
3. Behavioral Risk: Captures undesirable behaviors (irrelevant exploration, uninformative failures, unsafe operations, hardcoded solutions)

Segment Filtering and Selective SFT Objective

Binary selection mask: $m_{i,j} = 1$ if $q_{i,j} \geq \delta$ (threshold = 7), else 0.

The selective SFT objective:

$$\mathcal{L}_{\mathrm{SFT}} = -\frac{1}{N_{\mathrm{sel}}} \sum_{i} \sum_{j} m_{i,j} \sum_{t \in \mathcal{A}(s_{i,j})} \log p_{\theta}(y_{i,t} | y_{i,<t})$$ $$N_{\mathrm{sel}} = \sum_{i} \sum_{j} m_{i,j} |\mathcal{A}(s_{i,j})|$$

where $\mathcal{A}(s_{i,j})$ denotes assistant-response token positions in segment $s_{i,j}$, and $N_{\mathrm{sel}}$ normalizes the objective into an average token-level loss. Unselected segments remain in context ($y_{i,<t}$) but do not contribute to the loss.

Empirical Validation / Results

Experimental Setup

Training Data: SWE-rebench OpenHands Trajectories (67,074 trajectories generated by Qwen3-Coder-480B-A35B-Instruct with OpenHands); 32,161 resolved trajectories used as the candidate pool.

Base Models: GLM-4.7-Flash, Qwen3-30B-A3B-Instruct-2507, Qwen3-Coder-30B-A3B-Instruct

Benchmarks: SWE-Bench Verified (500 tasks), SWE-Bench Pro (731-task public split)

Baselines: Raw Model, Resolved-Trajectory SFT (100%), Random-10% SFT

Training: Max sequence length 131,072; batch size 32; AdamW; cosine LR schedule; peak LR $4 \times 10^{-6}$

RQ1: Effectiveness

Main Results (Table 1):

Key findings:

• SWE-Prime achieves the highest Resolved Rate across all models and benchmarks
• Relative improvements over raw models: 18.8%–57.1% on SWE-Bench Verified, 17.6%–140.2% on SWE-Bench Pro
• Outperforms Resolved-Trajectory SFT by up to 24.2% (Verified) and 12.2% (Pro)
• Substantially outperforms Random-10%, demonstrating value of quality-aware selection
• Stage 2 (segment-level) provides complementary benefits beyond Stage 1 alone
• Consistent gains across Python, JavaScript, TypeScript, and Go subsets

RQ2: Hyperparameter Analysis

Trajectory Retention Ratio (Figure 4a):

• 5%: 46.2% → 10%: 50.2% (peak) → 20%: 49.6% → 30%: 49.2%
• Too few trajectories limit coverage; beyond 10%, additional data provides no benefit

Segment Score Threshold (Figure 4b):

• Threshold 5: 46.8% → 7: 53.2% → 8: 49.6% → 9: 47.0%
• Lower thresholds retain low-value supervision; overly restrictive thresholds exclude useful segments

Configuration: 10% retention ratio + threshold of 7, frozen and applied across all experiments. The configuration generalizes across models and benchmarks.

RQ3: Behavioral Analysis

Process Quality (Figure 5):

• SWE-Prime reduces redundancy by up to 0.188
• Increases observe-before-edit rate by up to 9.6 percentage points
• Improves tool success rate by up to 0.094

Interaction Efficiency:

• Reduces average interaction turns by 4.1 to 12.7 turns relative to Resolved-Trajectory SFT
• Reductions accompanied by higher Resolved Rates (not premature termination)

Theoretical and Practical Implications

Implications for SFT Data Curation

1. Quality over quantity: The results challenge the prevailing "scale-first" approach to trajectory collection. Retaining more successful trajectories does not necessarily improve downstream performance; a smaller, carefully selected subset can provide more effective supervision.

2. Multi-granularity matters: Trajectory-level filtering alone is insufficient. The consistent gains from Stage 2 demonstrate that low-value behaviors exist even within high-quality trajectories, and segment-level selection provides complementary benefits.

3. Semantic segments as evaluation units: Grouping consecutive steps into behaviorally coherent segments (rather than evaluating isolated steps) enables more accurate assessment of intent, outcome, and contribution.

4. Representativeness prevents bias: Cluster-based selection (HDBSCAN on issue embeddings) prevents the selected subset from over-focusing on a few issue types, improving coverage and generalization across programming languages.

Practical Implications

• Data efficiency: Achieving better performance with 10% of the data significantly reduces training costs and computational requirements
• Generalizable configuration: The selected hyperparameters (10% retention, threshold 7) transfer across models and benchmarks without retuning
• Behavioral improvement: Beyond outcome metrics, SWE-Prime produces models with more reliable tool use, fewer redundant actions, and more structured problem-solving processes

Conclusion

SWE-Prime demonstrates that effective coding-agent SFT depends not only on successful task outcomes, but also on the quality, representativeness, and learnability of the resulting supervision. The two-stage, multi-granularity approach provides:

1. Stage 1: Trajectory-level selection using process quality (workflow, tool reliability, redundancy, git-hacking prevention), result quality (patch scope), and data representativeness (HDBSCAN clustering)
2. Stage 2: Segment-level selection using contribution, learnability, and behavioral risk criteria with a selective loss objective

Main takeaways:

• Training on 10% of SWE-Prime-selected trajectories outperforms full-dataset SFT
• Relative gains up to 12.2% (SWE-Bench Pro) and 24.2% (SWE-Bench Verified)
• Consistent improvements across three models and multiple programming languages
• Selected configuration generalizes without retuning

Future directions implied by this work include:

• Investigating whether even more fine-grained selection (e.g., at the step level within segments) could yield further gains
• Exploring adaptive retention ratios and thresholds based on pool characteristics
• Extending the framework to other long-horizon agent tasks beyond software engineering