# Quality, Not Just Outcome: A Structural Scoring Framework for Selecting Coding-Agent Trajectories for Supervised Fine-Tuning

> TQS, a structural scoring framework evaluating coding-agent trajectory process quality, boosts SWE-Bench Verified pass@1 to 27.6, outperforming random selection by 2.8 points at 43 ms per trajectory.

- **Source:** [arXiv](https://arxiv.org/abs/2610.13152)
- **Published:** 2026-10-12
- **Permalink:** https://picx.dev/p/0G5WNd
- **Whiteboard:** https://picx.dev/p/0G5WNd/image

## Summary

# Summary: Quality, Not Just Outcome: A Structural Scoring Framework for Selecting Coding-Agent Trajectories for Supervised Fine-Tuning

## Summary (Overview)

- **Core contribution**: The paper introduces **TQS (Trajectory Quality Score)**, a structural scoring framework for coding-agent trajectories that evaluates process quality rather than just final outcome, operating directly on trajectory structure in linear time without requiring LLM judges.
- **Key finding**: Models trained on top-ranked trajectories (by TQS) achieve 27.6 pass@1 on SWE-Bench Verified, outperforming random selection (24.8) by 2.8 points and bottom-ranked selection (21.8) by 5.8 points, demonstrating that successful trajectories vary substantially in their value as supervision.
- **Framework design**: TQS characterizes trajectory quality through three structural dimensions—information focus, intent–action alignment, and policy dynamics—operationalized via seven efficient, interpretable measures computed from event logs alone.
- **Scalability advantage**: TQS processes trajectories at ~43 ms per trajectory (87.5 trajectories/second with 4 workers), making corpus-scale selection practical (~49 minutes for 258K trajectories at ~$0.1–$0.2 in cloud CPU costs), versus ~$18.6K estimated for GPT-4o judging on the same corpus.
- **Validation**: TQS achieves 0.732 concordance in distinguishing successful from failed trajectories, 78.7% agreement with human evaluators and 75.4% with Claude Sonnet-5 on ordering successful trajectories, and monotonic resolution rate (54.1%→86.9%) across TQS deciles.

## Introduction and Theoretical Foundation

### Background and Motivation

Coding agents that interact with sandboxed environments to inspect repositories, execute tests, and edit files have achieved strong performance on software engineering benchmarks like SWE-Bench Verified. A critical ingredient is **trajectory supervision**: training on multi-turn agent demonstrations consisting of reasoning traces, tool calls, tool observations, and final patches. For medium-scale open-weight models, the common practice is to collect trajectories from more capable teacher agents and retain those whose final patches pass the benchmark test suite.

### The Core Problem

The paper identifies a fundamental limitation of outcome-level filtering:

> "Outcome-level correctness does not necessarily imply process-level quality."

Among successful trajectories, behavioral patterns vary substantially:
- Some localize bugs quickly, maintain coherent search scope, and transition efficiently from exploration to modification
- Others repeatedly inspect the same files, interrupt stalled shell commands, or postpone meaningful edits until late in the interaction budget

These differences matter because trajectory-level SFT exposes the student to the **full interaction process**. Inefficient or unstable interaction patterns become part of the training signal.

### Theoretical Foundation: Structured Convergence

The paper proposes that high-quality trajectories exhibit **structured convergence**: as an agent approaches a solution, a strong trajectory becomes increasingly organized around the relevant fix. This organization manifests at three levels:

1. **Information focus**: The agent focuses observations and file operations on task-relevant evidence
2. **Intent–action alignment**: Stated intent (thoughts) aligns with tool actions
3. **Policy dynamics**: The interaction policy moves from broad exploration toward productive edit, test, and recovery behavior

### Limitations of Existing Approaches

- **Perplexity-style scores** are confounded by length and tokenization
- **LLM-based judges** are costly at scale (hundreds of thousands of trajectories, billions of tokens)
- Neither approach characterizes how effectively an agent organizes its **search process over time**

## Methodology

### Trajectory Representation

A trajectory $\tau = (s_0, s_1, \ldots, s_{n-1})$ is a sequence of $n$ steps, where each step is:

$$
s_t = (a_t, m_t, o_t, F_t, \varepsilon_t),
$$

where $a_t \in \mathcal{A}$ is an action type (think, read_file, write_file, execute_bash, search, submit, other); $m_t$ is the free-text thought; $o_t$ is the tool observation; $F_t \subseteq \mathcal{F}$ is the set of file paths touched; and $\varepsilon_t \in \{0,1\}$ indicates whether $o_t$ matches a predefined error pattern.

### The Seven TQS Components

#### (A1) Observation Entropy Collapse (OEC)
Character-level Shannon entropy of observations, measuring whether observations concentrate around a smaller body of evidence:

$$
H(o_t) = -\sum_c p_t(c) \log_2 p_t(c),
$$
$$
s_{\mathrm{OEC}} = \operatorname{clamp}\left(1 - \min_t r_t, 0, 1\right),
$$

where $r_t = \tilde{H}_t / \bar{H}_{\mathrm{open}}$ is the ratio of the three-step moving average entropy to the mean entropy of the first observations.

#### (A2) Progressive Scope Narrowing (PSN)
Kendall's $\tau_K$ between the distinct files touched in a five-step window and the step index over the second half of the trajectory:

$$
\text{scope}_t = \left|\bigcup_{k=\max(0,t-4)}^{t} F_k\right|,
$$
$$
s_{\text{PSN}} = \text{clamp}\left(\frac{1-\tau_K}{2}, 0, 1\right).
$$

A shrinking active file set gives negative $\tau_K$ and a higher score.

#### (B1) Intent–Action Consistency (IAC)
A lexicon-based parser maps thoughts to coarse intended actions, measuring consistency between stated intent and actual action:

$$
s_{\mathrm{IAC}} = \frac{1}{|\mathcal{I}|} \sum_{t \in \mathcal{I}} \mathbf{1}[\hat{a}_t = a_t],
$$

where $\mathcal{I}$ is the set of steps where the intended action is defined.

#### (C1) Tool Transition Entropy (TTE)
Normalized entropy of adjacent action transitions, penalizing long single-tool loops:

$$
s_{\mathrm{TTE}} = \frac{-\sum_{i,j} p(i \rightarrow j) \log_2 p(i \rightarrow j)}{\log_2 49}.
$$

#### (C2) Solution Commitment Point (SCP)
Gaussian-shaped score favoring the first substantive edit (error-free write_file) occurring between 20% and 50% of the trajectory:

$$
s_{\mathrm{SCP}} = \begin{cases} 0, & \text{no substantive edit;} \\ \exp\left(-d(u)^2/(2\sigma^2)\right), & \text{otherwise,} \end{cases}
$$

where $u = t^*/n$ is the relative time of the first substantive edit, $d(u) = \max(0, 0.2-u, u-0.5)$, and $\sigma = 0.2$.

#### (C3) Post-Error Diversity (PED)
Measures adaptation after errors by combining action-type change with Jaccard distance between successive file sets:

$$
d_t = \frac{1}{2}\left(\mathbf{1}[a_{t+1} \neq a_t] + \text{Jacc}(F_{t+1}, F_t)\right),
$$
$$
s_{\mathrm{PED}} = \frac{1}{|\mathcal{E}_\tau|} \sum_{t \in \mathcal{E}_\tau} d_t.
$$

#### (D1) Dirty Pattern Indicator (DPI)
Guardrail capturing categorical failures (early stopping, no substantive edit, repeated shell-command loops, repeated errors):

$$
p = 0.4\mathbf{1}[\text{early}] + 0.4\mathbf{1}[\text{no edit}] + \min(0.3, 3\,\mathrm{lp}) + \min(0.3, 0.6\,\mathrm{er}),
$$
$$
s_{\mathrm{DPI}} = 1 - \min(1, p).
$$

### Aggregation

Fixed, hand-set weights: $w_{\mathrm{OEC}} = 0.30$, $w_{\mathrm{IAC}} = 0.25$, $w_{\mathrm{PSN}} = w_{\mathrm{TTE}} = w_{\mathrm{PED}} = w_{\mathrm{DPI}} = 0.10$, and $w_{\mathrm{SCP}} = 0.05$. The final TQS is:

$$
\text{TQS}(\tau) = \frac{\sum_{c \in \mathcal{C}^*(\tau)} w_c \cdot s_c(\tau)}{\sum_{c \in \mathcal{C}^*(\tau)} w_c}, \tag{1}
$$

where $\mathcal{C}^*(\tau)$ is the set of components defined on $\tau$. Trajectories with $n < 3$ or component coverage below $4/7$ are excluded.

### Component Admission Protocol

Each component must pass three criteria:
1. **Concordance > 0.5**: Rate at which the measure assigns a higher score to a passing trajectory than a paired failing trajectory from the same task
2. **Length correlation $|\rho| < 0.25$**: Absolute Spearman correlation with trajectory length
3. **Behavioral diversity preservation**: No mode collapse in selected subsets

## Empirical Validation / Results

### Discriminant Validity

| Measure | Conc. | $|\rho|$ | Outcome |
|---------|-------|---------|---------|
| OEC | 0.640 | 0.103 | √ kept |
| IAC | 0.612 | 0.087 | √ kept |
| PSN | 0.585 | 0.142 | √ kept |
| SCP | 0.598 | 0.058 | √ kept |
| TTE | 0.563 | 0.218 | √ kept |
| PED | 0.541 | 0.034 | √ kept |
| DPI | 0.591 | 0.071 | √ guardrail |
| **TQS Composite** | **0.732** | **0.092** | — |
| HCR | 0.527 | 0.014 | × diversity |
| BUS/TGCR/DCS-f. | ≤ 0.479 | < 0.25 | × conc. |
| DCS-Jacc. | — | 0.729 | × length |

### Main Results on SWE-Bench Verified

| ID | Method | $\mathcal{D}$ | Easy | Medium | Hard | Overall | Δ Base |
|----|--------|--------------|------|--------|------|---------|--------|
| B0 | Base (no SFT) | — | 36.6 | 19.2 | 2.2 | 24.4 | — |
| B1 | Random-K | 1,024 | 40.7 | 16.9 | 2.2 | 24.8 | +0.4 |
| B2 | Bottom-K | 1,024 | 33.5 | 16.9 | 0.0 | 21.8 | −2.6 |
| B3 | Shortest-K | 1,024 | 37.6 | 18.4 | 0.0 | 24.2 | −0.2 |
| B4 | DPI-only-K | 1,024 | 37.6 | 19.9 | 2.2 | 25.2 | +0.8 |
| B5 | LLM-Judge-K (GPT-4o) | 1,024 | 43.3 | 21.1 | 6.7 | 28.4 | +4.0 |
| B6 | Top-LM-K (length-matched) | 1,024 | 40.7 | 20.7 | 4.4 | 27.0 | +2.6 |
| **Ours** | **Top-K (highest TQS)** | **1,024** | **42.3** | **20.3** | **6.7** | **27.6** | **+3.2** |

### Direct Validation Among Successful Runs

| Validation statistic | Human | Sonnet-5 |
|----------------------|-------|----------|
| High vs. Low agreement | 85.2 (208/244) | 81.1 (198/244) |
| Mid vs. Low agreement | 74.2 (190/256) | 77.0 (197/256) |
| High vs. Mid agreement | 76.8 (172/224) | 67.4 (151/224) |
| Overall agreement | 78.7 (570/724) | 75.4 (546/724) |
| Pooled concordance | 73.3 (631/861) | 76.2 (656/861) |
| Full-order match | 45.7 (137/300) | 41.3 (124/300) |
| Spearman ρ | 0.417 | 0.402 |

### Dimension Ablation

| Selection signal | pass@1 |
|------------------|--------|
| Full TQS-Top | 27.6 |
| w/o Information focus | 24.6 |
| w/o Intent-action alignment | 23.0 |
| w/o Policy dynamics | 25.8 |
| Random-K | 24.8 |
| Base (no SFT) | 24.4 |

### Behavioral Diagnosis on Rollouts

| Process metric | Training Data (Top-K) | Top model | Random model | Bottom model |
|----------------|----------------------|-----------|--------------|--------------|
| Mean Steps | 34.2 | 50.66 | 57.64 | 63.46 |
| Mean OEC | 0.603 | 0.571 | 0.521 | 0.484 |
| Mean PSN | 0.753 | 0.570 | 0.548 | 0.568 |
| Mean SCP | 0.994 | 0.777 | 0.798 | 0.772 |
| Mean PED | 0.826 | 0.192 | 0.267 | 0.164 |
| Mean DPI | 0.996 | 0.931 | 0.923 | 0.906 |

### External Outcome Check

The resolved rate increases monotonically from **54.1% in D1 to 86.9% in D10** across TQS deciles (32.9-point spread, no inversion at decile boundaries).

## Theoretical and Practical Implications

### Theoretical Implications

1. **Process quality is orthogonal to outcome**: The paper demonstrates that "passing" is too coarse a signal for coding-agent SFT data—trajectories with identical outcomes can differ substantially in their value as supervision.

2. **Structured convergence as a quality framework**: The three-dimensional characterization (information focus, intent–action alignment, policy dynamics) provides a principled way to think about what makes agent trajectories good training data.

3. **Length is correlated but insufficient**: The Top-LM-K result (27.0 vs. Random-K's 24.8) shows that length captures some quality signal, but TQS provides additional discriminative power beyond length alone.

### Practical Implications

1. **Scalable data selection**: TQS makes corpus-scale trajectory filtering practical (49 minutes CPU time vs. $18.6K for LLM judging), enabling repeated rescoring and iterative data curation.

2. **Cost-quality tradeoff**: TQS approaches the effectiveness of GPT-4o rubric-based judging (27.6 vs. 28.4 pass@1) at a fraction of the cost, with the gap narrowing on harder tasks.

3. **Behavioral imprint**: Selecting high-TQS supervision leaves measurable behavioral improvements in the fine-tuned policy—shorter interactions, more focused observations, and fewer dirty patterns.

4. **Hard-task sensitivity**: The Top–Random gap increases with task difficulty (1.6 on Easy → 3.4 on Medium → 4.5 on Hard), suggesting that process quality matters most for challenging problems.

## Conclusion

### Main Takeaways

- **TQS provides a lightweight, effective alternative to LLM-based data selection** for coding-agent trajectories, operating directly on event-log structure in linear time.
- **Process-level trajectory quality is an important factor** in coding-agent supervision, with top-ranked trajectories improving pass@1 by 3.2 points over base and 2.8 points over random selection.
- **All three structural dimensions contribute** to the downstream benefit, with intent–action alignment being the most critical (removing it drops performance below random selection).
- **Low-quality successful trajectories can actively hurt** downstream performance (Bottom-K: 21.8 vs. base: 24.4).

### Future Directions

- Recalibrating parsers and thresholds for different action spaces, multilingual traces, or other step granularities
- Extending to GUI or multimodal agents (requiring new definitions of scope and intent)
- Testing on larger backbones (preliminary 7B experiments showed unstable selector differences)
- Multi-seed training runs for variance estimates
- Incorporating long-trajectory tails (>32K tokens) that may contain useful recovery behavior

### Key Limitations

- Single teacher model (Qwen3-Coder-480B) under OpenHands scaffold
- Fixed-budget setting with 1,024 trajectories per condition
- Single training run per condition (no multi-seed variance)
- TQS scores observed process structure, not whether the patch addresses the underlying root cause

---

_Markdown view of https://picx.dev/p/0G5WNd, served by PicX — AI-generated visual whiteboard summaries of research papers._