# Done, But Still Working: Anytime Evaluation of When Coding Agents Should Stop

> Anytime evaluation of coding agents shows they waste 42-66% of run cost after passing self-tests, and stopping rules can save up to 57% without losing quality.

- **Source:** [arXiv](https://arxiv.org/abs/2611.22901)
- **Published:** 2026-11-21
- **Permalink:** https://picx.dev/p/xbymJ2
- **Whiteboard:** https://picx.dev/p/xbymJ2/image

## Summary

## Summary (Overview)

- **Introduces anytime evaluation for coding agents**: A novel methodology that replays recorded agent runs, scores intermediate patches with hidden tests, and prices each prefix to measure when solutions become correct and what continued work costs.
- **Quantifies the cost of overwork**: Agents continue running 18+ commands after their first passing self-test, spending 42-66% of total run cost on post-pass work, yet often submitting the same patch they already had.
- **Demonstrates paired evaluation precision gains**: Comparing stopping rules against running-to-the-end on the same recorded runs reduces standard error 2-8×, equivalent to 4-60× more tasks in independent evaluation.
- **Provides hindsight bounds and a decomposition**: Stopping at passing self-tests can save at most 25-57% of cost; the quality effect decomposes exactly into firing rate × later-change share × net repair per change (Equation 3).
- **Shows stop distillation works**: Fine-tuning agents to imitate a "twice-passed" stopping rule saves 30% of inference cost over six seeds, though non-inferiority at the 2-point quality margin is not established.

---

## Introduction and Theoretical Foundation

### Background and Motivation

Coding agents (Jimenez et al., 2024; Yang et al., 2024; Wang et al., 2025) search repositories, edit code, and run tests to solve software issues. The mini-SWE-agent prompt explicitly asks agents to test edge cases before finishing (SWE-agent team, 2025). However, **a passing test does not necessarily end the run** — agents may run more tests, revise patches, or perform cleanup, incurring additional cost while potentially fixing or breaking things.

The paper addresses three questions:
1. How much do agents spend after their tests pass?
2. When does continued work improve the final patch?
3. Can an agent stop earlier without losing those improvements?

A benchmark score for the final patch cannot answer these questions because it doesn't reveal whether an earlier patch was already correct.

### Theoretical Foundation

The core theoretical contribution is **anytime evaluation** — treating coding agents as anytime algorithms (Dean et al., 1988; Zilberstein, 1996) whose answer is their current patch, measured against a performance profile of quality vs. time.

**Proposition 1 (Condition for exact stop-and-select evaluation):** If a policy (τ, σ) is prefix-measurable (acts only by ending the run and choosing among its patches) and does not change the agent or environment before stopping, then the submitted patch and cost have the same joint distribution whether evaluated online or computed from a recorded run without the policy. This is a **truncation identity** enabling offline evaluation of stopping rules without generating new trajectories.

---

## Methodology

### Formal Framework

A run issues commands $k = 1, \ldots, T$. After command $k$, the agent has observed prefix $h_k$. Let $\pi_k$ be the working-tree patch after command $k$, and $Q(\pi) \in \{0, 1\}$ whether the task's hidden tests pass on $\pi$. A stop-and-select policy $(\tau, \sigma)$ ends the run after command $\tau \leq T$ and submits $\pi_\sigma$, $\sigma \leq \tau$.

The utility of a policy is:

$$U(\tau, \sigma) = \lambda Q(\pi_\sigma) - \text{cost}_\tau(c, p) \tag{1}$$

where $\lambda$ is the value of a resolved task in cost units.

### Cost Pricing

For requests $t = 1, \ldots, T'$ of a run, with $P_t$ prompt tokens, $H_t$ cached tokens, and $O_t$ output tokens:

$$\text{cost}_k(c, p) = \sum_{t \leq t_k} (P_t - H_t) + c \sum_{t \leq t_k} H_t + p \sum_{t \leq t_k} O_t \tag{2}$$

Prices are relative to an uncached input token, with $c \in \{0.1, 1\}$ and $p = 4$.

### Stopping Rules Studied

- **First pass**: Submit at the first passing self-test
- **Twice passed**: Submit at the first passing self-test whose patch equals that of the previous passing self-test
- **Last pass** (selection sensitivity): Don't stop; submit the last passing self-test's patch if the agent changed the patch afterwards
- **Baselines**: Step caps, stability rules, learned policies

### The Decomposition

For a rule that submits its current patch ($\sigma = \tau$), the change in resolution relative to running to the end decomposes exactly as:

$$\Delta Q = -r \cdot u \cdot \delta \tag{3}$$

where:
- $r$ = share of runs in which the rule fires
- $u$ = share of fired runs whose final patch differs from the patch at the stop
- $\delta = E[Q(\pi_T) - Q(\pi_\tau) | \text{fired, changed}]$ = net repair per change

Only $\delta$ needs hidden tests; $r$, $u$, and the saving can be read from unlabelled runs.

### Stop Distillation

Fine-tuning objective with stop examples at the rule's stopping point and continue examples at earlier commands:

$$L(\theta) = -\sum_{\text{runs}} \left[ \log p_\theta(a_{\text{stop}} | h_{\tau_\ell}) + \sum_{k \in K} \log p_\theta(a_{k+1} | h_k) \right] \tag{4}$$

### Experimental Setup

- **Models**: Qwen3.6-27B (main), Qwen3-Coder-30B-A3B, Qwen3.8-27B, Qwen3.5-9B, Gemma-4-26B-A4B, Gemma-4-31B-it, GLM-4.7-Flash, DeepSeek-V4-Flash (8 models, 4 families)
- **Scaffolds**: mini-SWE-agent 2.4.6 and SWE-agent 1.1.0
- **Tasks**: SWE-bench Verified (Django, SymPy + 7 additional repositories with built environments)
- **Environments**: Executable sandbox with Linux namespaces, no network, per-command time limits
- **~6,000 runs total** across derivation, fresh tasks, new repositories, and replication sets

---

## Empirical Validation / Results

### 1. Agents Keep Working After Tests Pass

| Model | Runs with passing self-test | Commands after first pass | Cost after first pass |
|-------|---------------------------|--------------------------|----------------------|
| Qwen3.6, mini | 60% | 18 | 42% |
| Qwen3.6, SWE-agent | 90% | — | 66% |
| DeepSeek-V4-Flash | 41% | — | ~40% |
| Qwen3.8-27B | 46% | — | ~40% |
| Qwen3-Coder, Gemma, GLM | ≤6% | — | — |

In mini-SWE-agent, Qwen3.6 submits the patch it already had in 80–87% of runs after its first passing self-test.

### 2. Paired Evaluation Precision Gains

**Table 1: Standard error (points) of resolution change — offline vs. online**

| Agent | Tasks | Offline SE | Online SE | Factor |
|-------|-------|-----------|-----------|--------|
| Qwen3.6, mini | 178 | 0.40 | 2.68 | 45.6 |
| Qwen3.6, SWE-agent | 193 | 1.28 | 2.58 | 4.0 |
| Qwen3.8 | 371 | 0.23 | 1.81 | 60.5 |
| Qwen3.5-9B | 115 | 0.62 | 4.15 | 45.1 |
| DeepSeek-V4-Flash | 114 | 0.88 | 3.94 | 20.2 |

Detecting a 2-point change with 80% power takes **56 tasks offline but 1,184 online** — more than SWE-bench Verified contains.

### 3. Hindsight Bounds and Rule Performance

**Hindsight savings (cost saved at c = 0.1):**
- **O_pass** (stop at passing self-tests without losing quality): 57% (SWE-agent), 25% (mini), 17–19% (Qwen3.8, DeepSeek), 8% (Qwen3.5-9B), ≤5% (rarely-passing models)
- **O_any** (truncate anywhere): 73–90%, with ~half from stopping successful runs early and ~half from abandoning never-succeeding runs

**Table 2: Paired rule comparisons for Qwen3.6-27B in mini-SWE-agent (change in resolution / change in cost)**

| Rule | Derivation (712) | Fresh tasks (230) | New repos (156) | New runs (386) |
|------|-----------------|-------------------|-----------------|----------------|
| Self-test passed | +0.4 / −23.9% | −3.9 / −31.1% | −1.3 / −27.3% | −1.8 / −30.5% |
| Self-test passed twice | +0.7 / −17.5% | −1.7 / −23.1% | −1.3 / −17.6% | −1.0 / −22.8% |
| Last pass (no stopping) | +0.4 / 0% | +0.0 / 0% | +0.0 / 0% | +0.3 / 0% |

The **twice-passed rule** captures 68–77% of the hindsight bound for three models in mini-SWE-agent, and 95% in SWE-agent (but at −4.4 points resolution).

### 4. Why Stopping Loses: Decomposition Results

- **SWE-agent loses 2.8 points vs. mini's 0.3** despite similar net repair per change (δ = 0.08 vs. 0.07) because SWE-agent changes its patch after a pass in **40% of stopped runs** vs. **10%** for mini
- **DeepSeek-V4-Flash has negative δ** (−0.18 twice-passed, −0.20 first-pass): later changes break more than they repair, so stopping *improves* resolution by 0.9–1.7 points
- **Cross-agent prediction** (exploratory): mean absolute error 0.65 points (first pass) and 0.31 (twice-passed) vs. 1.05 and 0.50 for baseline — not significant, fails for DeepSeek

### 5. Telling, Asking, and Teaching

- **Prompting to stop**: saved only 8.4% (preregistered)
- **Asking the agent**: said yes in only 6–7% of 439 cases
- **Stop distillation** (twice-passed rule, Qwen3.6-27B, LoRA rank 16):
  - New-repository tasks: −27.9% cost at −0.6 points
  - Fresh Django/SymPy: −27.6% cost at +2.5 points
  - **Six seeds on 256 tasks: −30.0% cost at −1.2 points** (−3.1 to +1.2 by seed)
  - Qwen3.8-27B: −43.2% cost at +0.2 points
- **Controls**: plain fine-tuning saved nothing (+9.0% cost); random stop labels made the agent give up (2.2% resolved)

### Scoring Convention Sensitivity

Scoring the worktree diff without staged changes **understated resolution by up to 7 points** for some models, turned the 1.3-point gain of submitting the last passing patch into 0.3, and reversed the cross-agent prediction result.

---

## Theoretical and Practical Implications

### Theoretical Contributions

1. **Anytime evaluation framework**: Replay with per-step scoring and pricing enables exact evaluation of truncation and selection policies without new trajectories, conditional on faithful state preservation (Proposition 1).

2. **Exact decomposition** (Equation 3): The quality effect of stopping separates into firing rate, later-change share, and net repair — only the last requires hidden tests, enabling prediction from unlabelled runs.

3. **Precision gains**: Paired offline comparison is worth 4–60× more tasks than independent online comparison, making small effect sizes detectable within benchmark constraints.

### Practical Implications

- **Early stopping opportunities are substantial but model-dependent**: Savings range from under 1% to 57% depending on model and scaffold, with quality effects ranging from −5.4 to +1.7 points.
- **The twice-passed rule is a practical default**: Saves 18–23% with modest quality loss (−1.0 to −1.7 points), captures 68–77% of the theoretical bound.
- **Stop distillation is trainable**: Agents can learn to stop from their own runs, saving 30% robustly, though quality preservation at the 2-point margin remains uncertain.
- **Scoring conventions matter**: The patch definition used for evaluation can reverse conclusions — staged changes must be included.

---

## Conclusion

### Main Takeaways

1. **Coding agents overwork substantially** after their own tests pass, spending up to 66% of run cost on post-pass activity while often submitting the same patch.

2. **Stopping rules offer significant cost savings** but their quality effects vary dramatically across models and scaffolds — determined by whether agents change their patches after a pass and whether those changes repair or break correctness.

3. **Anytime evaluation enables precise offline comparison** of stopping rules, with precision gains equivalent to 4–60× more tasks.

4. **Stop distillation works**: Agents can learn to stop from their own runs labeled by a rule, saving 30% of inference cost, though non-inferiority at the 2-point quality margin is not established.

5. **Scoring conventions are critical**: The definition of "the patch" (tracked-file diff vs. full diff vs. submitted patch file) can reverse conclusions about when agents should stop.

### Future Directions

- Validating on frontier API agents and unrestricted reasoning settings
- Full patch-definition robustness (complete base-relative patch protocols)
- Improving the "give up early" component (abandoning never-succeeding runs), identified as the larger open problem
- Extending stop distillation to more model families and scaffolds
- Developing learned monitors that can predict run failure before it occurs

### Key Limitations

- All models are open and locally served; no frontier API agents tested
- Test-command patterns may miss valid checks (custom scripts)
- Final-patch agreement does not verify every intermediate state
- Non-inferiority at the 2-point quality margin is not established for distillation
- Cost excludes sandbox compute, monitoring overhead, and wall-clock latency

---

_Markdown view of https://picx.dev/p/xbymJ2, served by PicX — AI-generated visual whiteboard summaries of research papers._
