Summary (Overview)

• Core contribution: The paper formalizes a "specification–oracle gap" in test-graded coding-agent benchmarks—a mismatch where tests require information (names, paths, literals) that the natural-language specification never states, causing correct implementations to fail.
• Static audit tool: The authors build a static audit covering five languages (Go, Python, Rust, TypeScript, Java) that classifies every name-, path-, and literal-level dependency of every scored test, with held-out precision of 89.2% and recall of 92.9%.
• Key findings on SWE-Milestone: 25 of 98 graded milestones contain hidden dependencies; build units with such dependencies fail in 80.6% of runs vs. 48.4% for comparable units without them.
• Leaderboard robustness: Removing affected tests leaves ranking largely intact (Kendall's τ ≈ 0.98), but scores shift within affected repositories (e.g., go-zero score rises from 38.33 to 42.08).
• Prevalence follows construction: Benchmarks where specifications are LLM summaries or release notes show gaps in a third or more of tasks; benchmarks with screened or test-written specifications show almost none.

---

Introduction and Theoretical Foundation

Background

Coding-agent benchmarks have evolved from isolated issues to long-horizon software evolution, where an agent carries one codebase through dependent changes. These benchmarks are built automatically: a pipeline groups a project's commits into milestones, writes a natural-language specification (SRS) for each, and reuses the project's own tests as the grading oracle.

The Core Problem

This construction joins two artefacts of different kinds:

• The specification describes behaviour.
• The oracle is code that calls helpers, references constants, imports modules at fixed paths, and compares exact messages.

Where the oracle depends on details the specification never states, an agent may implement the specified behaviour correctly yet still fail. This is a specification–oracle gap.

Formal Definition

For a task $t$ with specification $S_t$, starting code base $B_t$, and oracle $O_t$ (a set of tests from a reference solution), each test $o \in O_t$ depends on a set $D(o)$ of artefacts. An artefact is fixed by the task if defined in $B_t$ or named in $S_t$ (plus inferable conventions). The gap is:

$$G(o) = D(o) \setminus F_t$$

where $F_t$ is the set of fixed artefacts.

Key Theoretical Distinctions

The paper introduces three critical concepts:

1. Under-specification (Classes A1, A2): The agent could only reproduce the dependency by guessing or memory.
2. Chain inconsistency (Classes B1–B3): Inconsistencies along the milestone chain—B3 is starkest: the reference test references a name the milestone's own gold state doesn't define, so even the reference implementation cannot pass it.
3. Blast radius: A hidden name or path fails an entire build unit (the group of tests compiled together), not just one test.

Dependency Classes (Table 1)

---

Methodology

Static Audit Architecture

The audit consists of:

• Five language-specific extractors (Go, Python, Rust, TypeScript, Java) that report names, module paths, and literals that each state defines and each test references.
• A language-independent core that diffs gold states, classifies dependencies, and maps required tests to build units.

Inferable Dependency Rules

A dependency is inferable (not counted as "hard") if:

1. It combines a trial-base name with an affix the codebase already uses systematically (e.g., Allow → AllowCtx).
2. It is a new module/file whose name the specification gives, placed in an existing directory (front-end convention).
3. It is a method whose signature matches an interface of the trial base or the specification.

Experimental Design (Five Research Questions)

• RQ1 (Prevalence): How many tests/milestones depend on unstated, non-inferable information?
• RQ2 (Realisation): Do these dependencies actually cause failures?
• RQ3 (Impact): How do scores/rankings change when affected tests are removed?
• RQ4 (Precision/Cost): How accurate and costly is the audit?
• RQ5 (Other benchmarks): Does the gap appear in benchmarks built differently?

Counterfactual Test

For Go, the authors rename one hidden identifier in the reference solution's non-test code—a counterfactual for an agent implementing the behaviour under another name—to verify the predicted blast radius.

---

Empirical Validation / Results

RQ1: Prevalence (Table 2)

Key insight: Tests at risk are few overall (0.6%) but concentrated—go-zero has 7.1% of required tests in blocked units, affecting 15 of 23 milestones.

RQ2: Realisation

• Blocked units fail in 80.6% of (unit, run) pairs vs. 48.4% for comparable new-test units.
• Odds ratio of 4.64 (95% CI 2.92–7.37) from logistic regression with unit size and repository as covariates.
• Blocked units fail 1.67× as often as comparable units (95% CI 1.32–2.04).
• Counterfactual result: Renaming one hidden identifier stops compilation in 126 of 128 Go cases, and failing tests are 99.5% of those the audit predicts.

RQ3: Impact on Scores and Rankings (Table 3)

Critical finding: The leaderboard ranking is robust—changes fall within what random removals of the same size produce. However, per-repository scores shift: go-zero mean score rises from 38.33 to 42.08, and resolve rate from 4.35% to 5.71%.

RQ4: Precision and Cost

• Precision: 89.2% (58/65 flagged units correct; 95% Wilson interval 79.4–94.7%)
• Recall: 92.9% (13/14; 68.5–98.7%)
• Runtime: 28.1 minutes on a 16-core workstation, ≤1,333 MiB memory, no model calls.

RQ5: Other Benchmarks (Table 4)

Key insight: Prevalence follows construction—where specifications are LLM summaries or release notes, a third or more of tasks are affected; where screened or written for tests, almost none. The harness decides whether a gap bites (e.g., SWE-Chain's harness reports failed imports back to the agent, reducing the lift to 1.12).

---

Theoretical and Practical Implications

Theoretical Implications

1. Benchmark validity: A specification–oracle gap means a benchmark measures upstream implementation reproduction alongside task-solving ability—the "generative analogue of an annotation artefact" in NLP.

2. Chain inconsistencies: Long-horizon benchmarks introduce a new failure class absent from single-task benchmarks: dependencies on other milestones' code that no agent could possibly satisfy in all topological orders.

3. Blast radius modelling: The paper formalizes how a single hidden identifier can block an entire build unit and cascade through dependent milestones.

Practical Implications

1. Corrected filter lists: The paper releases lists (3,321 conservative or 3,964 full entries) that waive tests with hard hidden dependencies, each with a reason, passing the harness's validation.

2. Specification addenda: For all 25 affected milestones, the authors generate addenda stating required interface conventions (153 total, e.g., "the package rest/handler must declare valueSSE (relied on by 84 required tests)")—keeping the tests and removing guessing.

3. Recommendations for benchmark builders:

   • Run a dependency audit when generating specifications.
   • Replay reference solutions in several topological orders to repair order-dependent failures.
   • Isolate build units during grading so one compile failure can't erase a milestone.
   • Publish per-test results to enable auditing.

---

Conclusion

Main Takeaways

The paper demonstrates that generated specifications paired with verbatim tests—while cheap and realistic—let tests depend on information tasks never state. In SWE-Milestone, a quarter of graded milestones contain such dependencies; affected build units fail 1.67× as often as comparable units, and renaming a single hidden identifier in the reference breaks its unit exactly as the audit predicts.

Future Directions

1. Measuring the effect of specification addenda with new agent runs (the addenda are released but not yet evaluated).
2. Extending the audit to cover behavioural gaps (escaping rules, weighting schemes) beyond name-, path-, and literal-level dependencies—the current counts are a lower bound.
3. Addressing memorisation vs. skill: The paper finds reproduction of hidden names is a stable trait of models, unrelated to training cutoff, suggesting agents may pass via memorisation rather than reasoning.

Limitations

• Realisation is observational (groups may differ in difficulty); the counterfactual covers Go only.
• The audit is name-level, not type-checking; it misses methods whose names exist on other types and may over-flag interface methods from external libraries.
• DeepSWE is not an out-of-sample test (its review prompted two of the audit's rules).
• SRS matching is lexical—a specification may describe a name in other words (over-reporting) or mention an identifier without meaning it as an interface (under-reporting).