Summary of "Built on Broken Assumptions (BOBA): Why Coding Agents Cannot Be Left Alone"

Summary (Overview)

• Core finding: Coding agents complete 95% of individual requests but only 51% of chains of 20 consecutive requests built on their own prior work, revealing a "Built on Broken Assumptions" phenomenon where unverified mistakes compound.
• Key contribution: Introduces BOBA-Bench, a benchmark of 280 chains of 20 consecutive requests derived from 140 real open-source Python repositories (5,600 total requests), designed to evaluate agents in continuous, unattended development scenarios.
• Root cause identified: The degradation is not caused by worse code quality (verified agent code is nearly as good a base as human code) but by silent misses—mistakes agents don't notice because their own tests pass, which then become broken foundations for subsequent requests.
• Verification gap: Agents cannot self-verify—when asked to review implementations, they are correct only 53% of the time (barely better than a coin flip). A single external "verification failed" message plus one repair round lifts chain completion from 51% to 59%.
• Implications: Current benchmarks that test agents one request at a time on human-written code dramatically overestimate real-world performance in autonomous, multi-request development scenarios.

Introduction and Theoretical Foundation

Background and Motivation

Coding agents (e.g., Codex, Claude Code, OpenHands) are increasingly deployed as autonomous developers that handle chains of requests, where each request builds on the previous one's output. While trajectory studies confirm that agents:

• Rarely catch mistakes that nothing flags for them
• Report wrong work as complete

Existing benchmarks evaluate agents on:

• A single request
• Human-written code
• Verified once

This creates a fundamental mismatch between how agents are evaluated and how they're actually used in practice.

Prior Work Limitations

The paper identifies gaps in existing repository-evolution benchmarks:

Key insight: Unattended development can only be measured by controlling what it lacks: verification. The paper argues that the central variable should be whether code was verified, not who wrote it.

Methodology

BOBA-Bench Construction

The benchmark transforms real repository commit histories into chains of 20 requests through a four-stage process:

1. Mine history: Consecutive commits that change tests and source together become a request if new tests fail before and pass after the human patch
2. Produce spec: Claude Opus 4.8 writes specifications from test diffs, commit messages, and issues (never from source code)
3. Audit specifications: Three model families (GLM-5.2, Fable 5, GPT-5.6) must unanimously accept identical text
4. Certify chains: 20 consecutive requests form a chain, sealed with frozen dependencies, where human patches must pass every hidden test at every state

Evaluation Protocol

The autonomous evaluation protocol delivers requests one at a time:

• Each request in a fresh conversation (agent remembers nothing of earlier requests)
• Graded against hidden tests after every request
• Agent told nothing about its performance
• All 20 requests run regardless of failures

Evaluation Settings

Five experimental settings isolate different variables:

Models Evaluated

Ten models on all 280 chains (8 attempts each): Claude Opus 5.5, Opus 5, Opus 4.8, Fable 5, GPT-6 Astra, GPT-6.1-sol, GPT-5.6-sol, GLM-5.3, GLM-5.3-Flash, Kimi K3.

Empirical Validation / Results

Chain Completion Findings

Table 2: Chain completion of ten coding agents on BOBA-Bench

Key results:

• Average chain completion: 51% (vs. 66–84% individual request completion)
• Strict pass@1 (no request missed, no test broken): only 39% on average
• Silent misses occur even within the first five requests across all models
• 23–43% of attempts break a previously passing test later in the chain

Authorship vs. Verification Analysis

Figure 4: Verified foundations hold whoever built them

Key insight: When code is verified, agent-written code is nearly as buildable as human code. The degradation comes from unverified mistakes, not from authorship.

Failure Root Causes

Four primary causes of silent misses identified:

1. Regressions (most common): Agents break behavior the request didn't touch (37–55% of first failures)
2. Missed requirements: Part of specification left unimplemented (14% of failures)
3. Wrong logic / integration errors: Wrong implementation or unreachable code (18% of failures)
4. Trusting own tests: A green local test run barely lowers miss probability (8.3% vs. 8.6% for all submissions)

Cross-Model Review Experiment

Figure 5: No model is a reliable reviewer

Key finding: No model separates wrong from correct implementations well enough to serve as a check. The best Youden J is only 0.15 (where 1.0 is perfect).

Effect of External Verification

Table 41: Verified under the handoff protocol

Key findings:

• A single "verification failed" message + one repair round lifts chain completion by 1–22 points
• Naming the failing test achieves in one round what insisting achieves in ~3 rounds
• Repaired states are no longer broken foundations: next request fails on them no more often than on certified states
• False alarms (rejecting correct work) cost little: ~98% of phantom repair rounds leave work passing

Theoretical and Practical Implications

For System Builders

• Keep the verification line open in both directions: The verifier must see results; the model must receive verdicts
• Widen what the verifier sees rather than multiplying rounds: naming failing tests beats repeated generic rejection
• Multi-agent systems are especially exposed: models reviewing finished implementations barely beat chance, so the verifier's line must reach subagents directly

For Model Builders

The paper calls for prioritizing certification as much as implementation:

1. Agents must tell wrong implementations from correct ones (currently barely better than chance)
2. Agents must repair a miss in one round when told only "verification failed" (currently repair only a third to a half)
3. Both must be achievable without a supervisor

For Benchmark Builders

Three properties enable observing the phenomenon:

1. Agent's work persists across requests
2. Correctness decided by a contract the agent cannot see or rewrite
3. Requests depend on one another

Single-request benchmarks have none of these properties, which is why the phenomenon has gone unmeasured.

For Users

Two practical recommendations:

1. Check before you build on it: Run or test a request's output before issuing the next request
2. Say what you saw: Even imperfect verification feedback helps; occasional false alarms cost little

Conclusion

The paper demonstrates that coding agents build on broken assumptions: they complete most individual requests but only about half of the chains those requests form, primarily because they cannot detect their own mistakes. The problem is not the quality of code they write (verified agent code is nearly as good as human code) but their inability to verify their own work.

Future directions:

• Model builders should make certifying a request as much a goal as implementing it
• A single external "that's wrong" recovers part of the loss, but more than half of misses survive a repair round
• The benchmark and all artifacts are released at https://github.com/amazon-agi/BOBA-bench for reproduction and extension

Bottom line: Coding agents are only as reliable as the checks they can reach—without verification, every request that follows is built on broken assumptions.