Summary (Overview)

• New benchmark introduced: SWEeper-Bench contains 200 tasks built from real bug fixes in 200 widely used open-source web applications, testing whether coding agents can discover bugs through interactive testing rather than merely fix pre-reported issues.
• Key finding: The best of 15 frontier agents passes only 59.0% of tasks, with 9 agents passing fewer than half, demonstrating the benchmark is far from solved.
• Main bottleneck identified: Through stage-by-stage trajectory analysis, the largest performance drop occurs at identifying the buggy behavior — agents reach the relevant feature in 75.4% of trajectories but only identify the bug in 50.0%.
• Finding vs. fixing gap: When given detailed bug descriptions instead of open-ended prompts, all five tested agents pass ~96% of tasks, showing the core difficulty lies in producing the bug report, not acting on it.
• Agentic verifier: Repairs are judged by what a user would observe in the browser, using an agentic verifier that achieves near-perfect accuracy and consistency (within 1 percentage point of ideal rates).

---

Introduction and Theoretical Foundation

Coding agents (e.g., Codex, Claude Code, Cursor) can now generate interactive software on the fly, but such software is inherently difficult to debug. Bugs often manifest only after specific sequences of user interactions—e.g., "a board shared with a colleague never shows up in the colleague's sidebar."

The core problem: The development loop remains reactive:

• Users must discover bugs in the wild
• Users report bugs to agents
• Agents wait for feedback before fixing

This loop is slow and expensive. As agents increasingly generate interactive software directly for users (e.g., building a team offsite app with sign-up forms, schedules, and budget trackers), no testing team ever sees the output. The authors argue agents must instead "close the loop on their own and sweep out as many bugs as possible before presenting their software."

Limitations of existing benchmarks:

• SWE-bench and successors assume bugs are already identified (via user-reported issues)
• They measure diagnosis and repair but not discovery
• They grade with hidden unit tests that check internal state rather than user-observable behavior, potentially rejecting valid alternative fixes

Research question: Can agents discover and fix the problems users will encounter in interactive software on their own?

---

Methodology

Task Construction

Each task pairs a buggy application with an open-ended prompt and two behavior tests:

1. Source selection: Tasks are built from merged pull requests fixing user-facing bugs in open-source web applications. The parent commit is the buggy version; the merged fix is the reference repair. Each of the 200 tasks comes from a different application across seven domains (Figure 4):

   • Content & knowledge (40 tasks), e.g., Ghost (55k stars), Outline (41k)
   • IT, security & dev tools (35), e.g., Uptime Kuma (92k), NocoDB (65k)
   • Collaboration & communication (31), e.g., Focalboard (26k), listmonk (24k)
   • Files, media & sharing (29), e.g., Stirling-PDF (93k), AList (50k)
   • Finance & business (28), e.g., Maybe (54k), Actual (29k)
   • Personal & lifestyle (22), e.g., Memos (63k), SiYuan (46k)
   • Commerce & payments (15), e.g., Bagisto (28k), EverShop (10k)

2. Open-ended prompt: A one-sentence request naming only a product area, e.g., "Find and fix issues in Focalboard's board sharing and membership." The agent never sees the original issue, PR, reference patch, repository history, or behavior tests. Outbound network access is blocked.

3. Agent environment: Isolated sandbox with buggy source code, running instance, command interface, and Chromium browser with Playwright.

Agentic Verifier

• Applies the agent's code changes to a fresh copy of the buggy application
• A verifier agent operates the repaired application in the browser without seeing source code
• Each task has two behavior tests:
  • Target test: Reproduces interactions exposing the bug, checks intended behavior restored
  • Preservation test: Checks nearby features still work
• A task passes only if both tests pass

Verifier reliability (Table 1):

Arrows mark ideal rates: 0% for target test on buggy version, 100% otherwise. All configurations come within one percentage point of ideal.

---

Empirical Validation / Results

Benchmark Results (Table 2)

Nearly all failures come from the target behavior test; preservation pass rates stay between 97.5% and 100%. Similar pass rates hide very different costs (up to 30x differences).

Stage-by-Stage Analysis (Figure 7)

Five cumulative stages per trajectory:

1. Identified target feature: 75.4% of trajectories (range 63.5%–91.4%)
2. Identified buggy behavior: 50.0% ← largest drop
3. Diagnosed root cause: 43.8%
4. Fixed the buggy behavior: 41.0%
5. Maintained existing behaviors: 40.8%

Key failure patterns:

• Agents often reach the feature but never run the interaction exposing the bug
• Even when triggering the failure, agents may not recognize it (e.g., Claude Opus 5 trusted unit tests that expected no sidebar update and concluded the bug was intended behavior)
• Misdiagnosis of root cause (e.g., blaming client-side encoding instead of server-side HTML escaping)

Behavioral Tendencies

• Longer work ≠ better results: Grok 4.6 averages 12 min/task (59.0% pass); Qwen 3.8 Flash averages 102 min/task (57.5% pass)
• More browser use ≠ better results: Browser actions range from 371–1,082 per task (17%–62% of turns), but neither metric tracks pass rate

Ablation Studies

Browser prompting matters (Table 3):

Without explicit instruction, agents almost never open the browser, preferring code reading, shell commands, and HTTP requests.

Finding is harder than fixing (Table 4):

When given a detailed bug description, all agents pass ~96%, and the gap between agents nearly disappears.

Harness effect (Table 5): GPT-5.6 Luna performs best in its native harness (Codex, 42.5%), ranging down to 35.0% with Claude Code or Muse Code.

Time budget: Pass rates rise from 20 to 40 minutes but plateau beyond, showing more time alone does not close the gap.

---

Theoretical and Practical Implications

Implications for Agent Design

1. The testing gap is the core challenge: Agents must (a) choose which scenarios to try, (b) run them through the interface, and (c) recognize failures when they appear. Recognition is the weakest link—agents often see the bug but don't recognize it as a bug.

2. Existing tests can mislead: Claude Opus 5 on the Focalboard task trusted unit tests that never checked the sidebar and concluded the empty sidebar was intended behavior. Agents must judge behavior by what the software is meant to do, not only by what tests expect.

3. Interactive testing is not yet a spontaneous capability: Agents rarely use the browser unless explicitly instructed, suggesting this behavior needs to be incentivized or trained.

4. Verifier design matters: Judging repairs by user-observable behavior (rather than hidden unit tests) accepts any correct implementation and could reward testing behavior during training.

Implications for Evaluation

• Existing benchmarks (SWE-bench and successors) measure the stages after a bug report exists
• SWEeper-Bench evaluates the step that produces the report—a critical missing capability
• The benchmark measures recall (one known bug per task) rather than precision, providing a focused signal on bug discovery

---

Conclusion

SWEeper-Bench demonstrates that current agents fix bugs reliably once given a bug report (96% pass rate) but struggle to find bugs autonomously (59% best pass rate). The gap lies in testing, not repair—specifically in designing interactive tests and recognizing failures when they occur.

Future directions:

• Agents that use applications the way users would and judge behavior by intent
• Verifiers that check user-observable behavior to reward testing during training
• Extension beyond web applications to other interactive software
• Evaluation of visual design, ease of use, and responsiveness (currently limited to functional correctness)

Limitations acknowledged:

• Covers only web applications
• Scores one known bug per task (measures recall, not precision)
• Fixes to other real issues earn no credit
• Unnecessary changes unpenalized unless they break preservation tests

Closing question: As agents take on more of building software, it is worth asking more of them than fixing the bugs we report: can they find those bugs before we do?