SWE-Bench Counter-Edit: Summary of Key Findings
Problem Definition: The paper introduces a new evaluation protocol called Counter-Edit, which tests how coding agents handle user interventions that directly conflict with the task they are solving. Unlike existing benchmarks where user feedback is textual, this protocol injects small, plausible but incorrect code edits mid-task, forcing the agent to detect, reconcile, and potentially revert these changes to successfully complete the original objective.
Methodology:
- Starting from 300 SWE-bench Verified tasks, a controlled "Counter-Edit" condition was created by:
- Identifying task-relevant code regions (via multiple autonomous runs)
- Injecting small, buggy code modifications into those regions
- Validating each edit with three conditions: (1) edit alone fails the task, (2) reference patch solves the task, (3) edit + reference patch still fails (Equation 3)
- Delivering the edit mid-task with a contextual user message when the agent touches relevant code
- Evaluated 9 coding models (Claude Opus 4.8, GPT 5.5, GLM 5.1, MiniMax M2.7/M2.5, Qwen 3.7 Max, Qwen3-Coder, Kimi K2.6, DeepSeek V4 Pro) across 100+ tasks, comparing autonomous ("Vanilla") vs. Counter-Edit performance
Key Findings
1. Performance Degradation
- Average resolve rate drops by 6.ales points across models under Counter-Edit compared to Vanilla performance
- Nearly 16% of majority-solved Vanilla tasks flip to unresolved under intervention
- Solved-to-unresolved transitions outnumber the reverse by ~3:1
2. Model Heterogeneity
| Model | Vanilla | Counter-Edit | Δ |
|---|---|---|---|
| Claude Opus 4.8 | 81.3 | 79.5 | -1.8 |
| GPT 5.5 | 80.5 | 79.2 | -1.3 |
| GLM 5.1 | 70.5 | 65.Top models are resilient: | |
| Model | Counter-Edit Resolve | Drop (pts) | |
| --- | --- | --- | |
| Claude Opus 4.8 | 60.0 | -10.0 | |
| DeepSeek V4 Pro | 50.0 | -8.0 | |
| GPT 5.5 | 36.0 | -4.0 | |
| GLM 5.1 | 20.0 | -14.0 | |
| Qwen3-Coder | 28.0 | -8.0 |
7. Resource Utilization
Resource consumption increases substantially under Counter-Edit:
- Average +14.屏幕上 calls (from ~25 to ~29)
- Average +37K output tokens (from ~106K to ~143K)
- Stronger models (GPT 4.1, Claude 3.7) show minimal degradation
- Weaker models (DeepSeek V3, GLM 4.7) show significant degradation
Why the gap persists for weaker models:
-
Pattern authentication: Weaker models cannot distinguish the user's malicious edit from the legitimate task specification. They treat the Counter-Edit as authoritative guidance, persisting along the wrong trajectory rather than questioning it.
-
Insufficient verification: After following the user's lead, they fail to verify whether their resulting workflow prospectively satisfies the task's expected benefit—they accept the edited instructions at face value.
Recommendations for application design:
- For weaker models: Design Counter‑Edits with more explicit traceability (e.g., timestamps, clearly-marked suggestions) so they can distinguish them from authoritative instructions.
- For stronger models: Ensure the user‑edit is genuinely plausible (not overly simplistic), as stronger models excel at background justification but rarely re‑derive explicit reasoning from prior outputs.
Evaluation Protocol: Counter-Edit Benchmark
Implementation Details
| Component | Description |
|---|---|
| Tasks | 60% from SWE-bench Verified; 40% from SWE-bench Pro |
| Edit size | 1 counter-edit per 3 tasks (avg 4.0 lines changed) |
| Cooperativity | User proceeds with the task before/after the edit |
| Metrics | Pass@1 for code success, plus execution-quality metrics |
Bottom Line: Shared-Workspace Edits uniquely stress-test whether an agent can detect, reconcile against, and recover from external modifications to the working directory. While top models (GPT 5.5, Claude 3.7, GLM 5.1) remain robust, open-weight models (DeepSeek V4 Pro, Qwen 3.7 Max, MiniMax M2.5) are significantly more sensitive to such interventions—degrading by 11–17 points. The gap persists even when the user's edit is simple, benign, and explicitly câblée as advice.
Related papers
- LEGO-RL: Harness-Native Reinforcement Learning for Coding Agents
LEGO-RL enables native coding-agent harnesses to train with policy-gradient RL, boosting SWE-bench Verified scores by up to 9.4 points without altering their control flow.
- Understanding Axes of Difficulty For Long Context Tasks Via PredicateLongBench
PREDICATELONGBENCH shows frontier LLMs collapse on simple long-context retrieval tasks when predicate arity, quantifier complexity, or adversarial decoys increase, even with unchanged answers.
- Hardening Agent Benchmarks with Adversarial Hacker-Fixer Loops
The hacker-fixer loop, using verifier access and a shared defense pool, automates hardening of agent benchmarks, eliminating all tested reward-hacking attacks on KernelBench and reducing them on Terminal Bench.