# The Verification Horizon: No Silver Bullet for Coding Agent Rewards > Verification is now the harder problem for coding agents and must co-evolve with the generator to prevent reward hacking. - **Source:** [arXiv](https://arxiv.org/abs/2606.26300) - **Published:** 2026-06-27 - **Permalink:** https://picx.dev/p/4mA5OF - **Whiteboard:** https://picx.dev/p/4mA5OF/image ## Summary ## Summary (Overview) - **Verification is now harder than generation**: As coding agents become more powerful, producing candidate solutions has become easier while reliably verifying them has become the *harder* problem—an inversion of the classical intuition. - **Three dimensions of verification quality**: The paper characterizes verification signals along **scalability**, **faithfulness**, and **robustness**, and argues that achieving all three simultaneously is the central difficulty. No single mechanism can solve verification once and for all. - **Four distinct reward constructions are studied**: (1) test-driven rewards for SWE tasks, (2) rubric/interactive judges for frontend tasks, (3) direct user feedback as verifier for real-world agent tasks, and (4) an automated agent evaluator for long-horizon tasks. - **Quantitative results**: Behavior monitoring reduces the hacked resolved rate from **28.57% to 0.56%** while improving the clean resolved rate from **40.22% to 60.53%** across three SWE-Bench variants. Span-KTO (using user feedback) achieves up to **+13.3 pp** improvement on a private benchmark. Evaluator-filtered RFT improves from **11.41 to 23.52** on OpenHands. - **Core observation**: Verification must **co-evolve** with the generator—no fixed reward function can remain effective as policy capability grows. ## Introduction and Theoretical Foundation The paper frames the verification challenge for coding agents through two foundational insights: 1. **Inversion of the classical asymmetry**: Brooks’s (1987) “No Silver Bullet” lesson from software engineering is reinterpreted for coding agents. As foundation models develop stronger reasoning and harness engineering advances, *generating* candidate solutions has become easy, but *reliably verifying* them has become the harder problem. 2. **The proxy–intent gap**: Every verifier is only a **proxy** for human intent, never the intent itself. This creates a twofold difficulty: - **Faithfulness is inherently hard**: Intent is underspecified by nature—users often cannot articulate full expectations until a counterexample exposes an omission. - **Optimization widens the gap**: When a proxy is used as a reward signal, the generator learns to exploit the divergence between proxy and intent, leading to **reward hacking**—not a bug that can be patched, but an inevitable consequence of sustained optimization toward an imperfect objective. The paper formally connects this to computability theory via **Rice’s theorem** (Rice, 1953): every non-trivial semantic property of a program is undecidable, independently supporting the claim that perfect verification is impossible. Three dimensions of verification signal quality are defined: - **Scalability**: Can the signal be produced cheaply at the scale required for training? - **Faithfulness**: How much of the true user intent does the signal reflect? - **Robustness**: Can the verifier’s judgments hold across diverse/adversarial inputs and withstand optimization pressure from a strengthening generator? Most existing approaches satisfy only two: unit tests (scalable + robust, but narrow), LLM judges (scalable + faithful, but vulnerable to exploitation), human expert review (faithful + robust, but cannot scale). The intersection of all three—*cheap, deep, and resistant to gaming*—remains missing, motivating a **co-evolutionary** approach. ## Methodology ### 2. Test-Driven Rewards for SWE-like Tasks **Automated Data Pipeline** (SWE-Universe): Given an issue-linked pull request, the pipeline separates the change into a fix patch and a test patch, restores the repository to the pre-fix state, and constructs a Dockerized environment with a unified verifier `evaluation.sh` whose binary pass/fail result serves as the test-driven reward. **Agentic Quality Judge**: To address reward faithfulness (false positives / false negatives), the paper decomposes semantic quality into two dimensions: - `instruct_clear`: whether the instruction sufficiently expresses the intended task. - `instruct_ut_align`: whether the tests faithfully operationalize the instruction. An agentic judge (based on MiniSWEAgent) actively explores the repository—inspecting files, executing commands, reading tests—and produces dimension-level judgments aggregated into an `overall_good` label. The judge is evaluated on a human-annotated benchmark (Table 1). **Behavior Monitoring**: To mitigate reward hacking (e.g., solution artifact retrieval, test tampering), the paper introduces a **trajectory-level behavior monitor** during RL. For each rollout, the monitor audits command history, network accesses, git operations, etc., against a pattern set \( \mathcal{P} \). When a high-risk pattern is matched, a **token-level penalty** is applied to reduce the reward. The pattern set is updated iteratively: after each training interval, new shortcut strategies discovered by the strengthening policy are added to \( \mathcal{P} \). ### 3. Interactive Judge for Frontend Tasks **Rubric-based Static Judge**: Decomposes evaluation into structured dimensions (Functional, Content, Visual, Layout, UX, Technical). The judge takes rendered screenshots and source code as input, producing a score. Rubrics improve inter-annotator agreement and cross-judge consistency (Table 4). **Agentic Interactive Judge**: A three-stage pipeline (Figure 7): 1. **Action Planning**: Given the rendered page and rubrics, an action planner generates a complete action list in a single pass (using a pre-defined vocabulary: click, scroll, navigate, fill form, press key, etc.). 2. **Execution**: A Playwright-based render server executes actions in a live browser and records the interaction trace (screen recordings, state changes). 3. **Scoring**: A judge model evaluates sampled frames from the trace against rubric criteria, producing the final score. This approach avoids length-exploitation hacking (to which static judges are susceptible) because the reward derives from runtime behavior rather than source code. ### 4. User Feedback as Verifier **Feedback Annotation Pipeline**: From real user–agent conversations (125,528 trajectories, 535,737 rounds), an LLM-as-Judge (Qwen-Plus) annotates each round with: - **polarity**: positive, neutral, or negative. - **confidence**: high/medium/low. - **negative reason category** (execution error, misunderstanding, omission, overaction, inefficiency, communication). - **user_fairness**: whether the user’s evaluation is objectively fair. Key characteristics (Figure 8): highly asymmetric polarity (76.6% neutral, 20.0% negative, 3.5% positive); negative signals are high-confidence; errors concentrate in execution (56.6%) and misunderstanding (21.1%). **Training Methods**: - **Reweight SFT (RW-SFT)** : Applies differentiated loss weights to tokens of different polarities: \[ \mathcal{L}_{\text{RW-SFT}}(\theta) = -\mathbb{E}_t \left[ w(p_t) \log \pi_\theta(y_t | x, y_{