Finding Side-Effect Loopholes in Verifiable Rewards

Summary (Overview)

• Core finding: This paper adapts mutation testing to audit verifiers in reinforcement learning with verifiable rewards (RLVR) benchmarks, revealing that verifiers in AppWorld and WorkArena pass runs that complete the task and make unrequested side-effect changes.
• Key results: All false positives came from unrequested changes (6 in AppWorld, 36 in WorkArena); none came from missing or wrong work. WorkArena's form verifier passed forms with an extra unrequested field in all 36 valid pairs, and AppWorld's tests passed duplicate writes in 6 tasks across 2 templates.
• Reward implications: Three recent RL methods (CANOPY, Entropy-Preserving RL, ProCredit) compute rewards from AppWorld's tests; the duplicate-write run earns the same maximum reward as the correct run under all three.
• Simple repair: Adding one count assertion per affected AppWorld template rejected all six duplicates and passed all six correct runs.
• Methodological contribution: The paper establishes a rigorous audit framework distinguishing false positives from harmless edits via an independent end-state check, providing a lower bound on verifier gaps.

---

Introduction and Theoretical Foundation

Background

The paper addresses a critical gap in RLVR: when a benchmark's verifier supplies the reward, a run that does the task and causes a side effect may pass if the verifier only checks the requested change (the postcondition) but not the frame condition—that nothing else changed.

Key Definitions

Postcondition: Each requested record exists with the requested values. Frame condition: Nothing else changes (including no second copy of a requested record).

The frame condition is hard to check fully because it requires comparing the entire end state with the expected one—the classic frame problem [McCarthy and Hayes, 1969, Borgida et al., 1995].

Related Problems

• Reward hacking: Exploiting a gap between what a reward measures and what its designer intended [Skalse et al., 2022, Pan et al., 2022]
• Negative side effects: An agent completes its task and also disturbs something the reward never mentions [Amodei et al., 2016]

The Core Question

Do RL reward verifiers notice a run that does the work and also makes an unrequested change?

The Mutation Testing Analogy

The paper adapts mutation testing [Jia and Harman, 2011], which judges a test suite by planting faults in a program, to audit verifiers:

• Verifier plays the role of the test suite
• Agent's run plays the role of the program
• Edit to a correct run plays the role of a planted fault
• End-state check acts as the oracle [Barr et al., 2015], distinguishing harmless edits (equivalent mutants) from true gaps

Adversary Model

The adversary knows the task, controls the run, acts only through the benchmark's normal actions from a clean start, and cannot modify the verifier, the evidence the verifier reads, or the environment after the run starts. This excludes reward tampering [Everitt et al., 2021].

---

Methodology

Three Kinds of Tests

The Five-Step Audit Pipeline

After a person writes the edits and a harness for each benchmark, five steps run with no per-run human decision and no model call:

1. Listing the edited runs
2. Replaying them from a clean start (reset database in AppWorld; fresh ServiceNow episode in WorkArena)
3. End-state check: Reads what the run changed before the verifier is called, using queries separate from the verifier's code
4. Verifier calls: The unmodified verifier grades the run at a fixed version
5. Classification: A fixed rule assigns each run an outcome

Outcome Classification

• Rejected: Verifier fails the run
• Harmless edit: Verifier passes and end state is correct
• False positive: Verifier passes and end-state check shows the wrong change the edit names

AppWorld End-State Check

The end-state check counts writes to each table and compares counts with the correct run's. This is a lower bound: the count errs in one direction (an edit nobody wrote finds nothing; a check comparing no values can miss a wrong one).

Applicability Conditions

The method applies only to benchmarks that provide:

1. Reset-able start states shareable between correct and edited runs
2. A correct run the verifier passes
3. Programmatic access to agent actions
4. A deterministic verifier callable unmodified and re-gradeable as a patched copy
5. An end-state check that sees each edit's change
6. A way to pin what the verifier expects

---

Empirical Validation / Results

Setup

• AppWorld: 48 tasks (16 templates × 3 versions), 8 templates in official train split, 8 in dev split
• WorkArena: 13 single-step tasks (4 forms, 9 catalog items) × 3 configurations

AppWorld: Repeated Writes

The unrequested-change edit repeats the last write of the correct run:

• 39 of 48 runs passed; 33 were harmless (repeat changed nothing)
• 6 false positives across all 3 versions of 2 templates:
  • Work-dinner payment task: Repeated last request leaves an extra request and notification
  • Song-queue task: Same song queued a second time

Why the tests miss it: Payment tests match new requests to expected shares by receiver (so a duplicate merges); song-queue tests compare the queue as a set (cannot see a duplicate).

AppWorld: The Count Assertion Repair

The repair also rejected none of 13 natural agent runs the original tests accepted.

WorkArena: Form Verifier After Submit

• 36 complete and valid pairs (12 candidate fields × 3 seeds, in 2 form templates: incident and change request)
• Every pair: Edited run saved the requested value in the extra field, all task fields matched, and the verifier passed both runs
• 48 unresolved pairs: 27 hardware-asset pairs failed the configuration-fixed check; 21 stopped at hidden/read-only/disabled inputs

The verifier's monitor (a page script) is meant to flag unrequested field changes, but the flag was unset when the verifier read it in all 147 episodes that reached the verifier.

WorkArena: Per-Step Validation

The probe explains why: a drop-down set the flag before submit (so per-step validation catches it), but text/reference fields set the flag only during the submit click, which the page load after submit loses.

Aggregate Results

† The 36 complete and valid pairs of the paired experiment (verifier called once after submit).

Reward Implications

AppWorld's work-dinner payment template is in the official train split (difficulty 3); the song-queue template is in the dev split (difficulty 1).

Under all three reward formulas (CANOPY: 1 when all tests pass; Entropy-Preserving RL: fraction passing; ProCredit: all-tests reward + progress term $c \cdot \Delta(\text{fraction of initially failing tests that pass})$), the duplicate earns the correct run's maximum reward for any $c > 0$.

---

Theoretical and Practical Implications

Theoretical Significance

1. Systematic loopholes are detectable: The paper demonstrates that verifier gaps can be found without model-generated adversaries, adaptive search, or natural failure corpora—hand-written edits suffice.

2. The frame condition is the weak point: Every false positive came from an unrequested change; none from missing or wrong work. Both benchmarks include checks meant for unrequested changes, yet these runs got past both.

3. Reward hacking can be latent: The duplicate-write loophole is systematic (holds across a task template), which prior work [Egashira et al., 2026] shows can stall or collapse RL training. Yet none of AppWorld's 1,277 released passing runs exploits it—the loophole is latent.

4. Score- and reward-based audits are insufficient: Audits judging exploits by score or reward cannot separate the duplicate from the correct run, since their scores and rewards are equal. Only an end-state check against a correct run reveals the difference.

Practical Implications

• Simple repairs work: One count assertion per template rejected all duplicates and failed no correct runs—a minimal, practical fix.
• Verifier call frequency matters: WorkArena's per-step validation catches some extra fields (drop-downs, text/reference filled first) but not others (text/reference filled last).
• Authors' responsibility: Before a verifier becomes a reward, authors can count what a run adds or compare the whole end state with the expected one.

Relationship to Prior Work

The paper extends Sun [2026]'s state-mutation approach: while Sun's snapshot method yielded no applicable collateral mutation in AppWorld, this paper's edits run through AppWorld's own API, so the duplicate request and its notification reach the database without being built by hand. It also connects to work on outcome validity [Zhu et al., 2025], verifier false positives in RLVR [Zhang, 2026], and the finding that RLVR still learns from a noisy verifier as long as its true-positive rate exceeds its false-positive rate [Rad et al., 2026].

---

Conclusion

Main Takeaways

1. Mutation testing audits verifiers: A benchmark verifier is a test suite for agent behavior, so mutation testing can check it.
2. Side-effect loopholes exist in both benchmarks: Runs that did the task and made one unrequested change still passed.
3. The gap lies in the frame condition: Missing or wrong work produced no false positives.
4. RL rewards inherit the gap: Three recent RL methods give the duplicate the correct run's maximum reward.
5. Simple repairs are effective: One count assertion per affected template rejected all duplicates and failed no correct runs.

Open Questions

1. Does RL optimization find the duplicate on its own? Training policies on the original and repaired AppWorld tests would answer this.
2. Can automated auditing replace hand-written edits? When a language model reads a verifier and proposes candidate loopholes, the bottleneck shifts to confirming which candidates produce an unauthorized end state and repairing the verifier without rejecting valid alternative runs.

Key Quote

"Before a verifier becomes a reward, its authors can count what a run adds or compare the whole end state with the expected one."