# Finding Side-Effect Loopholes in Verifiable Rewards

> Mutation testing of RLVR verifiers reveals all false positives stem from unrequested side-effect changes, not missing work, and simple count assertions can fix these loopholes.

- **Source:** [arXiv](https://arxiv.org/abs/2610.09142)
- **Published:** 2026-10-31
- **Permalink:** https://picx.dev/p/43lOuG
- **Whiteboard:** https://picx.dev/p/43lOuG/image

## Summary

# Finding Side-Effect Loopholes in Verifiable Rewards

## Summary (Overview)

- **Core finding**: This paper adapts mutation testing to audit verifiers in reinforcement learning with verifiable rewards (RLVR) benchmarks, revealing that verifiers in AppWorld and WorkArena pass runs that complete the task *and* make unrequested side-effect changes.
- **Key results**: All false positives came from unrequested changes (6 in AppWorld, 36 in WorkArena); none came from missing or wrong work. WorkArena's form verifier passed forms with an extra unrequested field in all 36 valid pairs, and AppWorld's tests passed duplicate writes in 6 tasks across 2 templates.
- **Reward implications**: Three recent RL methods (CANOPY, Entropy-Preserving RL, ProCredit) compute rewards from AppWorld's tests; the duplicate-write run earns the same maximum reward as the correct run under all three.
- **Simple repair**: Adding one count assertion per affected AppWorld template rejected all six duplicates and passed all six correct runs.
- **Methodological contribution**: The paper establishes a rigorous audit framework distinguishing false positives from harmless edits via an independent end-state check, providing a lower bound on verifier gaps.

---

## Introduction and Theoretical Foundation

### Background

The paper addresses a critical gap in RLVR: when a benchmark's verifier supplies the reward, a run that does the task *and* causes a side effect may pass if the verifier only checks the requested change (the **postcondition**) but not the **frame condition**—that nothing else changed.

### Key Definitions

> **Postcondition**: Each requested record exists with the requested values.
> **Frame condition**: Nothing else changes (including no second copy of a requested record).

The frame condition is hard to check fully because it requires comparing the entire end state with the expected one—the classic **frame problem** [McCarthy and Hayes, 1969, Borgida et al., 1995].

### Related Problems

- **Reward hacking**: Exploiting a gap between what a reward measures and what its designer intended [Skalse et al., 2022, Pan et al., 2022]
- **Negative side effects**: An agent completes its task and also disturbs something the reward never mentions [Amodei et al., 2016]

### The Core Question

> Do RL reward verifiers notice a run that does the work and also makes an unrequested change?

### The Mutation Testing Analogy

The paper adapts **mutation testing** [Jia and Harman, 2011], which judges a test suite by planting faults in a program, to audit verifiers:

- **Verifier** plays the role of the test suite
- **Agent's run** plays the role of the program
- **Edit to a correct run** plays the role of a planted fault
- **End-state check** acts as the oracle [Barr et al., 2015], distinguishing harmless edits (equivalent mutants) from true gaps

### Adversary Model

The adversary knows the task, controls the run, acts only through the benchmark's normal actions from a clean start, and cannot modify the verifier, the evidence the verifier reads, or the environment after the run starts. This excludes reward tampering [Everitt et al., 2021].

---

## Methodology

### Three Kinds of Tests

| Kind of test | AppWorld (API calls) | WorkArena (browser) | A pass means |
|---|---|---|---|
| **Missing or wrong work** | Claim success and write nothing; drop the last write; keep only the first of a repeated write; change one written value | Claim success and write nothing; leave the form unsubmitted; leave out a task field | A false positive, unless the record is still correct |
| **Unrequested change** | Repeat the last write | Also fill a field the task never mentions | A false positive, unless the change alters nothing |
| **Wrong task** | Grade one task's correct run with another task's verifier | Same | An acceptance by the wrong task's verifier |

### The Five-Step Audit Pipeline

After a person writes the edits and a harness for each benchmark, five steps run with **no per-run human decision and no model call**:

1. **Listing the edited runs**
2. **Replaying them** from a clean start (reset database in AppWorld; fresh ServiceNow episode in WorkArena)
3. **End-state check**: Reads what the run changed *before* the verifier is called, using queries separate from the verifier's code
4. **Verifier calls**: The unmodified verifier grades the run at a fixed version
5. **Classification**: A fixed rule assigns each run an outcome

### Outcome Classification

- **Rejected**: Verifier fails the run
- **Harmless edit**: Verifier passes and end state is correct
- **False positive**: Verifier passes and end-state check shows the wrong change the edit names

### AppWorld End-State Check

The end-state check counts writes to each table and compares counts with the correct run's. This is a **lower bound**: the count errs in one direction (an edit nobody wrote finds nothing; a check comparing no values can miss a wrong one).

### Applicability Conditions

The method applies only to benchmarks that provide:
1. Reset-able start states shareable between correct and edited runs
2. A correct run the verifier passes
3. Programmatic access to agent actions
4. A deterministic verifier callable unmodified and re-gradeable as a patched copy
5. An end-state check that sees each edit's change
6. A way to pin what the verifier expects

---

## Empirical Validation / Results

### Setup

- **AppWorld**: 48 tasks (16 templates × 3 versions), 8 templates in official train split, 8 in dev split
- **WorkArena**: 13 single-step tasks (4 forms, 9 catalog items) × 3 configurations

### AppWorld: Repeated Writes

The unrequested-change edit repeats the last write of the correct run:

- **39 of 48 runs passed**; 33 were harmless (repeat changed nothing)
- **6 false positives** across all 3 versions of 2 templates:
  - **Work-dinner payment task**: Repeated last request leaves an extra request and notification
  - **Song-queue task**: Same song queued a second time

**Why the tests miss it**: Payment tests match new requests to expected shares by receiver (so a duplicate merges); song-queue tests compare the queue as a set (cannot see a duplicate).

### AppWorld: The Count Assertion Repair

| Template | Added assertion | Result |
|---|---|---|
| Work-dinner payment | Number of new payment requests = number of coworkers who still owe | All 6 duplicates fail; all 6 correct runs pass |
| Song queue | Queue must grow by exactly the number of songs added | All 6 duplicates fail; all 6 correct runs pass |

The repair also rejected none of 13 natural agent runs the original tests accepted.

### WorkArena: Form Verifier After Submit

- **36 complete and valid pairs** (12 candidate fields × 3 seeds, in 2 form templates: incident and change request)
- **Every pair**: Edited run saved the requested value in the extra field, all task fields matched, and the verifier passed both runs
- **48 unresolved pairs**: 27 hardware-asset pairs failed the configuration-fixed check; 21 stopped at hidden/read-only/disabled inputs

The verifier's monitor (a page script) is meant to flag unrequested field changes, but **the flag was unset when the verifier read it** in all 147 episodes that reached the verifier.

### WorkArena: Per-Step Validation

| Extra field | Filled | Runs | Outcome under per-step validation |
|---|---|---|---|
| Any kind | First | 11 | Reward 0 before submit |
| Drop-down | Last | 4 | Reward 0 before submit |
| Text or reference | Last | 7 | Reward 1 at submit, as a correct run |
| None (correct run) | — | 11 | Reward 1 at submit |

The probe explains why: a drop-down set the flag before submit (so per-step validation catches it), but text/reference fields set the flag only during the submit click, which the page load after submit loses.

### Aggregate Results

| Kind of test | AppWorld Graded | AppWorld Rejected | AppWorld Harmless | AppWorld False pos. | WorkArena Graded | WorkArena Rejected | WorkArena Harmless | WorkArena False pos. |
|---|---|---|---|---|---|---|---|---|
| Missing or wrong work | 168 | 168 | 0 | 0 | 93 | 84 | 9 | 0 |
| Unrequested change | 48 | 9 | 33 | 6 | 36† | 0 | 0 | 36 |
| Wrong task | 2,263 | 2,263 | n/a | n/a | 426 | 426 | n/a | n/a |

† The 36 complete and valid pairs of the paired experiment (verifier called once after submit).

### Reward Implications

AppWorld's work-dinner payment template is in the **official train split** (difficulty 3); the song-queue template is in the **dev split** (difficulty 1).

| Task template | Official split | Trains on it | Selects checkpoints on its split | Duplicate vs. correct run |
|---|---|---|---|---|
| Work-dinner payment | Train, difficulty 3 | CANOPY, Entropy-Preserving RL, ProCredit | None; selection uses dev | Both pass 4 of 4 tests; duplicate adds one payment request and one notification |
| Song queue | Dev, difficulty 1 | None | LOOP, GVPO, SAGE, Entropy-Preserving RL | Both pass 6 of 6 tests; duplicate adds one song-queue write |

Under all three reward formulas (CANOPY: 1 when all tests pass; Entropy-Preserving RL: fraction passing; ProCredit: all-tests reward + progress term $c \cdot \Delta(\text{fraction of initially failing tests that pass})$), the duplicate earns the correct run's maximum reward for any $c > 0$.

---

## Theoretical and Practical Implications

### Theoretical Significance

1. **Systematic loopholes are detectable**: The paper demonstrates that verifier gaps can be found without model-generated adversaries, adaptive search, or natural failure corpora—hand-written edits suffice.

2. **The frame condition is the weak point**: Every false positive came from an unrequested change; none from missing or wrong work. Both benchmarks include checks meant for unrequested changes, yet these runs got past both.

3. **Reward hacking can be latent**: The duplicate-write loophole is systematic (holds across a task template), which prior work [Egashira et al., 2026] shows can stall or collapse RL training. Yet none of AppWorld's 1,277 released passing runs exploits it—the loophole is *latent*.

4. **Score- and reward-based audits are insufficient**: Audits judging exploits by score or reward cannot separate the duplicate from the correct run, since their scores and rewards are equal. Only an end-state check against a correct run reveals the difference.

### Practical Implications

- **Simple repairs work**: One count assertion per template rejected all duplicates and failed no correct runs—a minimal, practical fix.
- **Verifier call frequency matters**: WorkArena's per-step validation catches some extra fields (drop-downs, text/reference filled first) but not others (text/reference filled last).
- **Authors' responsibility**: Before a verifier becomes a reward, authors can count what a run adds or compare the whole end state with the expected one.

### Relationship to Prior Work

The paper extends Sun [2026]'s state-mutation approach: while Sun's snapshot method yielded no applicable collateral mutation in AppWorld, this paper's edits run through AppWorld's own API, so the duplicate request and its notification reach the database without being built by hand. It also connects to work on outcome validity [Zhu et al., 2025], verifier false positives in RLVR [Zhang, 2026], and the finding that RLVR still learns from a noisy verifier as long as its true-positive rate exceeds its false-positive rate [Rad et al., 2026].

---

## Conclusion

### Main Takeaways

1. **Mutation testing audits verifiers**: A benchmark verifier is a test suite for agent behavior, so mutation testing can check it.
2. **Side-effect loopholes exist in both benchmarks**: Runs that did the task and made one unrequested change still passed.
3. **The gap lies in the frame condition**: Missing or wrong work produced no false positives.
4. **RL rewards inherit the gap**: Three recent RL methods give the duplicate the correct run's maximum reward.
5. **Simple repairs are effective**: One count assertion per affected template rejected all duplicates and failed no correct runs.

### Open Questions

1. **Does RL optimization find the duplicate on its own?** Training policies on the original and repaired AppWorld tests would answer this.
2. **Can automated auditing replace hand-written edits?** When a language model reads a verifier and proposes candidate loopholes, the bottleneck shifts to confirming which candidates produce an unauthorized end state and repairing the verifier without rejecting valid alternative runs.

### Key Quote

> "Before a verifier becomes a reward, its authors can count what a run adds or compare the whole end state with the expected one."

---

_Markdown view of https://picx.dev/p/43lOuG, served by PicX — AI-generated visual whiteboard summaries of research papers._