Summary (Overview)

• RRI (Recursive Reward Improvement) is a self-evolving pipeline that improves its own reward functions alongside its tasks and solver, using reward hacking as adversarial supervision—one of the first systems to do so for open-ended expert tasks without training data.
• Key innovation: Before the solver trains on a generated task, an adversary writes an answer that games the rubric while being as useless as possible; the same model writes an honest answer without seeing the rubric; up to two repair steps revise the rubric to make the honest answer win.
• Results: RRI improves adjusted scores on HealthBench Professional over the Generation Only baseline by 10.9 points at 9B and 12.4 points at 27B, and by 10.3 points in a fully self-contained setting with no external model.
• Out-of-domain generalization: RRI-27B trained on HealthBench Professional description improves by 6.6 points on PRBench Hard and 8.0 on ProfBench, despite never seeing tasks from either benchmark.
• Analysis: The authors classify 12,147 exploits into five categories of reward failure plus one category of incorrect fixes, identifying that nearly 99% of model-generated question-rubric pairs are hackable by naive generation.

---

Introduction and Theoretical Foundation

Background and Motivation

Recursive Self-Improvement (RSI) has gained attention for its ability to work with limited data. In a proposer–solver loop, a proposer model writes training tasks and a solver model learns to solve them through reinforcement learning. Such loops have succeeded in mathematics and coding, where answers can be checked automatically. However, open-ended expert tasks (e.g., clinical report generation, legal analysis) have no answer key, making simple question-answer generation infeasible.

The Core Challenge

For open-ended tasks, the proposer must also write a rubric—a weighted list of content an answer should include and errors it should avoid. The unique challenge is comprehensiveness:

"For naively generated rubric, solvers tend to exploit it, naming the expected topics without the substance behind them, which corrupts the entire learning process."

Key Theoretical Foundations

1. Reward hacking (Amodei et al., 2016; Skalse et al., 2022): Solvers exploit ambiguous or incomplete reward functions.
2. Self-play loops (Absolute Zero, R-Zero, SPICE): These write tasks and grade by execution or answer key, which open-ended tasks lack.
3. Modern defenses against reward hacking correct biases that hold across tasks (Chen et al., 2024a; Park et al., 2024), but no longer apply when the rubric varies across questions.

Problem Statement

The core unsolved problem: How to supervise model-written rewards without a human reference?

---

Methodology

RRI consists of five components:

3.1 Proposer

• Receives only a general description of the target task and no training data
• Writes diverse clinician requests that double as retrieval queries
• Retrieves medical passages from a reference database
• Writes the completed task and rubric in one call from those references
• Each task consists of a request $x$ and a rubric $R = \{(c_j, p_j)\}_{j=1}^{m}$, where $c_j$ is a criterion and $p_j \in [-10, 10] \setminus \{0\}$ is its signed weight

3.2 Adversary Rubric Hacking & Repair

Hack stage: The adversary receives each new task and rubric and writes a reward hacking answer—one that satisfies as many criteria as possible while being as useless as possible. In parallel, the same frozen model writes an honest answer from the task alone, without seeing the rubric.

The lead of the exploit is computed as:

$$\Delta = s(y_{exploit}) - s(y_{honest})$$

The adversary wins unless the honest answer outscores the exploit by at least a margin $\delta = 0.20$.

Patch stage: When the adversary wins, the model rewrites the question–rubric pairs. The judge evaluates each proposed criterion against both answers, and it is retained only when:

$$\text{negative criterion met}(A) \land \neg \text{met}(B), \quad \text{positive criterion met}(B) \land \neg \text{met}(A)$$

In other words, a criterion is kept only when it gives the honest answer a higher score than the hacking answer. Approximately four fifths of accepted criteria are positive.

Admission: After at most two repair rounds, every task enters the training pool. A rewrite is adopted only when a fresh probe shows a smaller $\Delta$ than before.

3.3 Solver

• Generates a group of $n$ responses per task
• Trained with Group Relative Policy Optimization (GRPO):

$$\hat{A}_i = s(y_i) - \frac{1}{n}\sum_{k} s(y_k)$$

• Can make up to two tool calls per rollout for citation purposes (clinical database or web search)

3.4 Meta-Optimizer

• Updates the proposer input at the end of each step
• Classifies failure modes, rubric defects, and task defects
• Assigns each case one of three verdicts: model failed a fair task, generator at fault, or model succeeded
• Compares corpus statistics (criteria per task, negative share, use-case mix, task self-similarity) with benchmark reference
• Rewrites guidance blocks appended to the two generation prompts

---

Empirical Validation / Results

Experimental Setup

Scoring: Each benchmark scores an answer $y$ against its rubric. The rubric score is:

$$r(y) = \frac{\sum_{j \in M(y)} p_j}{\sum_{j: p_j > 0} p_j}$$

HealthBench Professional adds its official length adjustment:

$$s(y) = r(y) - \lambda (\ell(y) - \ell_0)/500$$

where $\ell(y)$ is answer length in characters, $\ell_0 = 2000$, and $\lambda = 0.0147$.

RQ1: In-Domain Improvement (HealthBench Professional)

Key findings:

• Writing tasks benefit most: +39.1 points at 9B, +14.7 at 27B, +25.9 in all-self setting
• Difficult tasks improve by 18.6 points vs. 2.8 on typical tasks (9B setting)
• Adding the adversary to a loop that already co-evolves tasks and rewards adds +2.7 points (Co-evolve Only: 0.506 → Full RRI: 0.533)

Comparison with existing rubric-reward methods (9B GPT Teacher):

RQ2: Out-of-Domain Generalization

RQ3: Hack–Patch–Improve Loop Analysis

The full analysis covered 12,147 samples across three datasets (7,032 for HealthBench Pro, 3,473 for PRBench, 1,642 for ProfBench). Key statistics:

• Nearly 99% of model-generated question-rubric pairs classified as hackable
• One third of exploits carry wrong content that no criterion penalizes
• A further third turn on how a loosely worded criterion is read

Example of Rubric Repair (Atrial Fibrillation Task)

---

Theoretical and Practical Implications

Theoretical Contributions

1. Reward verification without references: RRI demonstrates that adversarial self-checking can validate model-written rewards without human annotations, preference labels, or golden rubrics—a fundamental advance for open-ended self-improvement.

2. Reward hacking as supervision signal: Rather than treating reward hacking as a failure to be eliminated, RRI uses exploits as constructive feedback to improve the reward function itself.

3. Meta-level evolution: The meta-optimizer makes improvements persistent by rewriting generation prompts, preventing the model from repeating the same reward-design errors.

Practical Implications

1. Medical/legal/financial AI: Enables training on scarce expert tasks without expert-written training data, potentially democratizing specialized AI development.

2. Scalable self-improvement: The all-self setting (frozen 9B model serving every auxiliary role) demonstrates that self-improvement works without a stronger teacher model.

3. Transferable skills: Training on medical tasks improves performance on finance, legal, chemistry, physics, and consulting tasks—suggesting the repair process teaches generalizable professional reasoning skills.

---

Conclusion

Main Takeaways

1. RRI is one of the first self-evolving pipelines that improves its reward alongside its tasks and solver, with reward hacking as adversarial supervision, requiring zero training data.

2. Consistent improvements across settings: +10.9 to +12.4 points on HealthBench Professional over Generation Only baselines, with gains largest on difficult and writing tasks.

3. Out-of-domain transfer: Improvements generalize to completely different professional domains (finance, law, chemistry, physics, consulting), with RRI adding 5.3 points on PRBench Hard and 3.3 points on ProfBench over Generation Only.

4. The adversary is essential: Adding hack-then-patch rounds to a loop that already co-evolves tasks and rewards adds 2.7 points, showing that active reward testing provides unique value.

Future Directions

• Scaling to more domains and larger models: The authors note the 27B model benefits more from RRI than the 9B model (+14.2 vs. +10.4 points over untrained), suggesting larger models may gain more from repaired rewards.
• Reducing repair rounds: Current design uses at most two repair rounds as a compute/benefit trade-off; more sophisticated admission criteria could improve efficiency.
• Addressing remaining exploit categories: The analysis identifies specific failure modes (wrong content not penalized, loosely worded criteria) that future reward-generation methods should target.

The authors release all code, prompts, and configurations at https://github.com/DDVD233/RRI