Summary of "From RLVR to RLSVR: Task Transformation Induces Self-Verifiable Rewards for Open-Ended LLM Self-Improvement"

Summary (Overview)

• New paradigm (RLSVR): Extends RLVR to open-ended tasks by transforming the task into a proxy environment where rewards become verifiable by construction, inspired by self-supervised learning’s pretext-task principle.
• SpyRL instantiation: An information-asymmetric self-play game based on “Who Is the Spy?” where a predetermined spy identity turns output-quality assessment into a verifiable identity-recognition problem.
• State-of-the-art results: SpyRL outperforms existing self-improvement methods (R-Zero, Absolute Zero) on non-verifiable tasks (summarization, creative writing) and yields consistent gains on verifiable mathematical reasoning, with vote-based rewards closely aligned with human and LLM quality judgments.
• Verifiability is engineered: The work demonstrates that verifiability need not be an intrinsic property of a task—it can be created through task transformation, opening a path toward scalable, verifier-free self-improvement for general open-ended capabilities.

Introduction and Theoretical Foundation

Reinforcement Learning with Verifiable Rewards (RLVR) has driven progress in reasoning models (e.g., OpenAI o1, DeepSeek-R1) by enabling large-scale optimization with exact, rule-based reward signals. However, RLVR is limited to domains where correctness can be deterministically verified (e.g., math, coding). Open-ended tasks (creative writing, summarization) rely on human preferences, learned reward models, or LLM-as-a-Judge, introducing evaluation bias, judge capability bottlenecks, and additional inference costs.

The authors draw inspiration from self-supervised learning (SSL), which constructs pretext tasks whose supervisory signals are derived automatically from the data itself (e.g., masked language modeling, contrastive learning). SSL shows that when a proxy objective generates supervision automatically while preserving substantial capability overlap with the target task, learning can proceed without task-specific human annotation.

RLSVR extends this task-transformation principle to RLVR. Instead of approximating an unverifiable quality function $Q(x,y)$ with an external evaluator, RLSVR transforms the original open-ended task into a proxy environment $\mathcal{E}$ that:

1. Injects a latent variable $z$ (known only to the environment).
2. Conditions task execution on observations derived from $(x, z)$.
3. Defines a verifiable interaction (e.g., identity inference) whose outcome can be checked exactly against $z$.
4. Computes a rule-based reward $R$ from the interaction outcome.

The reward is called self-verifiable because it is deterministic, rule-based, and requires no human annotation, learned reward model, or external judge. RLSVR is thus "self-supervised learning for RLVR": the transformation $\Phi$ corresponds to the pretext task, $z$ to the automatically generated label, and standard RLVR optimization (e.g., GRPO) applies directly to $\mathcal{E}$.

Methodology

SpyRL is a concrete instantiation of RLSVR using information-asymmetric self-play. The framework consists of two stages per training epoch (see Algorithm 1):

1. Information-Asymmetric Performing Stage

• Sample an input $x \sim \mathcal{D}$ and a spy index $u \sim \text{Unif}(\{1,\dots,n\})$.
• Assign observations: civilians receive full $x$; the spy receives a degraded version $g(x)$ (e.g., continuous span masking of 20%–40% of the text).
• Each player $i$ generates an output $y_i \sim \pi^P_\theta(\cdot | o_i, \tau)$ for the target task (summarization, creative writing, or math problem construction/solution).

2. Detection Stage with Verifiable Rewards

• All outputs $Y = \{y_1,\dots,y_n\}$ are revealed.
• Each player votes on the spy identity: $v_i \sim \pi^D_\phi(\cdot | (o_i, Y))$.
• Detection reward: $r^D_i = \mathbb{I}[v_i = u]$ (deterministically verifiable).
• Group-based advantage (GRPO-style): $A^D_i = (r^D_i - \mu^D)/\sigma^D$, normalized within the group.

3. Two-Stage Coupled Optimization

• Performing reward (zero-sum between spy and civilians): $$r^P_u = -\beta(m_u - \bar{m}_c), \quad r^P_{c_j} = \frac{\beta}{n_c}(m_u - \bar{m}_c) - \lambda(m_{c_j} - \bar{m}_c)$$ where $m_u$ is votes received by the spy, $m_{c_j}$ by civilian $j$, $\bar{m}_c$ the average civilian votes, and $\beta, \lambda$ control competition strength.
• Role-Advantage Estimation (RAE): Calibrates role biases by subtracting role-specific baselines, preventing the optimizer from conflating the spy’s information disadvantage with poor policy quality.
• Alternating optimization: The performer and detector are updated alternately to avoid policy stagnation and maintain stable learning pressure.
• KL-regularized GRPO objectives: $$\mathcal{L}^P(\theta) = -\mathbb{E}\left[\frac{1}{n}\sum_{k\in\{u\}\cup C}\sum_t \min\left(\rho^P_{k,t}A^P_k, \text{clip}(\rho^P_{k,t},1-\epsilon,1+\epsilon)A^P_k\right)\right] + \beta_P \text{KL}(\pi^P_\theta \| \pi^P_{\text{ref}})$$ $$\mathcal{L}^D(\phi) = -\mathbb{E}\left[\frac{1}{n}\sum_{i=1}^n \min\left(\rho^D_i A^D_i, \text{clip}(\rho^D_i,1-\epsilon,1+\epsilon)A^D_i\right)\right] + \beta_D \text{KL}(\pi^D_\phi \| \pi^D_{\text{ref}})$$

The degradation operator $g(\cdot)$ is domain-specific: continuous span masking (20% for summarization/writing, 40% for math reasoning). All experiments use group size $n=5$ (default), batch size 1024, 100 epochs, and Qwen3-4B/8B backbones.

Empirical Validation / Results

Main Results on Non-Verifiable and Verifiable Tasks

Table 1: Summarization benchmarks (ROUGE-L and GPT-4o A/B win rates of SpyRL vs. each baseline)

Table 2: Creative writing (GPT-4o pairwise win rates; SpyRL vs. each baseline)

Table 3: Mathematical reasoning (accuracy %)

Additional Validations

• Human evaluation (Table 4): SpyRL achieves 80.0% overall win rate vs. Qwen3-4B on WritingPrompt, consistently higher across novelty, emotion, coherence, and consistency.
• Rubric-as-reward comparison (Table 5): SpyRL outperforms Qwen3.5-27B-RaR (59.3% overall win) and is competitive with GPT-4o-RaR while incurring zero external verifier cost.
• Domain-specific summarization (Table 6): Trained on PubMed, SpyRL raises ROUGE-L on arXiv, PubMed, and BillSum by an average of 4.9 points.
• Cross-task transfer (Table 7): Summarization and creative writing transfer positively in both directions; math reasoning does not transfer to writing tasks.
• Ablation (Table 8): Full two-stage optimization is crucial; removing it leads to rapid plateauing.
• Group size (Figure 5): $n=5$ provides the best trade-off (mean gain from 5.5 to 9.3 over base); further scaling shows diminishing returns.
• Role-Advantage Estimation (Table 9): Removing RAE degrades the average from 50.4 to 37.5, actively harming the model.
• Degradation operator sensitivity (Table 10): 20% vs. 40% masking ratio yields nearly indistinguishable results, showing robustness.

Figure 4 (vote-quality correlation): A positive correlation between the number of suspicion votes received and GPT-4o rank (lower quality → more votes) confirms that the vote-based reward aligns with actual task performance without external verifiers.

Theoretical and Practical Implications

• Theoretical: RLSVR unifies two previously separate approaches—RLVR (exact verification) and SSL (label-by-construction)—showing that verifiability can be engineered through task transformation rather than being an intrinsic property of a task. The latent variable $z$ plays the role of automatically generated labels, and the proxy environment $\mathcal{E}$ provides a rule-based reward that is exact and unbiased.
• Practical: SpyRL provides a scalable, verifier-free self-improvement pipeline for open-ended tasks. It eliminates the need for expensive human annotations, learned reward models, or LLM judges, reducing both cost and evaluation bias. The method is computationally efficient (no external verifier calls) and generalizes across domains (summarization, creative writing, reasoning). The cross-task transfer results suggest that training on one open-ended task can improve performance on related tasks.
• Limitations: The degradation operator $g(\cdot)$ must be designed per task, though experiments show it is robust to masking ratio. The framework currently requires multiple agents (default $n=5$), which increases inference cost per training step. The spy mechanism relies on the assumption that information deficit leads to detectably lower quality; tasks where this is not true (e.g., tasks where the spy can compensate) may require alternative transformations.

Conclusion

The paper proposes RLSVR, a paradigm that extends RLVR to open-ended tasks by transforming them into proxy environments with self-verifiable rewards. The instantiation SpyRL uses information-asymmetric self-play to turn output-quality assessment into a verifiable identity-recognition problem. Extensive experiments on summarization, creative writing, and mathematical reasoning show that SpyRL outperforms existing self-improvement methods (R-Zero, Absolute Zero) and even yields gains on already-verifiable reasoning tasks. The vote-based rewards align closely with human and LLM quality judgments. The key takeaway is that verifiability can be engineered through task transformation, opening a path toward scalable, verifier-free self-improvement for general open-ended capabilities. Models and code are released at https://github.com/wangqinsi1/RLSVR/tree/SpyRL.