Probing RLVR Training Instability through the Lens of Objective-Level Hacking

Summary (Overview)

• New theoretical framework: The paper introduces the concept of objective-level hacking to explain training instability in Reinforcement Learning with Verifiable Rewards (RLVR) for Mixture-of-Experts (MoE) large language models, distinguishing it from the well-known reward hacking phenomenon.
• Unified formulation: The authors provide a mathematical framework showing that token-level credit misalignment—whether from training-inference discrepancies or token-level clipping—induces system-level spurious signals in the optimization objective that drive pathological training dynamics.
• Causal mechanism identified: Through extensive experiments on a 30B MoE model (Qwen3-30B-A3B), the paper demonstrates that biased token-level weight distortions (not variance-based noise) causally trigger the abnormal growth of training-inference discrepancy, leading to training collapse.
• Positive feedback loop: The paper reveals a self-reinforcing cycle where training-inference discrepancy amplifies objective-level hacking, which in turn worsens the discrepancy—explaining the irreversibility of training collapse.
• MoE vs. dense comparison: The framework explains why MoE models suffer more severe instability than dense models: they exhibit both larger initial training-inference mismatch and substantially faster mismatch growth during training.

Introduction and Theoretical Foundation

Background and Motivation

Reinforcement learning with verifiable rewards (RLVR) has demonstrated remarkable success in enhancing reasoning capabilities of large language models across mathematics, code generation, and decision-making tasks. Methods like Group Relative Policy Optimization (GRPO) show advantages over supervised fine-tuning, including better generalization and robustness to catastrophic forgetting. However, prolonged RLVR training frequently exhibits instabilities, particularly in Mixture-of-Experts (MoE) architectures, manifesting as continuous performance decline accompanied by anomalies in token-level entropy and gradient norms.

Key Distinction: Reward Hacking vs. Objective-Level Hacking

The paper draws a critical analogy:

"Unlike reward hacking, which arises from exploitable verifiers, objective-level hacking emerges from token-level credit misalignment and is manifested as system-level spurious signals in the optimization objective."

While reward hacking exploits vulnerabilities in the reward verifier, objective-level hacking originates from misalignment in how credit is assigned to individual tokens during optimization, creating spurious signals that perturb the training objective itself.

Theoretical Foundation: GRPO Framework

The paper builds on the GRPO objective, where for each question $q \sim \mathcal{D}$, we sample $G$ responses $\{o_i\}_{i=1}^{G}$ from the old policy and optimize via a clipped surrogate objective:

$$\mathcal{J}(\theta)_{\mathrm{GRPO}} = \mathbb{E}_{q \sim \mathcal{D}, \{o_i\} \sim \pi_{\mathrm{train}}(\cdot | q; \theta_{\mathrm{old}})} \left[ \mathcal{L}_{\mathrm{clip}}(\theta) \right]. \tag{1}$$

The clipped loss is:

$$\mathcal{L}_{\mathrm{clip}}(\theta) = \frac{1}{G} \sum_{i=1}^{G} \frac{1}{|o_i|} \sum_{t=1}^{|o_i|} \min \Big(r_{i,t}(\theta) \hat{A}_{i,t}, \operatorname{clip}(r_{i,t}(\theta), 1-\varepsilon, 1+\varepsilon) \hat{A}_{i,t}\Big), \tag{2}$$

where $r_{i,t} \equiv \pi_{\mathrm{train}}(o_{i,t} \mid q, o_{i,<t}; \theta) / \pi_{\mathrm{train}}(o_{i,t} \mid q, o_{i,<t}; \theta_{\mathrm{old}})$ is the importance weight, and the advantage is group-normalized:

$$\hat{A}_{i,t} = \frac{R_i - \mathrm{mean}(\{R_i\}_{i=1}^{G})}{\mathrm{std}(\{R_i\}_{i=1}^{G})}. \tag{3}$$

The Training-Inference Discrepancy Problem

In practice, rollout generation and model training use different implementations, creating a mismatch between the inference distribution $\pi_{\mathrm{infer}}(\cdot | q; \theta_{\mathrm{old}})$ and training distribution $\pi_{\mathrm{train}}(\cdot | q; \theta_{\mathrm{old}})$. This makes the effective optimization off-policy:

$$\mathcal{J}'(\theta)_{\mathrm{GRPO}} = \mathbb{E}_{q \sim \mathcal{D}, \{o_i\} \sim \pi_{\mathrm{infer}}(\cdot | q; \theta_{\mathrm{old}})} \left[ \mathcal{L}_{\mathrm{clip}}(\theta) \right]. \tag{4}$$

The paper measures this discrepancy via the token-level importance weight ratio:

$$\rho_{i,t} = \frac{\pi_{\mathrm{train}}(o_{i,t} \mid q, o_{i,<t}; \theta_{\mathrm{old}})}{\pi_{\mathrm{infer}}(o_{i,t} \mid q, o_{i,<t}; \theta_{\mathrm{old}})}. \tag{5}$$

Methodology

Mathematical Framework for Objective-Level Hacking

The core theoretical contribution is deriving how different sources of instability perturb the optimization objective. Ignoring clipping, the ideal GRPO objective is:

$$\mathcal{J}(\theta) = \mathbb{E}_{\mathrm{train}} \left[ \sum_{i,t} X_{i,t}(\theta) \right], \tag{6}$$

where $X_{i,t}(\theta) \equiv \frac{r_{i,t}(\theta) \hat{A}_{i,t}}{G \cdot |o_i|}$. When training-inference discrepancy exists, the effective objective becomes $\mathcal{J}'(\theta) = \mathcal{J}(\theta) + \Delta\mathcal{J}(\theta)$, where:

$$\Delta\mathcal{J}(\theta) \simeq \sum_{i,t} \mathrm{Cov}_{\mathrm{train}}\big(X_{i,t}(\theta), \rho_{i,t}^{-1}\big). \tag{10}$$

This covariance term represents a spurious signal that drives the model toward unintended correlations between $X_{i,t}$ and $\rho_{i,t}^{-1}$.

Unified Formulation for Token-Level Modulations

The paper shows that token-level clipping similarly introduces bias. The clipped objective can be written as:

$$\mathcal{J}_{\mathrm{clip}}(\theta) = \mathcal{J}(\theta) + \Delta_{\mathrm{clip}}\mathcal{J}(\theta), \tag{13}$$ $$\Delta_{\mathrm{clip}}\mathcal{J}(\theta) = \mathbb{E}_{\mathrm{train}} \left[ \sum_{i,t} X_{i,t}(\theta) \left(\phi_{i,t} - 1\right) \right], \tag{14}$$

where $\phi_{i,t}$ is an indicator for whether token $o_{i,t}$ is clipped. This leads to the unified characterization:

$$\mathcal{J}_{\mathrm{dist}}(\theta) = \mathcal{J}(\theta) + \Delta_{\mathrm{dist}}\mathcal{J}(\theta). \tag{15}$$

Experimental Design

The experiments use Qwen3-30B-A3B (a 30B MoE model) trained on DAPO-Math-17k with AIME24 validation, implemented with the verl framework (vLLM for inference, Megatron for training). Key configurations:

• 128 problems sampled per step, 16 responses per problem
• 4 parameter update steps per sampling (off-policy setting)
• Maximum response length of 8K tokens
• Token-level clipping ranges of 0.2; sequence-level (GSPO) clipping ranges of 3e-4 and 4e-4

Four experimental settings were constructed:

1. Vanilla GRPO vs. TIS-corrected variant (to isolate initial discrepancy effects)
2. Varying token-level clipping strengths (right ranges: 0.2, 0.24, 0.28)
3. Sequence-level clipping (GSPO) as stable baseline
4. Active injection of token-level weight distortion (controlled experiments)

Empirical Validation / Results

Initial Training-Inference Discrepancy Effects

Truncated Importance Sampling (TIS) markedly reduces the growth rate of training-inference discrepancy while slowing token entropy decline and improving validation scores. However, the discrepancy is not completely eliminated, indicating multiple contributing sources.

Scatter analysis of $(\pi_{\mathrm{train}}, \pi_{\mathrm{infer}})$ shows:

• Early training: high Pearson correlation coefficient (PCC)
• Later stages: pronounced discrepancy with abnormal clustering along the $\pi_{\mathrm{infer}}$ dimension, attributed to limited numerical precision in inference mode

Token-Level Clipping Effects

A counterintuitive finding: stronger token-level clipping actually accelerates training-inference discrepancy growth, despite being designed to stabilize training.

Table 1: Variation of clip ratio during RLVR training

In contrast, sequence-level clipping does not exhibit the same anomalous discrepancy growth, providing indirect evidence for the role of token-level clipping.

Causal Evidence from Injected Distortions

The paper designs controlled experiments by injecting token-level weight distortion into the GSPO objective:

$$\mathcal{J}_{\mathrm{inj}} = \mathbb{E}_{\mathrm{train}} \left[ \sum_{i,t} \varphi_{i,t} \times Y_{i,t}(\theta) \right], \tag{17}$$ $$\varphi_{i,t} = \begin{cases} \delta, & \text{if } o_{i,t} \in S_{\mathrm{low-prob}}, \\ 1, & \text{otherwise}. \end{cases} \tag{18}$$

Key results:

• Even a modest 20% weight increase ($\delta = 1.2$) for low-probability tokens triggers discrepancy growth
• Stronger distortions induce faster discrepancy growth (statistically correlated)
• Both increasing and decreasing low-probability token weights reproduce the anomaly, confirming it's the distortion itself that matters
• Model performance shows irreversible degradation alongside token entropy anomalies

Bias vs. Variance

Critical controlled experiment: injecting variance-based noise ($\xi_{i,t} \sim \mathcal{N}(1, \sigma^2)$) does not trigger abnormal discrepancy growth:

$$\Delta\mathcal{J}_{\mathrm{var}}(\theta) = \mathbb{E}_{\mathrm{train}} \left[ \sum_{i,t} Y_{i,t}(\theta) (\xi_{i,t} - 1) \right] \simeq 0. \tag{22}$$

This confirms that biased distortion is the primary driver, as biased objective-level spurious signals can be persistently exploited by optimization.

Positive Feedback Loop

The paper documents a self-reinforcing mechanism:

1. Low-probability tokens exhibit wider dispersion from $\rho_{i,t} = 1$ (greater mismatch)
2. Importance weights $\rho_{i,t}$ of low-probability tokens consistently decrease as training progresses (survival bias effect)
3. This divergence exacerbates objective-level hacking, driving further discrepancy growth
4. The loop explains irreversibility of training collapse

MoE vs. Dense Models

Using DeepSeek-R1-Distill-Qwen-7B as a dense baseline, the paper finds:

MoE models exhibit both larger initial mismatch and substantially faster mismatch growth, likely due to expert activation inconsistency between training and inference.

Theoretical and Practical Implications

Theoretical Contributions

1. New analytical lens: The paper introduces objective-level hacking as a distinct failure mode from reward hacking, shifting focus from verifier vulnerabilities to optimization objective distortions.

2. Unified mathematical framework: Equations (10), (14), and (20) provide a unified formulation showing that diverse instability sources (infrastructure mismatch, clipping strategies, weight distortions) all manifest as covariance terms or biased perturbations in the optimization objective.

3. Causal understanding: The controlled injection experiments establish causality (not mere correlation) between token-level weight distortion and training-inference discrepancy growth.

Practical Implications

1. Algorithm design guidance: The findings suggest that RLVR algorithms should avoid token-level modulations that introduce biased perturbations, favoring sequence-level approaches (e.g., GSPO) for MoE training stability.

2. Stabilization strategy: Since biased (not variance-based) distortions are the primary driver, correction methods like TIS that address objective bias are more principled than infrastructure-level fixes alone.

3. MoE-specific considerations: The framework explains why MoE models are particularly vulnerable, suggesting that expert activation consistency should be a focus for stable training.

4. Monitoring tool: The surrogate metric $J \equiv \sum_{i,t} \hat{A}_{i,t} \times (\varphi_{i,t} - 1)$ (Eq. 23) provides a practical way to monitor objective-level hacking during training.

Conclusion

This paper provides a principled framework for understanding RLVR training instability in MoE models through the lens of objective-level hacking. The key insight is that token-level credit misalignment—whether from infrastructure discrepancies, clipping strategies, or other modulations—introduces system-level spurious signals in the optimization objective that drive pathological training dynamics.

The authors trace the mechanism behind the abnormal growth of training-inference discrepancy, demonstrating through controlled experiments that biased token-level weight distortions (not variance noise) causally trigger this growth. The resulting positive feedback loop between discrepancy and hacking explains the irreversibility of training collapse and the heightened vulnerability of MoE architectures.

Future Directions

The authors acknowledge that RLVR training is a complex multi-system interaction, and their framework does not capture the full picture of model collapse. Future work should explore:

• Additional sources of training instability (data quality, rollout engine choices, hardware configurations)
• Interaction effects between different instability mechanisms
• Algorithm designs that specifically protect the sensitive inference behavior of MoE models
• Extensions to other architectures and training paradigms

The framework offers a valuable new perspective for improving the interpretability and controllability of RLVR training, with direct implications for designing more robust reinforcement learning algorithms for MoE models.