# Probing RLVR Training Instability through the Lens of Objective-Level Hacking

> Objective-level hacking, driven by biased token-level weight distortions rather than variance, causally triggers training-inference discrepancy growth and irreversible collapse in MoE RLVR training.

- **Source:** [arXiv](https://arxiv.org/abs/2602.01103)
- **Published:** 2026-08-17
- **Permalink:** https://picx.dev/p/dbV39w
- **Whiteboard:** https://picx.dev/p/dbV39w/image

## Summary

# Probing RLVR Training Instability through the Lens of Objective-Level Hacking

## Summary (Overview)

- **New theoretical framework**: The paper introduces the concept of **objective-level hacking** to explain training instability in Reinforcement Learning with Verifiable Rewards (RLVR) for Mixture-of-Experts (MoE) large language models, distinguishing it from the well-known reward hacking phenomenon.
- **Unified formulation**: The authors provide a mathematical framework showing that token-level credit misalignment—whether from training-inference discrepancies or token-level clipping—induces system-level spurious signals in the optimization objective that drive pathological training dynamics.
- **Causal mechanism identified**: Through extensive experiments on a 30B MoE model (Qwen3-30B-A3B), the paper demonstrates that biased token-level weight distortions (not variance-based noise) causally trigger the abnormal growth of training-inference discrepancy, leading to training collapse.
- **Positive feedback loop**: The paper reveals a self-reinforcing cycle where training-inference discrepancy amplifies objective-level hacking, which in turn worsens the discrepancy—explaining the irreversibility of training collapse.
- **MoE vs. dense comparison**: The framework explains why MoE models suffer more severe instability than dense models: they exhibit both larger initial training-inference mismatch and substantially faster mismatch growth during training.

## Introduction and Theoretical Foundation

### Background and Motivation

Reinforcement learning with verifiable rewards (RLVR) has demonstrated remarkable success in enhancing reasoning capabilities of large language models across mathematics, code generation, and decision-making tasks. Methods like Group Relative Policy Optimization (GRPO) show advantages over supervised fine-tuning, including better generalization and robustness to catastrophic forgetting. However, **prolonged RLVR training frequently exhibits instabilities**, particularly in Mixture-of-Experts (MoE) architectures, manifesting as continuous performance decline accompanied by anomalies in token-level entropy and gradient norms.

### Key Distinction: Reward Hacking vs. Objective-Level Hacking

The paper draws a critical analogy:

> "Unlike reward hacking, which arises from exploitable verifiers, objective-level hacking emerges from token-level credit misalignment and is manifested as system-level spurious signals in the optimization objective."

While reward hacking exploits vulnerabilities in the reward verifier, objective-level hacking originates from misalignment in how credit is assigned to individual tokens during optimization, creating spurious signals that perturb the training objective itself.

### Theoretical Foundation: GRPO Framework

The paper builds on the GRPO objective, where for each question $q \sim \mathcal{D}$, we sample $G$ responses $\{o_i\}_{i=1}^{G}$ from the old policy and optimize via a clipped surrogate objective:

$$
\mathcal{J}(\theta)_{\mathrm{GRPO}} = \mathbb{E}_{q \sim \mathcal{D}, \{o_i\} \sim \pi_{\mathrm{train}}(\cdot | q; \theta_{\mathrm{old}})} \left[ \mathcal{L}_{\mathrm{clip}}(\theta) \right]. \tag{1}
$$

The clipped loss is:

$$
\mathcal{L}_{\mathrm{clip}}(\theta) = \frac{1}{G} \sum_{i=1}^{G} \frac{1}{|o_i|} \sum_{t=1}^{|o_i|} \min \Big(r_{i,t}(\theta) \hat{A}_{i,t}, \operatorname{clip}(r_{i,t}(\theta), 1-\varepsilon, 1+\varepsilon) \hat{A}_{i,t}\Big), \tag{2}
$$

where $r_{i,t} \equiv \pi_{\mathrm{train}}(o_{i,t} \mid q, o_{i,<t}; \theta) / \pi_{\mathrm{train}}(o_{i,t} \mid q, o_{i,<t}; \theta_{\mathrm{old}})$ is the importance weight, and the advantage is group-normalized:

$$
\hat{A}_{i,t} = \frac{R_i - \mathrm{mean}(\{R_i\}_{i=1}^{G})}{\mathrm{std}(\{R_i\}_{i=1}^{G})}. \tag{3}
$$

### The Training-Inference Discrepancy Problem

In practice, rollout generation and model training use different implementations, creating a mismatch between the inference distribution $\pi_{\mathrm{infer}}(\cdot | q; \theta_{\mathrm{old}})$ and training distribution $\pi_{\mathrm{train}}(\cdot | q; \theta_{\mathrm{old}})$. This makes the effective optimization off-policy:

$$
\mathcal{J}'(\theta)_{\mathrm{GRPO}} = \mathbb{E}_{q \sim \mathcal{D}, \{o_i\} \sim \pi_{\mathrm{infer}}(\cdot | q; \theta_{\mathrm{old}})} \left[ \mathcal{L}_{\mathrm{clip}}(\theta) \right]. \tag{4}
$$

The paper measures this discrepancy via the token-level importance weight ratio:

$$
\rho_{i,t} = \frac{\pi_{\mathrm{train}}(o_{i,t} \mid q, o_{i,<t}; \theta_{\mathrm{old}})}{\pi_{\mathrm{infer}}(o_{i,t} \mid q, o_{i,<t}; \theta_{\mathrm{old}})}. \tag{5}
$$

## Methodology

### Mathematical Framework for Objective-Level Hacking

The core theoretical contribution is deriving how different sources of instability perturb the optimization objective. Ignoring clipping, the ideal GRPO objective is:

$$
\mathcal{J}(\theta) = \mathbb{E}_{\mathrm{train}} \left[ \sum_{i,t} X_{i,t}(\theta) \right], \tag{6}
$$

where $X_{i,t}(\theta) \equiv \frac{r_{i,t}(\theta) \hat{A}_{i,t}}{G \cdot |o_i|}$. When training-inference discrepancy exists, the effective objective becomes $\mathcal{J}'(\theta) = \mathcal{J}(\theta) + \Delta\mathcal{J}(\theta)$, where:

$$
\Delta\mathcal{J}(\theta) \simeq \sum_{i,t} \mathrm{Cov}_{\mathrm{train}}\big(X_{i,t}(\theta), \rho_{i,t}^{-1}\big). \tag{10}
$$

This covariance term represents a **spurious signal** that drives the model toward unintended correlations between $X_{i,t}$ and $\rho_{i,t}^{-1}$.

### Unified Formulation for Token-Level Modulations

The paper shows that token-level clipping similarly introduces bias. The clipped objective can be written as:

$$
\mathcal{J}_{\mathrm{clip}}(\theta) = \mathcal{J}(\theta) + \Delta_{\mathrm{clip}}\mathcal{J}(\theta), \tag{13}
$$

$$
\Delta_{\mathrm{clip}}\mathcal{J}(\theta) = \mathbb{E}_{\mathrm{train}} \left[ \sum_{i,t} X_{i,t}(\theta) \left(\phi_{i,t} - 1\right) \right], \tag{14}
$$

where $\phi_{i,t}$ is an indicator for whether token $o_{i,t}$ is clipped. This leads to the unified characterization:

$$
\mathcal{J}_{\mathrm{dist}}(\theta) = \mathcal{J}(\theta) + \Delta_{\mathrm{dist}}\mathcal{J}(\theta). \tag{15}
$$

### Experimental Design

The experiments use **Qwen3-30B-A3B** (a 30B MoE model) trained on **DAPO-Math-17k** with **AIME24** validation, implemented with the verl framework (vLLM for inference, Megatron for training). Key configurations:
- 128 problems sampled per step, 16 responses per problem
- 4 parameter update steps per sampling (off-policy setting)
- Maximum response length of 8K tokens
- Token-level clipping ranges of 0.2; sequence-level (GSPO) clipping ranges of 3e-4 and 4e-4

Four experimental settings were constructed:
1. **Vanilla GRPO** vs. **TIS-corrected variant** (to isolate initial discrepancy effects)
2. **Varying token-level clipping strengths** (right ranges: 0.2, 0.24, 0.28)
3. **Sequence-level clipping (GSPO)** as stable baseline
4. **Active injection of token-level weight distortion** (controlled experiments)

## Empirical Validation / Results

### Initial Training-Inference Discrepancy Effects

**Truncated Importance Sampling (TIS)** markedly reduces the growth rate of training-inference discrepancy while slowing token entropy decline and improving validation scores. However, the discrepancy is not completely eliminated, indicating multiple contributing sources.

**Scatter analysis** of $(\pi_{\mathrm{train}}, \pi_{\mathrm{infer}})$ shows:
- Early training: high Pearson correlation coefficient (PCC)
- Later stages: pronounced discrepancy with abnormal clustering along the $\pi_{\mathrm{infer}}$ dimension, attributed to limited numerical precision in inference mode

### Token-Level Clipping Effects

A **counterintuitive finding**: stronger token-level clipping actually *accelerates* training-inference discrepancy growth, despite being designed to stabilize training.

**Table 1: Variation of clip ratio during RLVR training**

| Gradient Step | 4 | 200 | 400 |
|---|---|---|---|
| Token (low) | $1.50 \times 10^{-3}$ | $1.15 \times 10^{-3}$ | $1.26 \times 10^{-3}$ |
| Token (middle) | $1.92 \times 10^{-3}$ | $1.81 \times 10^{-3}$ | $2.84 \times 10^{-3}$ |
| Token (high) | $2.37 \times 10^{-3}$ | $2.63 \times 10^{-3}$ | $3.49 \times 10^{-3}$ |
| Seq. | 0.153 | 0.120 | 0.085 |

In contrast, sequence-level clipping does not exhibit the same anomalous discrepancy growth, providing indirect evidence for the role of token-level clipping.

### Causal Evidence from Injected Distortions

The paper designs controlled experiments by injecting token-level weight distortion into the GSPO objective:

$$
\mathcal{J}_{\mathrm{inj}} = \mathbb{E}_{\mathrm{train}} \left[ \sum_{i,t} \varphi_{i,t} \times Y_{i,t}(\theta) \right], \tag{17}
$$

$$
\varphi_{i,t} = \begin{cases} \delta, & \text{if } o_{i,t} \in S_{\mathrm{low-prob}}, \\ 1, & \text{otherwise}. \end{cases} \tag{18}
$$

Key results:
- **Even a modest 20% weight increase** ($\delta = 1.2$) for low-probability tokens triggers discrepancy growth
- **Stronger distortions induce faster discrepancy growth** (statistically correlated)
- Both increasing *and* decreasing low-probability token weights reproduce the anomaly, confirming it's the *distortion* itself that matters
- Model performance shows **irreversible degradation** alongside token entropy anomalies

### Bias vs. Variance

Critical controlled experiment: injecting **variance-based noise** ($\xi_{i,t} \sim \mathcal{N}(1, \sigma^2)$) does *not* trigger abnormal discrepancy growth:

$$
\Delta\mathcal{J}_{\mathrm{var}}(\theta) = \mathbb{E}_{\mathrm{train}} \left[ \sum_{i,t} Y_{i,t}(\theta) (\xi_{i,t} - 1) \right] \simeq 0. \tag{22}
$$

This confirms that **biased distortion is the primary driver**, as biased objective-level spurious signals can be persistently exploited by optimization.

### Positive Feedback Loop

The paper documents a self-reinforcing mechanism:
1. Low-probability tokens exhibit wider dispersion from $\rho_{i,t} = 1$ (greater mismatch)
2. Importance weights $\rho_{i,t}$ of low-probability tokens **consistently decrease** as training progresses (survival bias effect)
3. This divergence exacerbates objective-level hacking, driving further discrepancy growth
4. The loop explains **irreversibility of training collapse**

### MoE vs. Dense Models

Using DeepSeek-R1-Distill-Qwen-7B as a dense baseline, the paper finds:

| Model | Initial Mismatch | Final Mismatch | Growth |
|---|---|---|---|
| Dense (7B) | $1.059 \times 10^{-2}$ | $1.081 \times 10^{-2}$ | ~2% |
| MoE (30B) | $1.418 \times 10^{-2}$ | $2.664 \times 10^{-2}$ | ~88% |

MoE models exhibit both **larger initial mismatch** and **substantially faster mismatch growth**, likely due to expert activation inconsistency between training and inference.

## Theoretical and Practical Implications

### Theoretical Contributions

1. **New analytical lens**: The paper introduces objective-level hacking as a distinct failure mode from reward hacking, shifting focus from verifier vulnerabilities to optimization objective distortions.

2. **Unified mathematical framework**: Equations (10), (14), and (20) provide a unified formulation showing that diverse instability sources (infrastructure mismatch, clipping strategies, weight distortions) all manifest as covariance terms or biased perturbations in the optimization objective.

3. **Causal understanding**: The controlled injection experiments establish causality (not mere correlation) between token-level weight distortion and training-inference discrepancy growth.

### Practical Implications

1. **Algorithm design guidance**: The findings suggest that RLVR algorithms should **avoid token-level modulations that introduce biased perturbations**, favoring sequence-level approaches (e.g., GSPO) for MoE training stability.

2. **Stabilization strategy**: Since biased (not variance-based) distortions are the primary driver, correction methods like TIS that address objective bias are more principled than infrastructure-level fixes alone.

3. **MoE-specific considerations**: The framework explains why MoE models are particularly vulnerable, suggesting that expert activation consistency should be a focus for stable training.

4. **Monitoring tool**: The surrogate metric $J \equiv \sum_{i,t} \hat{A}_{i,t} \times (\varphi_{i,t} - 1)$ (Eq. 23) provides a practical way to monitor objective-level hacking during training.

## Conclusion

This paper provides a principled framework for understanding RLVR training instability in MoE models through the lens of **objective-level hacking**. The key insight is that token-level credit misalignment—whether from infrastructure discrepancies, clipping strategies, or other modulations—introduces system-level spurious signals in the optimization objective that drive pathological training dynamics.

The authors trace the mechanism behind the abnormal growth of training-inference discrepancy, demonstrating through controlled experiments that **biased token-level weight distortions** (not variance noise) causally trigger this growth. The resulting **positive feedback loop** between discrepancy and hacking explains the irreversibility of training collapse and the heightened vulnerability of MoE architectures.

### Future Directions

The authors acknowledge that RLVR training is a complex multi-system interaction, and their framework does not capture the full picture of model collapse. Future work should explore:
- Additional sources of training instability (data quality, rollout engine choices, hardware configurations)
- Interaction effects between different instability mechanisms
- Algorithm designs that specifically protect the sensitive inference behavior of MoE models
- Extensions to other architectures and training paradigms

The framework offers a valuable new perspective for improving the **interpretability and controllability** of RLVR training, with direct implications for designing more robust reinforcement learning algorithms for MoE models.

---

_Markdown view of https://picx.dev/p/dbV39w, served by PicX — AI-generated visual whiteboard summaries of research papers._