Summary (Overview)

• Identifies training-inference mismatch in LLM RL: separate training and inference engines produce different action distributions (policies (\pi) and (\mu)) even with synchronized parameters, causing a form of off-policyness.
• Proposes a new learning principle – Monotonic Inference Policy Improvement (MIPI) – which targets monotonic improvement of the inference policy (\mu) rather than the training policy (\pi), thereby aligning the objective with deployment reality.
• Introduces MIPU (Monotonic Inference Policy Update), a two-step framework: (1) a sampler-referenced policy update with truncated importance weighting to construct better candidates, and (2) an inference-gap-aware acceptance test that filters synchronized candidates based on a proxy of the post-update inference gap.
• Demonstrates effectiveness under high mismatch (FP8-quantized rollout) on Qwen3-4B and Qwen3-1.7B: MIPU achieves the highest average pass@1 across five math reasoning benchmarks (66.71% and 53.97%) and maintains stable training while baselines collapse.
• Ablation studies confirm complementarity: Step 1 improves candidate direction; Step 2 selectively accepts reliable candidates based on an inference-side signal, not merely by rejecting more updates.

Introduction and Theoretical Foundation

Background and Motivation
Reinforcement Learning (RL) for LLM post-training (e.g., GRPO, DAPO) often separates rollout generation (inference engine, e.g., vLLM) from gradient computation (training engine, e.g., FSDP). Implementation differences in precision, decoding, or serving backends cause the training policy (\pi) and the inference policy (\mu) to assign different probabilities to the same trajectories, even with identical model parameters. This training-inference mismatch induces a special off-policyness that can destabilize training and lead to collapse.

Objective Misalignment
Existing works (TIS, MIS, LR-decay) attempt to correct or filter mismatch on the training side. However, even if a training-side update improves (\pi), it does not guarantee an improvement in (\mu) – the policy actually used for deployment. The paper formalizes this as:

[ J(\pi_{k+1}) - J(\pi_k) \geq 0 \nRightarrow J(\mu_{k+1}) - J(\mu_k) \geq 0 ]

This motivates a new objective that directly targets inference-policy improvement.

MIPI Principle
The proposed Monotonic Inference Policy Improvement (MIPI) defines the target as (J(\mu_{k+1}) - J(\mu_k)). By decomposing this into three terms:

[ J(\mu_{k+1}) - J(\mu_k) = \underbrace{J(\mu_{k+1}) - J(\pi_{k+1})}{①\ \text{post-update inference gap}} + \underbrace{J(\pi{k+1}) - J(\pi_k)}{②\ \text{training-side update}} + \underbrace{J(\pi_k) - J(\mu_k)}{③\ \text{pre-update inference gap}} ]

MIPI separates candidate construction (terms ②+③) from post-synchronization verification (term ①).

Methodology

MIPU Framework
Monotonic Inference Policy Update (MIPU) realizes the MIPI principle in two complementary steps:

Step 1: Sampler-Referenced Policy Update

Targets (J(\pi_{k+1}) - J(\mu_k)) (terms ②+③). Instead of the standard GRPO surrogate that clips the training-side ratio (\pi_\theta/\pi_k) while rollouts come from (\mu_k), MIPU uses a truncated sampler-referenced correction. The key factorization:

[ \rho_i(\theta) = \frac{\pi_\theta(y_i|x)}{\mu_k(y_i|x)} = \underbrace{\frac{\pi_k(y_i|x)}{\mu_k(y_i|x)}}{w_i^k:\ \text{pre-update mismatch}} \cdot \underbrace{\frac{\pi\theta(y_i|x)}{\pi_k(y_i|x)}}_{r_i(\theta):\ \text{current update}} ]

The Step-1 surrogate is:

[ J_{S1}(\theta) = \mathbb{E}{x\sim\mathcal{D},{y_i}{i=1}^G \sim \mu_k(\cdot|x)} \left[ \frac{1}{G} \sum_{i=1}^G \bar{w}_i^k \min\left( r_i(\theta) \hat{A}_i^{\mu_k}, \ \text{clip}(r_i(\theta), 1-\epsilon, 1+\epsilon) \hat{A}_i^{\mu_k} \right) \right] ]

where (\bar{w}i^k = \min(w_i^k, w{\max})) is the truncated mismatch weight. PPO-style clipping is applied only to (r_i(\theta)), the current update ratio. Optimizing this yields a candidate training policy (\pi_{k+1}).

Step 2: Inference-Gap-Aware Update Acceptance

Evaluates the post-update gap (T_{\text{post}} = J(\mu_{k+1}) - J(\pi_{k+1})). Using the reverse performance difference identity, they construct a validation-based proxy:

[ \hat{T}{\text{post}} = -\mathbb{E}{x\sim\mathcal{D}{\text{val}},\ y_i\sim\mu{k+1}} \left[ \rho_i \hat{A}i^{\mu{k+1}} \right] ]

with length-normalized importance weight:

[ \rho_i = \exp\left( \frac{1}{T_i} \sum_{t=1}^{T_i} \log\frac{\pi_{k+1}(y_{i,t}|x, y_{i,<t})}{\mu_{k+1}(y_{i,t}|x, y_{i,<t})} \right) ]

The acceptance criterion is (\hat{T}_{\text{post}} \ge -c) (with tolerance (c \ge 0)). Updates failing this test are rejected and rolled back to the previous checkpoint. The full algorithm (Algorithm 1) includes checkpointing, synchronization, validation, and rollback.

Empirical Validation / Results

Setup

• Models: Qwen3-1.7B and Qwen3-4B.
• High-mismatch setting: FP8-quantized inference for rollout generation.
• Training data: filtered subsets of DAPO-Math-17 and DeepMath-103K.
• Evaluation: five mathematical reasoning benchmarks (MATH-500, AIME24, AMC23, Minerva, OlympiadBench). Pass@1 accuracy; avg@16 for small benchmarks.
• Baselines: standard GRPO, MIS (filtering), LR-decay.

Main Results (RQ1)

Table 1: Pass@1 accuracy (%) under FP8-quantized rollout.

MIPU achieves the best average performance on both model scales. Critically, the "Stable" column indicates that only MIPU maintains stable training without sharp degradation (Figure 2).

Ablation (RQ2)

Table 2: Ablation study on Qwen3-4B FP8-quantized.

Step 1 improves candidate direction; Step 2 alone prevents collapse but cannot improve poor candidates. The full method combines both, yielding the best overall performance and stability (Figure 3).

Analysis of Step 2 (RQ3)

• The post-update gap proxy (\hat{T}_{\text{post}}) carries a meaningful mismatch signal (correlates with inference-training KL divergence; Figure 4a).
• Compared to a random rollback control (70% rejection rate), Step 2 rejects fewer updates (53.5%) but maintains stable training while random rollback collapses. This shows Step 2 uses information, not mere conservatism.

Theoretical and Practical Implications

Theoretical Significance

• Shifts the perspective in LLM RL from training-policy improvement to inference-policy improvement, establishing a new objective (MIPI) that is naturally aligned with deployment.
• Provides a formal decomposition of inference-policy improvement that separates candidate construction from verification, enabling principled two-step optimization.
• Identifies that training-inference mismatch is not merely a system-level nuisance but an objective-level issue: improving the training policy does not imply improving the inference policy.

Practical Implications

• MIPU offers a drop-in framework that can be combined with various sampler-referenced update rules and acceptance criteria.
• The inference-gap proxy (\hat{T}_{\text{post}}) serves as a practical risk signal to filter unreliable synchronized updates, especially under high-mismatch conditions like low-precision inference.
• The complementary roles of Step 1 and Step 2 suggest that both candidate quality improvement and selective acceptance are necessary for stable training.

Limitations

• Experiments limited to moderate-scale models (up to 4B parameters); scalability to larger models needs verification.
• Step 2 implementation uses validation-based estimation; more efficient or direct optimization schemes are possible.
• The current paper does not provide a formal monotonic-improvement guarantee; MIPU reduces risk but does not eliminate it.

Conclusion

The paper revisits training-inference mismatch in LLM RL from an objective-level perspective, showing that standard RL updates optimize the training policy while deployment depends on the inference policy. To address this, the authors propose MIPI (Monotonic Inference Policy Improvement) and realize it through MIPU, a two-step framework: (1) sampler-referenced candidate construction using truncated importance weights, and (2) inference-gap-aware acceptance that filters unreliable synchronized candidates. Experiments under high-mismatch FP8-quantized rollout demonstrate that MIPU achieves better average reasoning performance and training stability compared to existing methods. The work suggests that training-inference mismatch should be treated as an objective-level challenge rather than merely a low-level system issue. Future directions include scaling to larger models, developing more efficient gap estimators, and exploring direct optimization of the post-update inference gap.