# The Mirage of Optimizing Training Policies: Monotonic Inference Policies as the Real Objective for LLM Reinforcement Learning > MIPU aligns RL with deployment by optimizing the inference policy directly, achieving stable training under high mismatch where baselines collapse. - **Source:** [arXiv](https://arxiv.org/abs/2606.29526) - **Published:** 2026-07-07 - **Permalink:** https://picx.dev/p/EV75Vd - **Whiteboard:** https://picx.dev/p/EV75Vd/image ## Summary ## Summary (Overview) - **Identifies training-inference mismatch** in LLM RL: separate training and inference engines produce different action distributions (policies \(\pi\) and \(\mu\)) even with synchronized parameters, causing a form of off-policyness. - **Proposes a new learning principle – Monotonic Inference Policy Improvement (MIPI)** – which targets monotonic improvement of the *inference* policy \(\mu\) rather than the training policy \(\pi\), thereby aligning the objective with deployment reality. - **Introduces MIPU (Monotonic Inference Policy Update)**, a two-step framework: (1) a sampler-referenced policy update with truncated importance weighting to construct better candidates, and (2) an inference-gap-aware acceptance test that filters synchronized candidates based on a proxy of the post-update inference gap. - **Demonstrates effectiveness under high mismatch (FP8-quantized rollout)** on Qwen3-4B and Qwen3-1.7B: MIPU achieves the highest average pass@1 across five math reasoning benchmarks (66.71% and 53.97%) and maintains stable training while baselines collapse. - **Ablation studies confirm complementarity**: Step 1 improves candidate direction; Step 2 selectively accepts reliable candidates based on an inference-side signal, not merely by rejecting more updates. ## Introduction and Theoretical Foundation **Background and Motivation** Reinforcement Learning (RL) for LLM post-training (e.g., GRPO, DAPO) often separates rollout generation (inference engine, e.g., vLLM) from gradient computation (training engine, e.g., FSDP). Implementation differences in precision, decoding, or serving backends cause the *training policy* \(\pi\) and the *inference policy* \(\mu\) to assign different probabilities to the same trajectories, even with identical model parameters. This `training-inference mismatch` induces a special off-policyness that can destabilize training and lead to collapse. **Objective Misalignment** Existing works (TIS, MIS, LR-decay) attempt to correct or filter mismatch on the training side. However, even if a training-side update improves \(\pi\), it does **not** guarantee an improvement in \(\mu\) – the policy actually used for deployment. The paper formalizes this as: \[ J(\pi_{k+1}) - J(\pi_k) \geq 0 \nRightarrow J(\mu_{k+1}) - J(\mu_k) \geq 0 \] This motivates a new objective that directly targets **inference-policy improvement**. **MIPI Principle** The proposed **M**onotonic **I**nference **P**olicy **I**mprovement (MIPI) defines the target as \(J(\mu_{k+1}) - J(\mu_k)\). By decomposing this into three terms: \[ J(\mu_{k+1}) - J(\mu_k) = \underbrace{J(\mu_{k+1}) - J(\pi_{k+1})}_{①\ \text{post-update inference gap}} + \underbrace{J(\pi_{k+1}) - J(\pi_k)}_{②\ \text{training-side update}} + \underbrace{J(\pi_k) - J(\mu_k)}_{③\ \text{pre-update inference gap}} \] MIPI separates candidate construction (terms ②+③) from post-synchronization verification (term ①). ## Methodology **MIPU Framework** **M**onotonic **I**nference **P**olicy **U**pdate (MIPU) realizes the MIPI principle in two complementary steps: ### Step 1: Sampler-Referenced Policy Update Targets \(J(\pi_{k+1}) - J(\mu_k)\) (terms ②+③). Instead of the standard GRPO surrogate that clips the training-side ratio \(\pi_\theta/\pi_k\) while rollouts come from \(\mu_k\), MIPU uses a truncated sampler-referenced correction. The key factorization: \[ \rho_i(\theta) = \frac{\pi_\theta(y_i|x)}{\mu_k(y_i|x)} = \underbrace{\frac{\pi_k(y_i|x)}{\mu_k(y_i|x)}}_{w_i^k:\ \text{pre-update mismatch}} \cdot \underbrace{\frac{\pi_\theta(y_i|x)}{\pi_k(y_i|x)}}_{r_i(\theta):\ \text{current update}} \] The Step-1 surrogate is: \[ J_{S1}(\theta) = \mathbb{E}_{x\sim\mathcal{D},\{y_i\}_{i=1}^G \sim \mu_k(\cdot|x)} \left[ \frac{1}{G} \sum_{i=1}^G \bar{w}_i^k \min\left( r_i(\theta) \hat{A}_i^{\mu_k}, \ \text{clip}(r_i(\theta), 1-\epsilon, 1+\epsilon) \hat{A}_i^{\mu_k} \right) \right] \] where \(\bar{w}_i^k = \min(w_i^k, w_{\max})\) is the truncated mismatch weight. PPO-style clipping is applied only to \(r_i(\theta)\), the current update ratio. Optimizing this yields a candidate training policy \(\pi_{k+1}\). ### Step 2: Inference-Gap-Aware Update Acceptance Evaluates the post-update gap \(T_{\text{post}} = J(\mu_{k+1}) - J(\pi_{k+1})\). Using the reverse performance difference identity, they construct a validation-based proxy: \[ \hat{T}_{\text{post}} = -\mathbb{E}_{x\sim\mathcal{D}_{\text{val}},\ y_i\sim\mu_{k+1}} \left[ \rho_i \hat{A}_i^{\mu_{k+1}} \right] \] with length-normalized importance weight: \[ \rho_i = \exp\left( \frac{1}{T_i} \sum_{t=1}^{T_i} \log\frac{\pi_{k+1}(y_{i,t}|x, y_{i,