# DenoiseRL: Bootstrapping Reasoning Models to Recover from Noisy Prefixes > DenoiseRL trains reasoning models to correct mistakes by injecting erroneous partial solutions from a weaker model into training rollouts, improving performance without stronger teacher models. - **Source:** [arXiv](https://arxiv.org/abs/2605.28421) - **Published:** 2026-05-29 - **Permalink:** https://picx.dev/p/ZCaXax - **Whiteboard:** https://picx.dev/p/ZCaXax/image ## Summary # DenoiseRL: Bootstrapping Reasoning Models to Recover from Noisy Prefixes ## Summary (Overview) * **Core Idea:** DenoiseRL is a reinforcement learning (RL) framework that improves reasoning in large language models (LLMs) by training them to "recover" from incorrect reasoning steps ("noisy prefixes") generated by a weaker model, instead of relying on stronger teacher models or curated datasets. * **Key Mechanism:** It injects erroneous partial solutions from a weak model into the policy's training rollouts, forcing the model to learn how to correct mistakes and reach the correct answer from a corrupted intermediate state. * **Main Contributions:** DenoiseRL consistently improves standard on-policy RL methods (GRPO and DAPO) across multiple mathematical reasoning benchmarks (MATH500, AMC23, AIME, BBEH) on two model scales (4B and 8B parameters). * **Key Findings:** The method induces stronger self-correction behavior, but the intensity of the noise (prefix length) must be carefully tuned. Updating the model's parameters based on the off-policy prefix tokens leads to training instability. * **Implication:** Provides a scalable, resource-efficient pathway for post-training LLMs by turning model failures into a valuable learning signal. ## Introduction and Theoretical Foundation Reinforcement learning has become a dominant method for enhancing the reasoning capabilities of large language models (LLMs). However, state-of-the-art approaches often depend on supervision from even stronger teacher models or require extensive human effort to curate difficult training datasets. This creates a structural limitation: how can we create strong models without relying on pre-existing stronger models? Prior work follows two main directions: 1. **Weak-to-Strong Generalization:** Using weaker models to supervise stronger ones, but performance is capped by the teacher's quality. 2. **Difficulty-Driven Data Synthesis:** Creating harder problems, but this requires complex, manual data engineering. **DenoiseRL** unifies these ideas by repurposing the weak model not as a teacher, but as a generator of "structured perturbations." It frames reasoning RL as a **denoising problem**: errors from a weak model are treated as corruptions to a reasoning trajectory, and the policy model is trained to reconstruct a correct solution from these corrupted starting points. This approach increases training difficulty automatically, diversifies the training states (exposing the model to a wider range of failure modes), and directly targets the underdeveloped capability of **recovery from mistakes**. ## Methodology ### 3.1 Denoising Reasoning The core idea is to prepend an incorrect partial solution (a "noisy prefix") from a weak model to the policy's generation. The policy is then trained to continue reasoning from this corrupted state to reach the correct answer. **Formal Setup:** * A pool of incorrect solutions $\mathcal{W}(q)$ is created offline for each training question $q \in \mathcal{D}$ by sampling a weak model $\pi_w$ and filtering out wrong answers. * If $\pi_w$ never produces a wrong answer in $M$ trials, $\mathcal{W}(q)$ is empty, and the training proceeds with standard rollouts for that question. ### 3.2 Reinforcement Learning for Recovering from Noisy Prefixes Each training step samples two types of rollouts for a question $q$: 1. **Main Rollouts ($N$ per problem):** Standard on-policy generation. $$y \sim \pi_{\theta}(\cdot | q)$$ 2. **Denoise Rollouts ($K$ per problem):** Start from a noisy prefix $w \sim \mathcal{W}(q)$. A prefix of length $p$ is retained using a fixed ratio $\rho$: $$p = \max\left(1, \lfloor \rho |w| \rfloor\right)$$ The policy then continues from this prefix: $$y_{>p} \sim \pi_{\theta}(\cdot | q, w_{1:p})$$ **Output Budget and Folding:** To ensure a fair comparison, both rollout types share a maximum response length $R$. The complete, "folded" response for a denoise rollout is: $$\tilde{y} = [\underbrace{w_{1:p}}_{\text{prefix}}, \underbrace{y_{p+1:p+L}}_{\text{continuation}}], \quad p + L \leq R$$ where $L = \min(T_{y_{>p}}, R - p)$. The verifier assigns a terminal reward $r(\tilde{y}; q) \in \{0, 1\}$ based on the final answer. **Crucially, gradient updates are applied only to the on-policy continuation tokens $y_{p+1:p+L}$.** **Token-level GRPO Objective:** The advantage is computed per problem group $\mathcal{G}(q)$ containing all $N+K$ rollouts: $$A_i = \frac{r_i - \mu_q}{\sigma_q + \epsilon}, \quad \mu_q = \frac{1}{N+K}\sum_{j \in \mathcal{G}(q)} r_j, \quad \sigma^2_q = \frac{1}{N+K}\sum_{j \in \mathcal{G}(q)} (r_j - \mu_q)^2$$ The per-token importance ratio is: $$r_{i,t}(\theta) = \frac{\pi_{\theta}(y_{i,t} | c_{i,t}, y_{i,