Summary (Overview)

• ThinkTwice is a novel two-phase RLVR (Reinforcement Learning with Verifiable Rewards) framework that jointly optimizes LLMs for both reasoning and self-refinement, using only binary correctness rewards without external supervision signals, critique annotations, or verifiers.
• The framework alternates between two GRPO-based training phases: (1) optimizing the model to solve reasoning problems, and (2) optimizing it to refine its own previously generated solutions to the same problems.
• Key results: On Qwen3-4B, ThinkTwice outperforms GRPO on AIME by 5 percentage points before refinement and by 11.5 points after one self-refinement step (pass@4). Across five math benchmarks and two model families (Qwen3-4B, Olmo3-7B), it achieves the highest average performance for both reasoning and self-refinement.
• Training dynamics analysis reveals an implicit rectify-then-fortify curriculum: refinement predominantly corrects errors early in training, then naturally shifts toward preserving and polishing already-correct solutions as the model improves.
• ThinkTwice adds only 3% training overhead per step and reaches its best checkpoint in 16% less wall-clock time than GRPO, demonstrating both efficiency and effectiveness.

---

Introduction and Theoretical Foundation

Background & Motivation

Reinforcement learning with verifiable rewards (RLVR) has become an effective paradigm for improving LLM reasoning capabilities (Shao et al., 2024; Guo et al., 2025). However, even strong reasoners produce solutions containing correctable errors—incomplete derivations, algebraic mistakes, or unproductive solution paths. Drawing from human problem-solving (Polya, 1945), self-refinement—revisiting initial solutions, identifying errors, and revising reasoning—offers a natural remedy.

Limitations of Existing Approaches

Theoretical Foundation

ThinkTwice is built on Group Relative Policy Optimization (GRPO), a critic-free PPO-style algorithm. The GRPO objective is:

$$J_{GRPO}(\theta) = \mathbb{E}_{x, \{y_i\}_{i=1}^{G}} \left[ \frac{1}{G} \sum_{i=1}^{G} L_i - \beta D_{KL}(\pi_\theta \| \pi_{ref}) \right]$$

where $L_i = \min(\rho_i A_i, \text{clip}(\rho_i, 1-\epsilon, 1+\epsilon) A_i)$ is the clipped surrogate objective, $\rho_i = \pi_\theta(y_i|x)/\pi_{\theta_{old}}(y_i|x)$ is the importance ratio, and $\beta$ controls KL penalty. The advantage is computed via group normalization:

$$A_i = \frac{r_i - \text{mean}(r_1, \ldots, r_G)}{\text{std}(r_1, \ldots, r_G)}$$

with binary correctness reward $r_i = \mathbb{1}[E(y_i) = a^*]$ for mathematical reasoning.

---

Methodology

Self-Refinement Formulation

Given a problem $x$ and a base solution $y_{base}$ generated by the current policy, the refinement prompt is constructed as a multi-turn conversation:

$$x_{refine} = [\text{User}: x][\text{Ast}: y_{base}][\text{User}: I_{refine}]$$

where $I_{refine}$ is a task-agnostic instruction containing no correctness signals. The refinement instruction asks the model to:

1. Review each calculation step-by-step for errors
2. Explicitly point out mistakes and explain the correct approach
3. Verify and clarify if the solution is already correct
4. Provide a refined solution and answer

ThinkTwice Two-Phase Training

Algorithm 1: ThinkTwice

for each training iteration t = 1, 2, ..., T do
    // Phase 1: Reasoning
    Sample batch of problems {x_j} from D
    Generate solutions {y_{j,1}, ..., y_{j,G}} ~ π_θ(·|x_j)
    Compute rewards r_{j,i} = 1[E(y_{j,i}) = a*_j]
    Update π_θ to π'_θ on reasoning samples
    
    // Phase 2: Refinement
    Sample refinement pairs from: {(x_j, y_{j,i})}
    Construct x_refine via Eq. (3)
    Generate refinements {y^(ref)_1, ..., y^(ref)_G} ~ π'_θ(·|x_refine)
    Compute rewards r^(ref)_i = 1[E(y^(ref)_i) = a*]
    Update π'_θ to π''_θ on refinement samples
end for

Key design choices:

1. Base Solution Sampling: Randomly picks one base solution per problem—this inherently covers the full spectrum of training samples and establishes an emergent curriculum (error correction early, polishing later).

2. Refined Solution Reward: Only the binary correctness reward is used; no extra signals enforce adherence to the refinement instruction's structure. The model is free to develop its own refinement strategies through RL.

Training Setup: Models trained on MATH training dataset (7,500 problems), evaluated on AIME, AMC, MATH500, Minerva Math, and OlympiadBench. Key hyperparameters: learning rate $1 \times 10^{-6}$, group size $G=8$, max response length 3000, temperature 1.0 (train) / 0.0 (val).

---

Empirical Validation / Results

Reasoning Performance (pass@4)

Table 1: Reasoning performance across five benchmarks

Notably, ThinkTwice's gains are obtained on direct prompting alone—before any self-refinement step—indicating the refinement training phase itself strengthens reasoning.

Self-Refinement Performance (pass@4)

Table 2: Self-refinement performance

† denotes training-free methods. ThinkTwice outperforms DAPO by 2.9 points and GRPO by 4.5 points on average.

Cross-Model Refinement

ThinkTwice as a refinement model achieves the highest score regardless of which model produced the base solution, demonstrating that its refinement capability generalizes beyond its own outputs.

Training Dynamics: Rectify-then-Fortify Curriculum

• Early training: ThinkTwice maintains consistently higher fix-wrong rate (fraction of incorrect base solutions becoming correct after refinement), peaking mid-training—refinement recovers useful learning signal from failed solutions.
• Late training: Damage-correct rate (correct bases becoming incorrect) drops near zero, while baseline's rate is consistently more than 5× higher. Refined outputs become significantly shorter and exhibit better formatting (higher rates of boxed answers and Final Answer markers) despite no format reward.

Training Cost

• Mean reward remains comparably stable to GRPO (no destabilization)
• ThinkTwice is only 3% slower in total wall-clock time (9.42h vs 9.15h)
• Reaches best checkpoint in 16% less wall-clock time (7.2h vs 8.6h) due to faster convergence (220 vs 280 steps)

---

Theoretical and Practical Implications

Theoretical Contributions

1. Principled joint optimization: ThinkTwice establishes joint training of reasoning and self-refinement as a principled methodology for RLVR, demonstrating that a single shared policy can learn both capabilities without external supervision.

2. Emergent curriculum: The two-phase design naturally creates a rectify-then-fortify dynamic—refinement transitions from error correction to solution preservation as the model improves, yielding a more rectified reward signal that accelerates convergence.

3. Minimal supervision requirement: ThinkTwice uses only binary correctness rewards in both phases, eliminating the need for process labels, critique annotations, correctness hints, or explicit verifiers—a significant theoretical advance for settings where external feedback is unavailable.

Practical Implications

1. Domain-agnostic design: The framework's design is applicable to any task with outcome-based rewards, including code generation and other verifiable domains.

2. Efficiency gains: Faster convergence (16% less wall-clock time to best checkpoint) with only 3% training overhead makes ThinkTwice practical for real deployment.

3. Improved refinement quality: ThinkTwice produces more concise, better-formatted, and more structurally sound refinements compared to baselines—the refinement acts as an "exploitation layer" that strips away dead ends and exploratory clutter.

4. Generalization: Cross-model refinement results show ThinkTwice's refinement capability transfers to improving other models' solutions, not just its own outputs.

---

Conclusion

ThinkTwice introduces a two-phase RLVR framework that jointly optimizes reasoning and self-refinement using the same binary correctness reward in both phases—without correctness signals, critique annotations, or external verifiers. Across five mathematical reasoning benchmarks and two model families, it consistently outperforms competitive baselines (GRPO, Dr. GRPO, DAPO) on both direct reasoning and self-refinement.

The training dynamics reveal an implicit rectify-then-fortify curriculum that yields a more rectified reward signal while adding only minimal training overhead. Qualitative case studies confirm three recurring refinement behaviors: (i) route switching (abandoning bad solution paths), (ii) solution completion (finishing stalled derivations), and (iii) late-stage fortification (compressing correct solutions).

Future directions include:

• Applying ThinkTwice to other outcome-reward tasks such as code generation
• Exploring multi-step iterative refinement (the framework natively supports arbitrary numbers of refinement turns within the multi-turn conversation format)

The codebase is publicly available at https://github.com/CSSLab/ThinkTwice.