# FIPO: Eliciting Deep Reasoning with Future-KL Influenced Policy Optimization

> FIPO uses a Future-KL divergence term to create a dense advantage signal, enabling models to produce significantly longer and more accurate reasoning chains without a critic model.

- **Source:** [arXiv](https://arxiv.org/abs/2603.19835)
- **Published:** 2026-04-02
- **Permalink:** https://picx.dev/p/JMTLfB
- **Whiteboard:** https://picx.dev/p/JMTLfB/image

## Summary

# FIPO: Eliciting Deep Reasoning with Future-KL Influenced Policy Optimization - Summary

## Summary (Overview)
*   **Core Contribution:** Introduces **Future-KL Influenced Policy Optimization (FIPO)**, a reinforcement learning algorithm designed to overcome reasoning bottlenecks in large language models (LLMs) by addressing the coarse-grained credit assignment problem inherent in standard Group Relative Policy Optimization (GRPO).
*   **Key Mechanism:** FIPO modifies the policy update by incorporating a discounted **Future-KL divergence** term. This creates a dense advantage formulation that re-weights tokens based on their estimated influence on subsequent trajectory behavior, distinguishing critical logical pivots from trivial tokens.
*   **Empirical Results:** On Qwen2.5-32B-Base, FIPO significantly outperforms baselines:
    *   Increases **AIME 2024 Pass@1 accuracy** from 50.0% (DAPO baseline) to a **peak of 58.0%** (converging at ~56.0%), surpassing DeepSeek-R1-Zero-Math-32B (~47.0%) and matching/beating o1-mini (~56.0%).
    *   Extends the **average chain-of-thought (CoT) length** from roughly **4,000 tokens to over 10,000 tokens**, breaking the "length stagnation" observed in standard baselines.
*   **Implication:** Demonstrates that establishing dense advantage formulations is a viable path for evolving outcome-based reward (ORM) algorithms to unlock the full reasoning potential of base models, **without requiring complex critic models** used in PPO-based approaches.

## Introduction and Theoretical Foundation
*   **Background:** Test-time scaling strategies (e.g., OpenAI's o-series, DeepSeek-R1) use large-scale reinforcement learning with verifiable rewards (RLVR) to elicit long chain-of-thought reasoning. Open-source efforts like DAPO reproduce GRPO-style training but face limitations.
*   **Problem:** Standard GRPO relies on **outcome-based rewards (ORM)** that are binary-verifiable only at the trajectory end. This results in a **coarse-grained credit assignment** where a uniform advantage is broadcast to every token, treating critical reasoning steps and trivial tokens equally.
*   **Consequence:** This imposes a **performance ceiling**, as models cannot converge to the complex, extended reasoning paths needed for difficult tasks. Reasoning trajectories plateau at intermediate lengths (~4,000 tokens).
*   **Insight from Prior Work:** Research shows RL updates are highly sparse, intervening at only a few "critical tokens" to keep reasoning on track. The instantaneous log-probability difference ($\Delta \log p$) indicates the direction of optimization but is a primitive, localized signal.
*   **FIPO's Foundation:** The goal is to leverage $\Delta \log p$ to formulate a more accurate measure of a token's true **downstream impact**, enabling automatic location and reinforcement of critical junctions during RL training.

## Methodology

**FIPO builds upon the GRPO/DAPO framework but introduces a Future-KL re-weighted advantage.**

### 1. Probability Shift: $\Delta \log p$
The atomic unit for credit assignment is the token-level probability shift between the current and old policy:
$$ \Delta \log p_t = \log \pi_\theta(o_t | q, o_{<t}) - \log \pi_{\theta_{\text{old}}}(o_t | q, o_{<t}) $$
A positive shift indicates reinforcement; a negative shift indicates suppression.

### 2. Future-KL Estimation
To capture the causal influence of a token, Future-KL is defined as the cumulative signed probability shift from the current step $t$ to the end of the sequence $T$:
$$ \text{FutureKL}_t = \sum_{k=t}^{T} \Delta \log p_k $$
This is a sample-based estimate of the KL divergence restricted to the future horizon.

**Refinements for Stability:**
*   **Masking Extreme Tokens:** To prevent variance from harmful actions, a binary filter $M_k$ excludes tokens whose importance ratio exceeds a Dual-Clip threshold $c$ (typically $c \geq 10$):
    $$ \text{FutureKL}_t = \sum_{k=t}^{T} M_k \cdot \Delta \log p_k, \quad M_k = \mathbb{I}\left( \frac{\pi_\theta(o_k | o_{<t})}{\pi_{\text{old}}(o_k | o_{<t})} \leq c \right) $$
*   **Soft Decay Window:** Incorporates a discount factor $\gamma \in (0, 1]$ to model diminishing influence over long horizons, prioritizing proximal signals. Parameterized as $\gamma = 2^{-1/\tau}$, where $\tau$ controls the effective half-life.
    $$ \text{FutureKL}_t = \sum_{k=t}^{T} M_k \cdot \gamma^{k-t} \cdot \Delta \log p_k $$

###民主 3. FutureKL Re-weighted Advantage with Clipping
The standard advantage estimate $\hat{A}_t$ is modulated by a future influence weight $f_t$:
$$ f_t = \text{clip}\left( \exp(\text{FutureKL}_t), 1 - \epsilon_{f_{\text{low}}}, 1 + \epsilon_{f_{\text{high}}} \right), \quad \tilde{A}_t = \hat{A}_t \cdot f_t $$
*   **Exponential Mapping:** Transforms the log-space cumulative signal to a multiplicative domain.
*   **Influence Weight Clipping:** Constrains $f_t$ to a bounded interval (e.g., $[1.0, 1.2]$ for 32B) to prevent excessive variance.

**Function:** When FutureKL > 0 (policy reinforces future), $f_t > 1$ magnifies the gradient signal (boosting positive advantages, harsher penalties for negative). When FutureKL < 0 (policy suppresses future), $f_t < 1$ attenuates the update.

### 4. FIPO Objective
Adopting the token-level formulation from DAPO, the objective to maximize is:
$$ J_{\text{FIPO}}(\theta) = \mathbb{E}_{(q,a)\sim\mathcal{D},\{o_i\}\sim\pi_{\theta_{\text{old}}}}\left[ \frac{1}{\sum_{i=1}^G |o_i|} \sum_{i=1}^{G} \sum_{t=1}^{|o_i|} \min\left( r_{i,t} f_{i,t} \hat{A}_{i,t}, \text{clip}(r_{i,t}, 1-\epsilon, 1+\epsilon) f_{i,t} \hat{A}_{i,t} \right) \right] $$
where $r_{i,t}$ is the importance ratio, $\hat{A}_{i,t}$ is the group relative advantage, and $f_{i,t}$ is the Future-KL importance weight.

**Table: Key Hyperparameters for Qwen2.5-32B-Base (FIPO vs. DAPO)**
| Hyperparameter | DAPO (Baseline) | FIPO (Ours) |
| :--- | :--- | :--- |
| **Shared Settings** | | |
| Base Model | Qwen2.5-32B-Base | Qwen2.5-32B-Base |
| Global Batch Size | 512 | 512 |
| Group Size (G) | 16 | 16 |
| Learning Rate | 1e-6 | 1e-6 |
| Max Response Length | 20,480 | 20,480 |
| **Method-Specific** | | |
| Mini-Batch Size | 32 | **64** (for stability) |
| Loss Function | DAPO | **Future-KL** |
| Future-KL Decay Rate ($\tau$) | - | **32.0** |
| Future-KL Clip Ratio | - | **[1.0, 1.2]** |
| Safety Threshold (Dual-Clip) | 10.0 | 10.0 |

## Empirical Validation / Results

**Primary Evaluation:** AIME 2024 and AIME 2025 benchmarks, using Qwen2.5-32B-Base trained on the public DAPO-17K dataset. Results are Pass@1 averaged over 32 samples (Avg@32).

**Table: Performance Comparison on AIME Benchmarks**
| Method | AIME 2024 | AIME 2025 |
| :--- | :--- | :--- |
| | Avg@32 | Cons@32 | Pass@32 | Avg@32 | Cons@32 | Pass@32 |
| **DAPO (Baseline)** | **50.0%** | **60.0%** | **80.0%** | **38.0%** | **47.0%** | **63.0%** |
| **FIPO (Ours)** | **56.0%** | **73.0%** | **83.0%** | **43.0%** | **50.0%** | **67.0%** |

**Key Findings:**

1.  **Length-Performance Scaling:** FIPO's performance gains are coupled with a **continuous expansion of response length**. While DAPO plateaus at ~4,000 tokens, FIPO scales the average CoT length to over 10,000 tokens. This length increase correlates strongly with improved accuracy ($R^2 > 0.78$ across stages).
2.  **Training Dynamics:**
    *   **Reward vs. Advantage:** DAPO maintains a higher mean *training reward* (due to shorter responses avoiding overlong penalties), but FIPO shows a **sustained upward trend in response-length-weighted mean advantage**, indicating longer valid reasoning chains yield increasingly positive signals.
    *   **Stability:** FIPO exhibits a steady increase in Policy KL and entropy, with low and consistent gradient norms, indicating smooth policy evolution. DAPO shows volatile fluctuations in gradient norm and entropy.
3.  **Qualitative Evolution:** Case studies reveal FIPO drives a qualitative transformation in reasoning strategy through distinct stages:
    *   Stage 1: Superficial planning (short, hallucinated).
    *   Stage 2: Linear execution (correct but single-pass CoT).
    *   Stage 3: **Emergent self-reflection** (cross-validation using alternative methods).
    *   Stage 4: **Systematic deep reasoning** (compute-heavy, multi-pass auditing and verification).

**Ablation Studies (Key Insights):**
*   **Filtering & Clipping:** The masking of extreme importance ratios and clipping of the influence weight $f_t$ are critical for training stability.
*   **Decay Horizon ($\tau$):** An intermediate horizon ($\tau=32$) strikes a balance, providing enough future signal without excessive volatility.
*   **Mini-Batch Size:** A larger mini-batch size (64 vs. 32) improves reproducibility and stability by reducing importance sampling variance.

## Theoretical and Practical Implications
*   **Algorithmic Significance:** Proves that **dense, token-level supervision can be achieved within the efficient GRPO framework without a critic model**, challenging the assumption that PPO's value network is necessary for fine-grained credit assignment.
*   **Unlocking Reasoning Depth:** Demonstrates that overcoming the "length stagnation" bottleneck is key to unlocking deeper reasoning capabilities in base LLMs. FIPO successfully elicits **inference-time reasoning** behaviors (like self-reflection) similar to advanced proprietary models.
*   **Open-Source Contribution:** The release of the complete training code and recipes (built on the `verl` framework) provides a scalable and accessible pathway for the research community to advance large-scale reasoning models.
*   **Scaling Insights:** Highlights fundamental differences in RL dynamics across model scales (7B vs. 32B). Smaller models may benefit from convergence to low-entropy, certain reasoning traces, while larger models leverage broad exploration enabled by dense advantage signals.

## Conclusion
*   **Main Takeaway:** **Future-KL Influenced Policy Optimization (FIPO)** effectively resolves the coarse credit assignment problem in ORM-based RL by creating a dense advantage formulation that re-weights tokens based on their downstream influence.
*   **Result:** This enables base models to break through performance ceilings, achieving significant gains in accuracy and eliciting substantially longer, more deliberate chain-of-thought reasoning.
*   **Future Directions:**
    1.  **Efficiency Optimization:** Transforming the elicited long reasoning paths into more concise forms.
    2.  **Task Generalization:** Exploring FIPO's efficacy beyond mathematical reasoning (e.g., coding, open-ended domains).
    3.  **Data Scalability:** Training on larger-scale or more diverse datasets.
    4.  **Model Scope:** Applying FIPO to models with pre-distilled Long-CoT capabilities.
    5.  **Bridging the Distillation Gap:** Addressing the performance gap between self-trained RL models and those distilled from larger teachers.

---

_Markdown view of https://picx.dev/p/JMTLfB, served by PicX — AI-generated visual whiteboard summaries of research papers._