Process-Verified Reinforcement Learning for Theorem Proving via Lean

Summary (Overview)

• Core Contribution: This paper introduces a novel reinforcement learning (RL) framework that leverages the Lean proof assistant as a process-level reward oracle, providing both global outcome signals and fine-grained tactic-level rewards during training—moving beyond the conventional use of Lean solely as a binary verifier.

• Key Innovation: The authors formalize Lean's symbolic feedback into a tactic-level MDP, using a first-error propagation rule and a first-token credit assignment strategy to transform structured proof feedback into dense, sound training signals within a GRPO-style objective.

• Empirical Results: The method consistently outperforms outcome-only GRPO baselines across MiniF2F and ProofNet benchmarks. For STP-Lean, it achieves up to +2.5%p improvement on MiniF2F (pass@64) and +1.4%p on ProofNet (pass@32), while demonstrating more stable training dynamics.

• Key Insight: The paper demonstrates that proof assistants can serve not only as verifiers at inference inference time but also as structured## Introduction and Theoretical Foundation

Background## Background and Motivation

The paper addresses a fundamental challenge in reinforcement learning for formal theorem proving: traditional RLVR (Reinforcement Learning from Verifiable Rewards) relies on sparse binary outcome signals, while formal proof assistants like Lean offer rich, structured feedback at each step of proof## 3.1 Lean4 Formalization

The paper formalizes Lean's feedback mechanism through three key functions:

• Parsing function $f: \mathcal{Y} \to \mathcal{T}^*$: extracts the sequence of tactics from a proof, sorted by starting position
• Global scoring function ### g: \mathcal{Y}# 3.2 Tactic-Level MDP

The framework models the proof generation process as a tactic-level MDP $\mathcal{M} = (\mathcal{S# 3.3 Credit Assignment in RL

The paper contrasts two RL approaches:

• PPO: Uses Generalized AdvantageSkip to content Advantage Estimation with# Methodology

4.1import numpy as np 1 Defining Tconst T# Tactic-Level Rewards

The## Empirical Validation / Results

The paper# Methodology

The paper defines two complementary reward signals:

• Outcome reward: $r_{\text{outcome}}(Y_i) = g(Y_i)$ where $g(Y_i) = 1$ if the proof passes Lean verification, 0 otherwise
• Process reward: $\varphi(Y, T)$ assigning values:
  • $1$ for successful tactics in complete# Empirical Validation / Results

The paper provides a detailed analysis of experimental results across multiple dimensions:

1. Main Results (Table 1): STP-Lean + Ours achieves 59.2% on MiniF2F## Theoretical and Practical Implications

The paper# Conclusion

The paper concludes with several key takeaways: