KnowRL: Boosting LLM Reasoning via Reinforcement Learning with Minimal-Sufficient Knowledge Guidance

Summary (Overview)

• Key Insight: Proposes a minimal-sufficiency perspective for hint-based Reinforcement Learning for Verifiable Reasoning (RLVR), identifying that performance improves sharply with a critical segment of knowledge (critical-segment effect) rather than monotonically with hint length.
• Core Method: Introduces KnowRL, a framework that decomposes hints into atomic Knowledge Points (KPs) and uses Constrained Subset Search (CSS) to select compact, interaction-aware subsets for training, addressing the pruning interaction paradox (removing one KP may help, but removing multiple together can hurt).
• Main Result: Trains KnowRL-Nemotron-1.5B, achieving a new state-of-the-art average accuracy of 74.16 (with KPs) and 70.08 (without KPs) across eight mathematical reasoning benchmarks at the 1.5B scale, significantly outperforming strong baselines.

Introduction and Theoretical Foundation

Reinforcement Learning for Verifiable Reasoning (RLVR) improves LLM reasoning by optimizing for rule-based correctness but suffers from reward sparsity on hard problems. Recent hint-based RL methods inject partial solutions or abstract templates to mitigate this. However, they treat hint design as a quantity expansion problem, leading to three key challenges:

1. Guidance Redundancy: Only a small subset of information is needed to trigger successful reasoning.
2. Cross-Hint Inconsistency: Longer hints can introduce branching or ambiguity.
3. Guidance-Efficiency Trade-off: Abstraction-based hints often rely on costly teacher models.

The paper argues that the core challenge is selecting minimal, coherent knowledge units sufficient to overcome reward sparsity. It introduces a minimal-sufficiency perspective, empirically demonstrating the critical-segment effect where accuracy exhibits a sharp jump once key knowledge is provided.

Methodology

KnowRL follows a workflow of constructing candidate KPs, selecting a minimal-sufficient subset, and using it for RL training.

1. KP Curation Pipeline

For each training problem, a three-stage pipeline constructs candidate KPs:

1. Generating Correct Solutions: Sample from a strong model (DeepSeek-R1) until a correct solution is obtained.
2. Extracting Raw Knowledge Points: Prompt the model to extract only indispensable mathematical principles from the correct solution, yielding an initial set $K = \{ k_1, k_2, ..., k_n \}$.
3. Leakage Verification: Automatically verify each KP to ensure it is generalizable and not instance-bound.

2. Problem-wise KP Subset Selection

The goal is to select the most beneficial KP configuration $K^* \subseteq K$ for each problem. Performance is estimated via offline accuracy $A$ under different configurations: $A_\emptyset$ (no KPs), $A_K$ (all KPs), and $A_{-i} = A(K \setminus \{k_i\})$ (leave-one-out).

Several selection strategies are explored:

• Max-Score: Selects the configuration with the highest accuracy among $\{\emptyset, K, K \setminus \{k_i\}\}$.
• Leave-One-Out (LOO) Strategies: A parameterized operator $\Phi_\epsilon(K)$ selects based on a tolerance $\epsilon$.
  • S-LOO ($\epsilon=0$): Strict selection.
  • T-LOO ($\epsilon=1/32$): Tolerant selection allowing one-sample-scale rollback.
• Consensus-Based Robust Selection (CBRS): Treats each of 8 evaluation runs independently. Defines near-optimal configurations for run $j$ as: $$O^{(j)} = \{ c \mid A^{(j)}(c) \geq \max_{c'} A^{(j)}(c') - \delta \}$$ with $\delta = 1/32$. The robust consensus $O^*$ is the intersection of these sets, or the configuration with the most votes.
• Constrained Subset Search (CSS): Designed to address the pruning interaction paradox. It first identifies:
  • $H = \{ k_i \mid A_{-i} \geq \max(A_K, A_\emptyset) \}$ (non-degrading KPs).
  • $N = \{ k_i \in H \mid A_{-i} \geq A_{\max} \}$ (near-optimal removals), where $A_{\max} = \max_i A_{-i}$. KPs in $N$ are removed directly. Let $C = H \setminus N$. CSS enumerates subsets only within $C$ (search space size $2^{|C|}$) and chooses the final configuration via:
  $$S^* = \arg \max_S A(S)$$ over all constrained candidates plus $\emptyset$ and $K$.

3. RL Training Integration

The curated KP subsets (using CSS) are integrated into RL training via difficulty-aware prompt injection (added under a ## Hint header). The training uses GRPO-style group-based optimization with entropy annealing for faster convergence.

Empirical Validation / Results

Offline KP Selection Evaluation

The table below compares selection strategies on the base Nemotron-1.5B model across eight benchmarks. CSS achieves the best trade-off between accuracy and KP compactness.

Table 1: Offline KP selection strategies on Nemotron-1.5B. Avg.#KP denotes the average number of selected key knowledge points per problem. Green numbers indicate improvements over w/o KP.

Final RL Training Results

KnowRL-Nemotron-1.5B was trained on the QuestA dataset using CSS-selected KPs.

Table 3: Evaluation results of RL training with CSS-selected KP data under different test-time prompting strategies (with and without KPs).

Key Findings:

1. State-of-the-Art Performance: KnowRL achieves the highest average scores, establishing a new SOTA at the 1.5B scale.
2. Internalized Reasoning: The substantial improvement without KP hints at inference (70.08) shows that KnowRL improves the underlying policy itself, not just hint-conditioned behavior.
3. Effectiveness on Hard Problems: Gains are particularly large on challenging competition-style benchmarks (e.g., +15.11 on AIME25, +15.49 on CMIMC25 with CSS).

Analysis of Training Data Improvement

Training set analysis shows KnowRL effectively overcomes reward sparsity:

• Base Model: 41.21% of queries had zero correct answers (mean accuracy 22.40%).
• KnowRL (w/o inference KPs): Zero-correct fraction reduced to 13.00%, all-correct bucket raised to 34.28% (mean accuracy 64.30%).
• KnowRL (w/ inference KPs): All-correct bucket further concentrated to 51.07% (mean accuracy 77.04%).

Theoretical and Practical Implications

• Theoretical: Introduces and formalizes the minimal-sufficiency perspective and the pruning interaction paradox for hint-based RL. Provides a principled framework (CSS) for interaction-aware subset selection.
• Practical: Demonstrates that compact, structured guidance is more effective and efficient than longer hints or heavy abstractions. The method reduces computational overhead by minimizing hint length and avoiding reliance on teacher models during online RL.

Conclusion

KnowRL presents an effective framework for RLVR that uses minimal-sufficient knowledge guidance. By decomposing hints into KPs and selecting robust subsets via CSS, it achieves new state-of-the-art results while maintaining efficiency. The work positions structured, compact guidance as a practical scaling principle for sparse-reward RL and opens directions for extending KP curation to broader reasoning domains.