# Bootstrapping Exploration with Group-Level Natural Language Feedback in Reinforcement Learning > GOLF improves RL sample efficiency by 2.2x using aggregated natural language feedback to guide exploration in sparse-reward tasks. - **Source:** [arXiv](https://arxiv.org/abs/2603.04597) - **Published:** 2026-03-13 - **Permalink:** https://picx.dev/p/D4Do4z - **Whiteboard:** https://picx.dev/p/D4Do4z/image ## Summary # Bootstrapping Exploration with Group-Level Natural Language Feedback in Reinforcement Learning ## Summary (Overview) - **Key Contribution**: Proposed **GOLF**, a novel RL framework that leverages **Group-level Natural Language Feedback** (aggregating external critiques and intra-group attempts) to produce actionable refinements that guide targeted exploration and improve training efficiency. - **Core Mechanism**: Aggregates complementary NL feedback sources, adaptively injects high-quality refinements as off-policy scaffolds in low-reward regimes, and jointly optimizes generation and refinement within a unified RL loop. - **Performance Gains**: Achieves superior performance across both verifiable (math, instruction following, code) and non-verifiable (chat, creative writing) benchmarks, with **~2.2x improvement in sample efficiency** compared to scalar-reward-only RL methods. - **Enhanced Exploration**: Maintains higher policy entropy and improves `Pass@k` scores, indicating broader solution coverage and more diverse exploration. - **Capability Development**: Joint training improves both direct problem-solving and self-refinement capabilities, enabling better utilization of NL feedback at inference time. ## Introduction and Theoretical Foundation **Background & Motivation**: Reinforcement Learning (RL) for Large Language Models (LLMs), such as RLHF and RLVR, typically relies solely on scalar rewards (success/failure signals). This leads to **inefficient exploration**, as the policy lacks explicit guidance on how to improve and must rely on costly trial-and-error. In many real-world scenarios, LLMs receive richer **Natural Language (NL) feedback** (e.g., error diagnoses, revision suggestions), but current RL algorithms (e.g., GRPO) do not fully exploit this information. **Core Insight**: NL feedback, when aggregated from **multiple complementary sources**, can be translated into actionable refinements that provide explicit guidance, densifying learning signals and alleviating exploration bottlenecks in sparse-reward regions. **Theoretical Basis**: The framework builds upon **Group Relative Policy Optimization (GRPO)**. The problem is exacerbated when group-normalized advantages collapse (e.g., all-zero reward groups), yielding vanishing gradients. GOLF addresses this by injecting high-quality refinements derived from NL feedback to restore informative advantages and provide targeted guidance. ## Methodology GOLF consists of three tightly coupled components: ### 1. Group-level Feedback Aggregated Refinement For each prompt $x$, sample a group of $N$ responses: $$G_{\text{gen}}(x) = \{y^{(i)}\}_{i=1}^{N}, \quad y^{(i)} \sim \pi_{\theta_{\text{old}}}(\cdot|x).$$ Receive scalar reward and critique: $(r^{(i)}, c^{(i)}) = R(x, y^{(i)})$. - **External Feedback**: Critique $c^{(i)$ associated with a specific response. - **Intra-group Feedback**: Alternative responses within $G_{\text{gen}}(x)$, containing complementary partial ideas. Collect the **failure set**: $$F(x) = \{ (y^{(i)}, c^{(i)}) | r^{(i)} = 0 \}.$$ Construct an **aggregated refinement prompt** by concatenating the prompt and the failure set: $$p_{\text{agg}}(x) = \text{CONCAT}(x, F(x)).$$ Conditioned on $p_{\text{agg}}(x)$, sample a refinement group: $$G_{\text{refine}}(x) = \{\tilde{y}^{(j)}\}_{j=1}^{N}, \quad \tilde{y}^{(j)} \sim \pi_{\theta_{\text{old}}}(\cdot|p_{\text{agg}}(x)),$$ and score each: $\tilde{r}^{(j)} = R(x, \tilde{y}^{(j)})$. ### 2. Adaptive Guidance via Mixed Policy Optimization **Adaptive Injection**: Compute the group's average reward: $$s(x) = \frac{1}{N} \sum_{y \in G_{\text{gen}}(x)} r(x, y).$$ Trigger injection when $s(x)$ falls below a threshold $\tau$ (default: $1/N$). Form the set of successful refinements: $$S_{\text{ref}}(x) = \{ \tilde{y} \in G_{\text{ref}}(x) | \tilde{r}(x, \tilde{y}) = 1 \}.$$ If $S_{\text{ref}}(x) \neq \emptyset$, randomly select $\tilde{y}^\star \in S_{\text{ref}}(x)$ and inject it by replacing one failed response in $G_{\text{gen}}(x)$. **Mixed Policy Optimization**: Let $G_{\text{aug}}(x) = G_{\text{on}}(x) \cup G_{\text{off}}(x)$, where $G_{\text{on}}$ are on-policy rollouts and $G_{\text{off}}$ are injected refinement trajectories. Optimize using a mixed objective: $$J_{\text{Mixed}}(\theta) = \frac{1}{Z} \left[ \sum_{i=1}^{N_{\text{on}}} \sum_{t=1}^{|\tau_i|} \text{CLIP}\left(r^{\text{on}}_{i,t}(\theta), \hat{A}_i, \epsilon\right) + \sum_{j=1}^{N_{\text{off}}} \sum_{t=1}^{|\tau_j|} \text{CLIP}\left(f(r^{\text{off}}_{j,t}(\theta)), \hat{A}_j, \epsilon\right) \right],$$ where: - $Z = \sum_{i=1}^{N_{\text{on}}} |\tau_i| + \sum_{j=1}^{N_{\text{off}}} |\tau_j|$ normalizes by total tokens. - $r^{\text{on}}_{i,t}(\theta) = \frac{\pi_\theta(\tau_{i,t}|x, \tau_{i,