# When Does Exploration Prevent Collapse? Three Regimes and the Path of the Reward in Self-Consistency Training

> Self-consistency training exhibits three regimes of exploration strength, with a rare-answer bonus preventing collapse only when placed before GRPO normalization.

- **Source:** [arXiv](https://arxiv.org/abs/2611.01307)
- **Published:** 2026-11-14
- **Permalink:** https://picx.dev/p/TwJHGw
- **Whiteboard:** https://picx.dev/p/TwJHGw/image

## Summary

# When Does Exploration Prevent Collapse? Three Regimes and the Path of the Reward in Self-Consistency Training

**Authors:** Jaeho Hwang, Uiwon Hwang

---

## Summary (Overview)

- **Core Problem:** Test-time reinforcement learning (TTRL) adapts models without labels by rewarding answers that agree with the majority, but this causes the policy to collapse onto its initial majority answer—potentially removing correct answers from the sample space entirely.
- **Key Contribution:** The paper models training on a single problem as a continuous flow of softmax logits and derives closed-form boundaries for the exploration coefficient that divide training into **three regimes**: (1) below a lower boundary where only collapsed policies are stable, (2) above an upper boundary where only broad policies are stable, and (3) a window between them where the initial policy decides the outcome.
- **Path of the Reward:** The authors prove that exploration terms passing through the entire reward path (geometry → optimizer → normalization) have thresholds that remain invariant to learning rate and optimizer scaling under exact gradients, while exploration that bypasses any step shifts these boundaries.
- **Bonus for Rare Answers:** A bonus for rare answers prevents collapse above $c_b^* = 1/\log(G-1)$ for groups of $G$ samples when placed before GRPO normalization, but requires 2.3–3.2× larger coefficients when placed after normalization.
- **Empirical Validation:** Predictions are validated on a network testbed, a language model with single-letter answers, and free-form TTRL on MATH-500; the bonus placed before normalization prevented collapse and preserved correct answers in samples, though it lowered pass@1.

---

## Introduction and Theoretical Foundation

### Background and Motivation

Reinforcement learning with verifiable rewards (often via GRPO) improves LLM reasoning but requires labels or verifiers for every problem. **Test-time reinforcement learning (TTRL)** removes this requirement by:
- Sampling a group of answers per problem
- Rewarding answers that agree with the majority of the group

Related methods reward agreement, low entropy, or self-certainty. However, **self-consistency rewards favor what the policy already prefers**, causing collapse onto the initial majority. When the majority is wrong, the correct answer can vanish from samples early in training, and even verifiers or selectors cannot recover it.

### Theoretical Foundation

The paper models the dynamics of a single problem's answer distribution under the expected gradient of a softmax policy. The policy is $p = \text{softmax}(z)$ with logits $z$, and the analysis uses:

- **Agreement reward:** $R_i = p_i$, with advantage $p_i - S_2$ where $S_k = \sum_i p_i^k$
- **Majority reward of TTRL:** rewards samples equal to the most frequent answer

Two gradient geometries are considered:

- **Natural Policy Gradient (NPG):** $V_i = R_i - \langle R \rangle_p$
- **Vanilla Policy Gradient:** $V_i = p_i(R_i - \langle R \rangle_p)$

Both are covered by the unified flow:

$$V_i = p_i^q (p_i - S_2), \tag{1}$$

with $q = 0$ for NPG and $q = 1$ for vanilla policy gradient.

### The Path of the Reward

The reward reaches parameters through three transformations:
1. **Geometry** (policy gradient converts reward to logit velocity)
2. **Optimizer** (preconditioner $A(\theta)$—learning rate × identity for SGD, coordinatewise scaling for Adam)
3. **Normalization** (GRPO divides advantages by the standard deviation of rewards in each group)

The leave-one-out agreement reward has variance:

$$\varsigma_G(p)^2 = (S_3 - S_2^2) + \frac{S_2 - S_3}{G - 1}. \tag{4}$$

**Exploration is "on the path"** if it passes through all three transformations; otherwise it is "off the path."

---

## Methodology

### The Tabular Flow

The paper analyzes the **lattice equation** for a leak (mixing policy with neighbors):

$$\partial_t p_i = \omega(\Delta p)_i + p_i(V_i - \langle V \rangle_p), \qquad \omega = \frac{\varepsilon}{\eta}, \tag{2}$$

where $(\Delta p)_i = \sum_{j \sim i}(p_j - p_i)$ is the graph Laplacian.

### Exploration Arms

Four placements of exploration are compared:

| Arm | Description |
|-----|-------------|
| **U** | Unnormalized (like Dr. GRPO) |
| **R** | Divides reward advantages but not exploration (like GRPO's KL penalty) |
| **B** | Divides both, drops equal-reward groups (like DAPO) |
| **C** | Bonus added before division (like DARE) |

### Key Theoretical Results

**Theorem 4.1 (Potential):** The probability flow induced by the loss term has a potential function:

$$F_c(p) = \frac{1}{3}S_3 - \frac{1}{4}S_2^2 - \frac{c}{4}\sum_{i,j} D_{ij}(p_i - p_j)^2$$

and is a gradient flow in the Shahshahani metric. Equilibria and stability are identical for NPG and vanilla policy gradient.

**Lemma 4.2 (Preconditioners):** For a network with symmetric positive definite preconditioner $A(\theta)$, stationary points are unchanged: $\theta$ is stationary iff $\mathcal{V}(z(\theta)) = 0$.

**Proposition 4.3 (Normalization):**
- (a) If both terms divided by $\varsigma$: equilibria unchanged with same unstable directions
- (b) If only selection divided: equilibria at coefficient $c$ equal equilibria of (3) at coefficient $c\varsigma(p)$

### Exact Regimes for Unordered Answers

**Proposition 4.4 (Smoothing toward uniform):** For $D = \frac{1}{n}\mathbf{1}\mathbf{1}^\top$, with flow $z' = p \odot (p - S_2) + c(\bar{p} - p)$:

- (a) Uniform policy stable iff $c > \frac{1}{n}$
- (b) Equilibria with one leading entry $M$: $c = \frac{n}{n-1}M(1-M)$, stable if $M > \frac{1}{2}$
- (c) If $c > \frac{n}{4(n-1)}$: all solutions converge to $\bar{p}$; if $c < \frac{1}{n}$: only committed equilibria are stable

**Corollary 4.5:** Below the lower boundary, the flow commits to the **initial plurality**.

### Certified Folds on Ordered Bins

**Theorem 4.6 (Computer-assisted folds):** For Gaussian kernels $D_w$ with $w \in \{1, \frac{3}{2}, 3, 4, 8\}$ on rings/lines of 128–512 bins, nondegenerate folds exist at $c^*$ and $c_{\text{rel}}$. For $w = 3$: $c^* = 0.147749\ldots$ and $c_{\text{rel}} = 0.291945\ldots$

### Bonus for Rare Answers

A natural bonus gives sample $j$ the value $b_j = -\log(N_{a_j}/G)$. Near collapse, the threshold is:

- **Arms C and U:** sign of $c_b \log(G-1) - 1$
- **Arm R:** sign of $c_b \log(G-1) - G/\sqrt{G-1}$

---

## Empirical Validation / Results

### Network Testbed

A tanh network maps 32 states to logits over $K = 256$ ordered bins with PPO on the leave-one-out agreement reward ($G = 64$):

- Threshold variation ≤ 1.09× over learning rates up to 10× apart for SGD and AdamW
- Leak followed the unit law $\omega = \varepsilon/\eta$ within 7%
- Thresholds lay 3–29% above folds for $w \geq 1.5$
- Normalizing only advantages removed the window (Figure 1c)

### Letter Test (Qwen2.5-1.5B-Instruct)

Ten-option MMLU-Pro questions with single-letter answers:

- Agreement reward: release at fold 0.2431 or within one evaluation (4 seeds)
- Majority reward: dividing only advantages moved release by factor 1.89–2.13 (predicted 2.07)
- Bonus thresholds: broad starts ran at 1.18–1.25× predicted threshold; arm R stayed collapsed at 0.50–0.88× its near-collapse threshold

### Free-Form TTRL (Qwen2.5-Math-1.5B on MATH-500)

128 problems with half wrong initial pluralities, $G = 8$:

| Metric | Arm C ($c_b = 0.6$) | Control |
|--------|---------------------|---------|
| F95e (collapsed share) | ≤ 0.065 (from step 100) | 0.66–0.79 |
| Pass@32 | 0.710 | 0.561 |
| Pass@1 (temp 1) | 0.365 | 0.481 |
| Best-of-32 (PRM) | 0.569 | 0.526 |

**Threshold placement:** Arm C threshold in (0.45, 0.6] vs predicted 0.514; arm R in (1.2, 1.8] vs predicted 1.375 for broad starts.

### Key Threshold Table

**Table 1:** Thresholds of the bonus for rare answers (population standard deviation)

| | $G = 8$ | $G = 16$ | $G = 32$ | $G = 64$ |
|---|---|---|---|---|
| before, $c_b^* = 1/\log(G-1)$ | 0.514 | 0.369 | 0.291 | 0.241 |
| after, broad, ten answers | 1.454 | 1.074 | 0.876 | 0.780 |
| after, broad, free-form answers | 1.375 | 0.938 | 0.705 | 0.565 |
| after, near collapse | 1.554 | 1.526 | 1.674 | 1.946 |

---

## Theoretical and Practical Implications

### What Exploration Can and Cannot Do

**Key result (Proposition A.13):** If the logit velocity is locally Lipschitz and commutes with every permutation of answers, the **order of answer probabilities is preserved**. Exploration therefore decides *whether, when, and how firmly* a question commits—**not which answer wins**.

**Correcting a wrong majority** requires symmetry-breaking signals:
- Differential smoothing (treating correct/incorrect answers differently)
- Verifiers that reject consistent but wrong answers
- Parameters shared across questions (which break symmetry implicitly)

### Placement Matters

| Method | Acts as Exploration? |
|--------|---------------------|
| Temperature | No—rescales time only |
| Clipping, Adam | No—move no equilibria but affect window outcomes |
| Normalization | No—rescales exploration it doesn't divide |
| KL penalty | Partly—pulls toward reference, bypasses GRPO division |
| Smoothing to $\bar{p}$ | Yes—boundaries $\frac{1}{n}$ and $\frac{n}{4(n-1)}$ |
| Entropy bonus | Yes—Potts window under NPG |
| Bonus for rare answers | Yes—$c_b^*$ before division, 2.3–3.2×$c_b^*$ after |
| Leak | Yes—threshold in $\varepsilon/\eta$ |

### Practical Recipe

- **On the path of the reward** (no division, as in Dr. GRPO, or when division reaches exploration too): equilibria and folds don't move with normalization
- **Bonus inside the reward just above $c_b^*$**: keeps a leading answer at an interior equilibrium (mode share 0.56–0.67 vs predicted 0.71–0.88)
- **Larger coefficients** lower the mode share and pass@1

---

## Conclusion

### Main Takeaways

1. The exploration coefficient has **three regimes** separated by lower and upper boundaries
2. Exploration **on the path of the reward** keeps boundaries invariant to learning rate and optimizer scaling; exploration **off the path** shifts them
3. GRPO's normalization **splits the threshold** of a bonus for rare answers into two (before vs. after division)
4. A bonus placed **before division** prevented collapse near its predicted threshold in both a language model and free-form TTRL, preserving correct answers in samples

### Limitations

- Theory treats **one state at a time**; questions shared parameters and committed together
- Thresholds for majority reward and bonus hold on **policies with one leading answer and equal others**
- Normalization is a **mean-field model**; only folds are certified on ordered bins
- Free-form TTRL used one model family, 128 selected problems, $G = 8$, 200 steps

### Future Directions

Combine exploration with **symmetry-breaking signals** (e.g., verifiers) to correct wrong majorities. The authors note that exploration alone cannot repair a collapsed model or correct a wrong plurality—it must be paired with signals that distinguish correct from incorrect answers.

---

**Impact Statement:** The results help practitioners keep more correct answers among samples than training without exploration, but do not show how to correct a wrong majority or repair a collapsed model.

---

_Markdown view of https://picx.dev/p/TwJHGw, served by PicX — AI-generated visual whiteboard summaries of research papers._