Summary of "Believe Your Model: Distribution-Guided Confidence Calibration"

Summary (Overview)

• Proposes DistriVoting: A novel test-time scaling (TTS) method that uses a Gaussian Mixture Model (GMM) to decompose the confidence distribution of generated reasoning trajectories into positive (correct) and negative (incorrect) components, and then applies a two-step filtering process to improve answer selection.
• Introduces SelfStepConf (SSC): A method to dynamically adjust the inference process by monitoring step-level confidence in real-time and triggering self-reflection when confidence drops, which increases the separation between confidence distributions of correct and incorrect answers.
• Theoretically and Empirically Validates: Provides theorems proving that increased separation ($\delta = \mu_{pos} - \mu_{neg}$) between distributions improves voting accuracy, and shows through extensive experiments that both DistriVoting and SSC consistently boost performance.
• Demonstrates Superior Performance: Experiments across 16 models (DeepSeek, Qwen series) and 5 reasoning benchmarks (HMMT2025, GPQA-D, AIME, BRUMO2025) show that DistriVoting outperforms state-of-the-art TTS methods like Self-Consistency, Best-of-N, and MoB.

Introduction and Theoretical Foundation

Background: Large Reasoning Models (LRMs) benefit from test-time scaling (TTS) techniques like Chain-of-Thought and repeated sampling, which generate multiple candidate answers. A core challenge is selecting the best answer without external labels or reward signals.

Motivation: Prior work shows that internal model confidence scores correlate with answer correctness and follow distinct statistical distributions for correct vs. incorrect trajectories. However, this distributional prior has not been fully exploited to guide answer selection. The overlap between high-confidence incorrect samples and low-confidence correct samples limits the reliability of confidence-based voting.

Theoretical Foundation: The paper models the confidence scores of trajectories as a mixture of two normal distributions:

• Positive distribution: $X_{pos} \sim \mathcal{N}(\mu_{pos}, \sigma_{pos}^2)$
• Negative distribution: $X_{neg} \sim \mathcal{N}(\mu_{neg}, \sigma_{neg}^2)$

Key Theorems:

• Theorem 2.1: Defines an integral ratio function $R(\mu_1, \mu_2)$ and proves it is strictly monotonically increasing with respect to $\delta = \mu_1 - \mu_2$. This implies that a larger separation between distribution means leads to a higher proportion of correct samples above the midpoint threshold.
• Theorem 2.2: Proves that the weighted voting accuracy $P_{vote}(\delta)$ has a lower bound that increases with $\delta$.

Methodology

1. Confidence Calculation

For a trajectory with $N$ tokens, confidence is computed using token negative log-probabilities:

$$C_{traj} = -\frac{1}{N_G \times k} \sum_{i \in G} \sum_{j=1}^{k} \log P_i(j)$$

where $G$ is the subset of tokens (e.g., the final answer step), $N_G$ is the number of tokens in $G$, and $k$ is the number of top probabilities considered.

2. SelfStepConf (SSC)

SSC intervenes during generation to improve trajectory quality.

• Reflection Trigger: Monitors step-level confidence $C_{G_m}$. Triggers self-reflection if the relative change $\Delta_{conf} = C_{G_m} / \tau_{conf}$ falls below a threshold $\delta$ and confidence is declining. The adaptive threshold $\tau_{conf}$ is updated via EMA when no reflection is triggered.
• Reflection Injection: When triggered, it forcibly swaps the highest-probability token's logit with that of a predefined "reflection token" to guide the generation, sampling with temperature=0 for a fixed number of steps. This does not affect the confidence calculation of the step.

3. DistriVoting

A two-stage filtering process applied after trajectory generation.

• GMM Modeling: Models the confidence distribution of all trajectories for a question as a Gaussian Mixture Model:

$$p(x) = \pi_1 \mathcal{N}(x | \mu_1, \sigma_1^2) + \pi_2 \mathcal{N}(x | \mu_2, \sigma_2^2)$$

The component with the higher mean is mapped to the positive distribution.

• GMM Filter: Selects trajectories belonging to the positive distribution $V_{pos}$ as the initial candidate voting pool.
• Reject Filter: Uses the trajectories from the negative distribution $V_{neg}$ to vote for a likely incorrect answer $A_{neg}$. It then filters out from $V_{pos}$ any trajectory whose answer matches $A_{neg}$, producing a refined pool $\hat{V}_{pos}$.
• Hierarchical Voting (HierVoting): The final voting on $\hat{V}_{pos}$ uses a hierarchical approach:
  1. Confidence scores are split into $N_C$ intervals.
  2. Weighted majority voting $f_{WMaj}$ is applied within each interval to get a sub-answer.
  3. A final weighted majority vote aggregates the sub-answers. The weighted majority voting function is:
  $$f_{WMaj}(V, C) = \arg \max_{ans} \sum_{traj \in V} \mathbb{I}(A_{traj} = ans) \cdot C_{traj}$$

Empirical Validation / Results

Main Results: Experiments on DeepSeek-R1-8B and Qwen3-32B across five benchmarks show consistent improvements.

Table 1: Main results of SelfStepConf (SSC) and DistriVoting across benchmarks. (Budget=128, 64 repetitions. SC=Self-Consistency, WSC=Weighted SC, BoN=Best of N, DIS=DistriVoting)

Key Findings:

1. GMM vs. Fixed Filter: GMM Filter consistently outperforms a fixed Top-50% filter (e.g., WSC-GMM 76.64% vs. WSC-Top50 74.75% for DeepSeek).
2. DistriVoting vs. Baselines: DistriVoting (DIS) outperforms weighted voting (WSC) under both filter settings.
3. SSC Contribution: Applying SSC (marked *) further improves results for both WSC and DIS, validating its role in enhancing distribution separation.

Ablation Studies:

• Clustering Methods: GMM outperforms K-Means and MeanShift in prediction accuracy and voting performance while being computationally efficient.
• Budget Sensitivity: DistriVoting shows significant advantages over conventional methods when the sample budget ($B$) is ≥ 16, with performance scaling with larger budgets.
• Filter Effectiveness: Analysis shows both Acc and Weighted Acc (WAcc) increase progressively through the GMM Filter and Reject Filter stages.

Table 4: Effectiveness Analysis of DistriVoting (DeepSeek-R1-8B)

SSC Analysis:

• Distribution Separation: SSC increases the mean difference $\delta$ between correct and incorrect confidence distributions (e.g., from 3.182 to 5.043 on HMMT2025).
• Sampling Efficiency: SSC improves pass@1 performance significantly, especially for mid-tier models, indicating improved sampling efficiency without expanding fundamental reasoning limits.
• Inference Dynamics: SSC interventions lead to higher confidence trajectories and often shorter responses without increasing overall time complexity significantly (~2.3% overhead).

Theoretical and Practical Implications

Theoretical Implications:

• Formalizes the relationship between confidence distribution separation and voting accuracy, providing a theoretical justification for methods that aim to increase $\delta$.
• Demonstrates that clustering confidence scores based on a bimodal normal assumption (via GMM) is an effective way to leverage distributional priors for answer selection.

Practical Implications:

• Efficiency: DistriVoting and SSC rely solely on internal model signals, avoiding the cost of training or querying external reward models.
• Effectiveness: Provides a robust, adaptive filtering mechanism that outperforms fixed-threshold methods, which require per-benchmark tuning.
• Generality: The methods are model-agnostic and show consistent gains across a wide range of model sizes and architectures on diverse reasoning benchmarks.

Conclusion

This paper addresses the problem of "confidently wrong" predictions in test-time scaling by proposing DistriVoting, a distribution-guided voting method, and SelfStepConf, a step-level confidence intervention technique. DistriVoting leverages a GMM to model confidence distributions and uses a two-stage filter to refine the candidate answer pool, while SSC actively improves the quality and separability of generated trajectories. Both theoretical analysis and extensive experiments across 16 models and 5 benchmarks demonstrate significant and consistent performance improvements over state-of-the-art TTS methods. The work advances the field by showing how to better utilize internal model signals for reliable and efficient answer aggregation.