Summary (Overview)

• Core Contribution: The paper introduces benchmark hinting, a method that converts a benchmark into a continuous sequence of easier versions indexed by a hint level $H$, enabling accurate downstream scaling laws even when small models score near zero on the original benchmark.
• Key Finding: Accuracy rises smoothly and predictably with hint level $H$, and there exists a simple joint structure where increasing $H$ acts similarly to increasing model scale, captured by a single joint law.
• Forecasting Results: On the Delphi pretraining ladder, a hinted scaling law fit on models trained with at most $10^{20}$ FLOPs forecasts the GSM8K accuracy of a $10^{23}$ FLOP model (25.9%) to within 3 percentage points, whereas a standard scaling law under-forecasts by 13 points.
• Broad Applicability: Across pretraining ladders and post-trained model families, hinted scaling laws reduce median forecasting error on GSM8K, AIME, and HLE from 21 percentage points to 4.3.
• Sample-Efficient Ranking: Hinting widens accuracy gaps between nearly indistinguishable models while preserving order, enabling accurate ranking with a single hinted rollout per problem—equivalent to five unhinted rollouts.

Introduction and Theoretical Foundation

Background and Motivation

Scaling laws (Kaplan et al., 2020; Hoffmann et al., 2022) extrapolate the effect of design choices from small, cheap training runs to expensive target scales. However, these laws typically predict pretraining loss, which is an imperfect proxy for downstream capabilities—lower loss does not reliably mean better task performance (Liu et al., 2023; Springer et al., 2025).

The fundamental problem: benchmark accuracy does not vary predictably with scale like pretraining loss does. Small models score near zero on challenging benchmarks, providing no signal for extrapolation (Wei et al., 2022). Scaling laws fit directly to accuracy are therefore often unreliable (Lourie et al., 2025).

Theoretical Foundation

The key insight is that benchmark accuracy varies predictably with a second variable: how much help the model is given. The paper formalizes this as benchmark hinting:

A benchmark $\boldsymbol{B} = (\mathcal{Q}, \omega)$ consists of a set of problems $\mathcal{Q}$ and a protocol $\omega$ that takes a model $M$ and a problem $x \in \mathcal{Q}$ and samples a solution. A benchmark hinting method maps $B$ and a hint level $H \in [0, \infty)$ to a new benchmark $B_H = (\mathcal{Q}, \omega_H)$ with the same problems but a protocol that makes problems easier as $H$ grows.

Three desiderata for a hinting method:

1. Identity at $H=0$: $\omega_0 = \omega$ and $B_0 = B$
2. Hint scaling: $\text{Acc}(M, H)$ should predictably increase in $H$ according to a simple law
3. Capability scaling: For fixed $H$, $\text{Acc}(M, H)$ should increase predictably with model capability (measured by scale variable $C$ such as $\log_{10}$ FLOPs or parameters)

Methodology

Three Hinting Methods

1. Truncation: Places the first $H$ fraction of the reference solution (cut at a word boundary) in the model's context before it answers. The solution is reworded to not reveal the specific answer string.

2. Masking: Places the reference solution in-context but replaces a randomly chosen $1-H$ fraction of words with [MASK]. At $H = 1$ the solution is intact; at $H = 0$ every word is hidden (and the hint is dropped entirely).

3. Logit Mixing: Decodes the model jointly with a stronger teacher model. At each decoding step, logits are mixed as:

$$s_{H}(x, y_{<t})_{j} = z_{\mathrm{model}}(x, y_{<t})_{j} + H \, z_{\mathrm{teacher}}(x, y_{<t})_{j}, \qquad H \in [0, \infty),\tag{1}$$

with $y_t \sim \text{softmax}(s_H / T)$. At $H = 0$ this is the evaluated model alone; as $H \to \infty$, sampling converges to greedy sampling of the teacher.

The Joint Scaling Law

The paper proposes a floored sigmoid law for joint modeling of scale $C$ and hint level $H$:

$$L(C, H) = \max\Bigl(F, L + (A - L)\, \sigma(\beta_{1} C + \beta_{2} H + \beta_{3})\Bigr), \qquad \sigma(z) = \frac{1}{1 + e^{-z}},\tag{2}$$

where:

• $C$ is either $\log_{10}$ FLOPs or $\log_{10}$ parameter count
• $F$ is the floor (accuracy sits flat at $F$ until an emergence point)
• $L$ is the lower asymptote
• $A$ is the upper asymptote (allows saturation below 1)
• $\beta_1, \beta_2, \beta_3$ are fitted parameters

With $A = 1$ and $L = F = 0$, this reduces to the plain sigmoid. The law implies iso-accuracy contours have constant slope $-\beta_1/\beta_2$.

Experimental Setup

• Pretraining ladder: Delphi (open-weight, 12 compute budgets from $3 \times 10^{18}$ to $10^{23}$ FLOPs, 447M to 25B parameters)
• Post-trained families: Qwen2.5 Instruct (6 models), Qwen3 (6), Qwen3.5 (5), Gemma-3 (5)
• Benchmarks: GSM8K (pretrained), AIME 2025/2026 and HLE subset (post-trained)
• Hint levels: $H \in \{0, 0.1, \ldots, 1\}$ for masking/truncation; $H \in [0, 16]$ for logit mixing
• Rollouts: 10 per problem (AIME/HLE), 32–64 per seed (GSM8K)

Empirical Validation / Results

Fit Quality of the Joint Law

Table 1: Mean absolute error (percentage points) of fitted laws

The joint law uses only 6 parameters per family, versus 42–102 for individual slice fits, yet increases MAE by at most 1.7 percentage points.

Forecasting Performance

Delphi pretraining ladder (GSM8K):

• Hinted scaling law (logit mixing with shared teacher, fit on models ≤ $10^{20}$ FLOPs) predicts the $10^{23}$ FLOP model's accuracy (25.9%) to within 3 percentage points
• Standard scaling law under-forecasts by 13 percentage points (same fit data) and by 12 even with models up to $10^{22}$ FLOPs in the fit
• Logit mixing: mean forecasting error of 3.3 points across all nine training sets; largest error 6.6 points when extrapolating from 0.03% of target compute

Post-trained families: Across all nineteen panels, the hinting ensemble forecast beats the baseline on the largest model in seventeen. Example: on Qwen2.5 with two fitted models, baseline predicts ~100% accuracy at 32B (true: 12%, error 88.2 points); the hinted forecast is off by only 3.0 points.

Key Phenomena

1. Smooth hint scaling: Accuracy rises remarkably smoothly in $H$ for every model, even those scoring near zero at $H=0$. Masking climbs steadily; truncation moves little until most of the solution is shown.

2. Joint structure: Iso-accuracy contours in the scale–hint plane run as roughly parallel diagonals, giving a simple exchange rate between hint level and model scale.

3. Logit mixing ceiling: With a shared teacher, every hint slice rises to a common ceiling set by the teacher. The fitted floor $F = 2.5\%$ sits just above the 0.5–2% unhinted accuracy of models below $10^{22}$ FLOPs.

Sample-Efficient Ranking

The hint-prior ranking combines one hinted rollout per problem with $k-1$ unhinted rollouts:

$$\hat{s}(M, k) = (1 - \alpha_{k})\, z\Big[\widehat{\mathrm{Acc}}_{1}(M, \bar{H})\Big] + \alpha_{k}\, z\Big[\widehat{\mathrm{Acc}}_{k-1}(M, 0)\Big], \qquad \alpha_{k} = \frac{k-1}{k-1+\lambda},\tag{3}$$

with $\lambda = 8$. Key results:

• At one rollout per problem, hint-prior leads avg@k by 0.33 in Kendall's τ on AIME and 0.27 on HLE
• avg@k requires ~5 rollouts (AIME) and ~6 (HLE) to match hint-prior's single-rollout performance
• The log-loss baseline (no rollouts) reaches τ = 0.33 on AIME and τ = −0.05 on HLE (below chance)

Theoretical and Practical Implications

Theoretical Contributions

1. Unified framework: Benchmark hinting provides a unified framework for methods that vary evaluation along a second parameter (number of rollouts, amount of finetuning data) while still predicting accuracy on the original benchmark.

2. Beyond the noise threshold: Hinted scaling laws operate below the noise threshold where standard accuracy scaling fails, extracting signal from models that score zero on the original benchmark.

3. Emergence modeling: The floored sigmoid law explicitly models emergence—accuracy sits flat at a floor $F$ until a critical scale, then follows a sigmoid.

Practical Implications

• Compute-efficient tuning: Developers can predict downstream accuracy of expensive training runs from much cheaper runs (5600× less compute in the Delphi case)
• Safety forecasting: Enables anticipating the scale at which larger models acquire dangerous capabilities
• Sample-efficient evaluation: Reduces rollout budgets for ranking models, particularly valuable for expensive agentic benchmarks (e.g., UK AISI cyber-range evaluations cost ~$3000–$5000 per rollout per model)
• General primitive: Extends naturally to agentic benchmarks (replaying part of a successful trajectory), reinforcement learning, and long-horizon tasks

Conclusion

The paper demonstrates that jointly modeling model capability and hint level enables better predictions of benchmark accuracy than modeling capability alone. Benchmark hinting—turning a benchmark into a continuum of easier versions via truncation, masking, or logit mixing—produces smooth, predictable accuracy curves even for models scoring near zero on the original benchmark.

Main limitations:

• Scarcity of open-source scaling ladders (only Delphi is a true scaling ladder)
• The simple joint law is an imperfect model of the true joint effect
• Requires benchmarks with verified solutions and legible reasoning
• Logit mixing requires a teacher sharing a tokenizer; post-trained family teachers produced poor forecasts

Future directions:

• Sample-efficient ranking of LLM agents on safety-relevant benchmarks
• Hinting for safety and red-teaming evaluations (reward hacking, multi-agent collusion)
• Alternative logit mixing variants not requiring shared tokenizers
• Better teacher instantiations for post-trained families
• Generalizing hinting to agentic benchmarks via trajectory replay