Summary (Overview)

• Core Problem: Scaling law construction requires training an exhaustive grid of configurations (model sizes, token budgets, hyperparameters), which is computationally prohibitive. Only the best-loss frontier (the loss envelope) is actually needed for fitting, so most trained configurations are discarded.
• Key Proposal: The authors formalize scaling law construction as a Bayesian Optimization (BO) problem and introduce compute slicing—progressively expanding the compute window during acquisition—which mirrors the compute-ordered evaluation of configurations in practice.
• Surrogate Fantasization: The framework augments observed configurations with surrogate-predicted (fantasized) evaluations to reconstruct the broader experimental grid, enabling accurate scaling law fitting without training every configuration.
• Main Result: The proposed framework recovers scaling law fits comparable to the full dense grid at 10–100× less compute cost, with predicted losses remaining within 1.6% of the full-data fit even at 6–7 orders of magnitude beyond the observed grid.
• Contribution: Introduction of a multi-metric evaluation suite (parameter regret, held-out MSE, envelope coverage, extrapolation error) that jointly captures the nuances of scaling law fit quality under constrained compute budgets.

---

Introduction and Theoretical Foundation

Scaling laws guide design choices for training large foundation models by fitting power laws between loss (or hyperparameters) and compute budget. The standard derivation procedure trains a dense grid of configurations across model sizes $N$, token budgets $D$, and hyperparameters $\lambda \in \Lambda$, with compute defined as $C = g(N, \bar{D})$.

The key theoretical insight is that fitting a parametric scaling law only requires the best-loss frontier—the minimum loss achieved at each compute scale:

$$L(C) = \min_{N, D, \lambda} L(N, D, \lambda) \quad \text{s.t.} \quad g(N, \bar{D}) = C$$

This observation creates two clear sources of efficiency gains:

1. Avoiding low-performing configurations: Evaluating sub-optimal hyperparameter settings at each compute budget is wasteful.
2. Approximating the dense envelope: A surrogate model can approximate the loss landscape, providing early signal on scaling law fits without exhaustively evaluating the full grid $\mathcal{G} = \mathcal{N} \times \mathcal{D} \times \Lambda$.

The paper contrasts this with classical HPO (which returns a single best-loss configuration) and multi-fidelity HPO (which focuses on early stopping of individual runs). Scaling law construction is fundamentally different: it requires the entire loss envelope across compute scales, not a single optimum.

Figure 1 illustrates the core motivation: fitting $L = E + A C^\alpha$ to different 4-point subsets of a 14-point envelope can produce drastically different fits—one closely recovering the full-data fit, another deviating significantly. This motivates the problem of intelligently selecting a small subset of configurations that reliably recovers the full scaling law.

---

Methodology

Problem Formulation

The framework treats scaling law data collection as an iterative BO loop. Let:

• $\mathcal{W}_j = \bar{\rho}[C_1, C_j]$ denote the compute window observed up to compute scale $C_j$
• $\mathcal{O}_t$ denote observations collected by iteration $t$
• $\mathcal{M}_t$ denote the surrogate model fitted to $\mathcal{O}_t$

Compute Slicing

At each iteration, the surrogate is fitted on all observations within the current window $\mathcal{W}_j$, while the acquisition search is performed over the expanded window $\mathcal{W}_{j+1}$ (which includes the immediate next compute slice). This deliberately myopic approach:

• Restricts acquisition to conservative, lower-compute investments initially
• Retains opportunity to improve upon losses observed at lower-compute slices
• Mirrors the compute-ordered evaluation of configurations in practice

The compute slice size is determined by the discrete choices of $N$ and $D$ in the study design (not tuned).

Surrogate Fantasization

At each iteration, the surrogate $\mathcal{M}_t$ is used to fantasize (predict) outputs at unobserved parts of the grid $\mathcal{G}$. The scaling law is then fitted on a proxy dense grid containing:

• Observed losses where available
• Surrogate predictions elsewhere

As $\mathcal{M}_t$ improves with more observations, the resulting fit approaches the full-data fit at a fraction of the total cost.

Experimental Setup

The authors simulate scaling law construction over predefined configuration grids from existing scaling experiments (e.g., OELLM-English). They evaluate 4 settings:

• Full: Standard BO viewing the entire search space $(N, D, \Lambda)$ uniformly
• Window: Compute-sliced acquisition with growing compute window
• With/without fantasization of pending configurations

The baseline uses standard BoTorch defaults: Gaussian Process surrogate with constant mean, Matérn-5/2 kernel, and Lower Confidence Bound (LCB) acquisition with $\kappa = 2$.

Evaluation Metrics

The paper argues that existing metrics (in-sample residual fit or held-out extrapolation error alone) are insufficient, since improvements in one don't translate predictably to the other. The proposed multi-metric suite includes:

1. Regret: Absolute difference of estimated scaling law parameters to ground truth
2. MSE: Mean-squared-error of predictions on held-out points to empirical loss
3. Envelope Coverage: Percentage coverage of the ground-truth loss-envelope set
4. Extrapolation Error: Relative % difference in loss prediction over large extrapolation ranges

---

Empirical Validation / Results

Main Results (Figure 2)

Results on $L = E + A C^\alpha$ comparing Full vs. Window acquisition strategies:

• Parameter Recovery: Both strategies recover coefficients well before the pool is exhausted; a small fraction of total acquisition compute yields coefficient estimates close to the full-data reference.
• Speedup: Window + fantasization achieves 10× speedup over Full + fantasization, and up to 100× speedup over Full + observed-only.
• Envelope Coverage: Window discovers evaluations on the loss envelope far earlier than Full, confirming that compute slicing produces the intended behavior.

Extrapolation Performance (Table 1)

Extending window + fantasization results to much larger extrapolation lengths (6–7 orders of magnitude beyond the grid):

Entries are relative % difference in predicted loss against the full-data fit (sans held-out), at different fractions of the compute needed to exhaust the grid. S.D. over 10 seeds.

Key observations:

• At just 1% of total budget, predicted losses stay within 1.6% of the full-data fit even at $10^{29}$ FLOPs (6–7 orders of magnitude higher than the grid)
• Spending more budget provides only marginal relative improvements
• The recovered parameters' predicted losses remain highly accurate across extreme extrapolation ranges

Additional Validations

The paper reports ablations across additional datasets, parametric forms, and search methods (detailed in Appendices C and D), confirming that the framework:

• Performs well across different settings and parametric forms
• Seamlessly integrates with standard BO formulations

---

Theoretical and Practical Implications

Theoretical Significance

1. New Formulation: Casting scaling law construction as an HPO problem bridges two previously distinct research areas—scaling law fitting and Bayesian optimization—providing a principled framework for efficient data collection.

2. Compute Slicing as a Principle: The demonstration that progressively expanding the compute window during acquisition improves recovery efficiency suggests that myopic, compute-ordered search is fundamentally better suited for envelope discovery than uniform global search.

3. Fantasization for Dense Reconstruction: Using surrogate predictions to reconstruct the dense grid enables scaling law fitting on simulated data, showing that the loss envelope can be approximated without dense measurement.

Practical Implications

• Compute Savings: 10–100× reduction in compute costs for scaling law construction makes scaling law discovery feasible for smaller research groups and for more frequent updates as new data becomes available.
• Extrapolation Reliability: The ability to predict losses within 1.6% at 6–7 orders of magnitude beyond the observed grid provides confidence in using sparse data collection for large-scale extrapolation.
• Framework Agnosticism: The compute slicing and fantasization mechanisms are compatible with any BO surrogate and acquisition function, making the framework easily adoptable.

---

Conclusion

The paper casts scaling law construction as a hyperparameter optimization problem and demonstrates that:

1. Compute slicing (progressively growing the compute window during acquisition) combined with surrogate fantasization (evaluating scaling law fits on a simulated dense grid proxy) allows BO to achieve highly accurate fits at 10–100× less compute than exhaustive grid evaluation.
2. The recovered parameters' predicted losses remain within 1.6% of the full-data fit even at 6–7 orders of magnitude beyond the observed grid.
3. A multi-metric evaluation suite is necessary to properly capture the nuances of scaling law fit quality.

Future Directions

• Specialized kernels and acquisition functions tailored for scaling law construction (e.g., incorporating scaling law fit variance into acquisition)
• Validation on novel, large-scale data collection runs for scaling law discovery
• The ultimate stress-test will be applying this framework to new scaling law discovery rather than retrospective simulation on existing datasets