# Amortizing Scaling Law Construction Costs · Pith Review

> Bayesian optimization with compute slicing and surrogate fantasization recovers scaling law fits at 10-100x less compute, with extrapolation error under 1.6%.

- **Source:** [arXiv](https://arxiv.org/abs/2609.05016)
- **Published:** 2026-09-26
- **Permalink:** https://picx.dev/p/15UwvD
- **Whiteboard:** https://picx.dev/p/15UwvD/image

## Summary

## Summary (Overview)

- **Core Problem**: Scaling law construction requires training an exhaustive grid of configurations (model sizes, token budgets, hyperparameters), which is computationally prohibitive. Only the best-loss frontier (the loss envelope) is actually needed for fitting, so most trained configurations are discarded.
- **Key Proposal**: The authors formalize scaling law construction as a Bayesian Optimization (BO) problem and introduce **compute slicing**—progressively expanding the compute window during acquisition—which mirrors the compute-ordered evaluation of configurations in practice.
- **Surrogate Fantasization**: The framework augments observed configurations with surrogate-predicted (fantasized) evaluations to reconstruct the broader experimental grid, enabling accurate scaling law fitting without training every configuration.
- **Main Result**: The proposed framework recovers scaling law fits comparable to the full dense grid at **10–100× less compute cost**, with predicted losses remaining within 1.6% of the full-data fit even at 6–7 orders of magnitude beyond the observed grid.
- **Contribution**: Introduction of a multi-metric evaluation suite (parameter regret, held-out MSE, envelope coverage, extrapolation error) that jointly captures the nuances of scaling law fit quality under constrained compute budgets.

---

## Introduction and Theoretical Foundation

Scaling laws guide design choices for training large foundation models by fitting power laws between loss (or hyperparameters) and compute budget. The standard derivation procedure trains a dense grid of configurations across model sizes $N$, token budgets $D$, and hyperparameters $\lambda \in \Lambda$, with compute defined as $C = g(N, \bar{D})$.

The key theoretical insight is that fitting a parametric scaling law only requires the **best-loss frontier**—the minimum loss achieved at each compute scale:

$$L(C) = \min_{N, D, \lambda} L(N, D, \lambda) \quad \text{s.t.} \quad g(N, \bar{D}) = C$$

This observation creates two clear sources of efficiency gains:
1. **Avoiding low-performing configurations**: Evaluating sub-optimal hyperparameter settings at each compute budget is wasteful.
2. **Approximating the dense envelope**: A surrogate model can approximate the loss landscape, providing early signal on scaling law fits without exhaustively evaluating the full grid $\mathcal{G} = \mathcal{N} \times \mathcal{D} \times \Lambda$.

The paper contrasts this with classical HPO (which returns a single best-loss configuration) and multi-fidelity HPO (which focuses on early stopping of individual runs). Scaling law construction is fundamentally different: it requires the **entire loss envelope** across compute scales, not a single optimum.

**Figure 1** illustrates the core motivation: fitting $L = E + A C^\alpha$ to different 4-point subsets of a 14-point envelope can produce drastically different fits—one closely recovering the full-data fit, another deviating significantly. This motivates the problem of intelligently selecting a small subset of configurations that reliably recovers the full scaling law.

---

## Methodology

### Problem Formulation

The framework treats scaling law data collection as an iterative BO loop. Let:
- $\mathcal{W}_j = \bar{\rho}[C_1, C_j]$ denote the compute window observed up to compute scale $C_j$
- $\mathcal{O}_t$ denote observations collected by iteration $t$
- $\mathcal{M}_t$ denote the surrogate model fitted to $\mathcal{O}_t$

### Compute Slicing

At each iteration, the surrogate is fitted on all observations within the current window $\mathcal{W}_j$, while the acquisition search is performed over the **expanded window** $\mathcal{W}_{j+1}$ (which includes the immediate next compute slice). This deliberately myopic approach:

- Restricts acquisition to conservative, lower-compute investments initially
- Retains opportunity to improve upon losses observed at lower-compute slices
- Mirrors the compute-ordered evaluation of configurations in practice

The compute slice size is determined by the discrete choices of $N$ and $D$ in the study design (not tuned).

### Surrogate Fantasization

At each iteration, the surrogate $\mathcal{M}_t$ is used to **fantasize** (predict) outputs at unobserved parts of the grid $\mathcal{G}$. The scaling law is then fitted on a **proxy dense grid** containing:
- Observed losses where available
- Surrogate predictions elsewhere

As $\mathcal{M}_t$ improves with more observations, the resulting fit approaches the full-data fit at a fraction of the total cost.

### Experimental Setup

The authors simulate scaling law construction over predefined configuration grids from existing scaling experiments (e.g., OELLM-English). They evaluate 4 settings:
- **Full**: Standard BO viewing the entire search space $(N, D, \Lambda)$ uniformly
- **Window**: Compute-sliced acquisition with growing compute window
- With/without fantasization of pending configurations

The baseline uses standard BoTorch defaults: Gaussian Process surrogate with constant mean, Matérn-5/2 kernel, and Lower Confidence Bound (LCB) acquisition with $\kappa = 2$.

### Evaluation Metrics

The paper argues that existing metrics (in-sample residual fit or held-out extrapolation error alone) are insufficient, since improvements in one don't translate predictably to the other. The proposed multi-metric suite includes:

1. **Regret**: Absolute difference of estimated scaling law parameters to ground truth
2. **MSE**: Mean-squared-error of predictions on held-out points to empirical loss
3. **Envelope Coverage**: Percentage coverage of the ground-truth loss-envelope set
4. **Extrapolation Error**: Relative % difference in loss prediction over large extrapolation ranges

---

## Empirical Validation / Results

### Main Results (Figure 2)

Results on $L = E + A C^\alpha$ comparing Full vs. Window acquisition strategies:

- **Parameter Recovery**: Both strategies recover coefficients well before the pool is exhausted; a small fraction of total acquisition compute yields coefficient estimates close to the full-data reference.
- **Speedup**: Window + fantasization achieves **10× speedup** over Full + fantasization, and up to **100× speedup** over Full + observed-only.
- **Envelope Coverage**: Window discovers evaluations on the loss envelope far earlier than Full, confirming that compute slicing produces the intended behavior.

### Extrapolation Performance (Table 1)

Extending window + fantasization results to much larger extrapolation lengths (6–7 orders of magnitude beyond the grid):

| Compute C (in FLOPs): | $10^{25}$ | $10^{27}$ | $10^{29}$ |
|---|---|---|---|
| **1% of Total Budget** | $1.09 \pm 1.84$ | $1.37 \pm 2.35$ | $1.56 \pm 2.67$ |
| **5% of Total Budget** | $0.22 \pm 0.34$ | $0.29 \pm 0.41$ | $0.36 \pm 0.45$ |
| **10% of Total Budget** | $0.04 \pm 0.04$ | $0.05 \pm 0.04$ | $0.06 \pm 0.10$ |

*Entries are relative % difference in predicted loss against the full-data fit (sans held-out), at different fractions of the compute needed to exhaust the grid. S.D. over 10 seeds.*

Key observations:
- At just **1% of total budget**, predicted losses stay within **1.6%** of the full-data fit even at $10^{29}$ FLOPs (6–7 orders of magnitude higher than the grid)
- Spending more budget provides only marginal relative improvements
- The recovered parameters' predicted losses remain highly accurate across extreme extrapolation ranges

### Additional Validations

The paper reports ablations across additional datasets, parametric forms, and search methods (detailed in Appendices C and D), confirming that the framework:
- Performs well across different settings and parametric forms
- Seamlessly integrates with standard BO formulations

---

## Theoretical and Practical Implications

### Theoretical Significance

1. **New Formulation**: Casting scaling law construction as an HPO problem bridges two previously distinct research areas—scaling law fitting and Bayesian optimization—providing a principled framework for efficient data collection.

2. **Compute Slicing as a Principle**: The demonstration that progressively expanding the compute window during acquisition improves recovery efficiency suggests that **myopic, compute-ordered search** is fundamentally better suited for envelope discovery than uniform global search.

3. **Fantasization for Dense Reconstruction**: Using surrogate predictions to reconstruct the dense grid enables scaling law fitting on simulated data, showing that the loss envelope can be approximated without dense measurement.

### Practical Implications

- **Compute Savings**: 10–100× reduction in compute costs for scaling law construction makes scaling law discovery feasible for smaller research groups and for more frequent updates as new data becomes available.
- **Extrapolation Reliability**: The ability to predict losses within 1.6% at 6–7 orders of magnitude beyond the observed grid provides confidence in using sparse data collection for large-scale extrapolation.
- **Framework Agnosticism**: The compute slicing and fantasization mechanisms are compatible with any BO surrogate and acquisition function, making the framework easily adoptable.

---

## Conclusion

The paper casts scaling law construction as a hyperparameter optimization problem and demonstrates that:
1. **Compute slicing** (progressively growing the compute window during acquisition) combined with **surrogate fantasization** (evaluating scaling law fits on a simulated dense grid proxy) allows BO to achieve highly accurate fits at 10–100× less compute than exhaustive grid evaluation.
2. The recovered parameters' predicted losses remain within 1.6% of the full-data fit even at 6–7 orders of magnitude beyond the observed grid.
3. A multi-metric evaluation suite is necessary to properly capture the nuances of scaling law fit quality.

### Future Directions

- **Specialized kernels and acquisition functions** tailored for scaling law construction (e.g., incorporating scaling law fit variance into acquisition)
- **Validation on novel, large-scale data collection runs** for scaling law discovery
- The ultimate stress-test will be applying this framework to new scaling law discovery rather than retrospective simulation on existing datasets

---

_Markdown view of https://picx.dev/p/15UwvD, served by PicX — AI-generated visual whiteboard summaries of research papers._