Summary of "Efficiently Estimating Optimal Hyperparameter Scaling Laws through Power-Law Entropy Search"

Summary (Overview)

• Core contribution: Introduces Power-Law Entropy Search (PLES), a cost-aware acquisition function built on multi-fidelity Bayesian optimization that efficiently estimates optimal hyperparameter scaling laws for large language model (LLM) training through adaptive experimentation.
• Key innovation: PLES searches for candidates that reduce the overall uncertainty of a scaling law estimate rather than optimizing a single objective function, naturally favoring informative small-scale experiments.
• Efficiency gains: PLES converges to accurate optimal hyperparameter scaling laws using less than one-tenth of the computational budget required by conventional grid search and other baselines.
• Practical benefits: Provides a natural stopping criterion based on posterior uncertainty at the target scale, and can recover scaling laws for multiple hyperparameters simultaneously even when designed for only one.
• Validation: Evaluated across synthetic benchmarks, surrogate models fitted to real LLM training data, and actual LLM pre-training runs (Llama-3).

Introduction and Theoretical Foundation

Background and Motivation

Optimal hyperparameter scaling laws describe how the best hyperparameters for LLM training (e.g., learning rate, batch size, weight decay) change with model and data scale. These laws allow practitioners to predict optimal configurations at production scales without expensive large-scale tuning.

The conventional approach—exhaustive grid search—is prohibitively expensive. For instance, Li et al. (2025) "trained over 3,700 LLMs from scratch across 100 trillion tokens, consuming nearly one million NVIDIA H800 GPU hours" to establish scaling laws for learning rate and batch size. Furthermore, scaling laws become outdated with new architectures, requiring repeated expensive re-estimation.

Theoretical Foundation

The paper builds on several established frameworks:

Gaussian Processes (GP): A GP defines a distribution over functions, $f \sim \mathcal{GP}(\mu, \kappa)$, fully specified by a prior mean function $\mu(x)$ and covariance kernel $\kappa(\mathbf{x}, \mathbf{x}')$. The posterior mean and variance given observations $\mathcal{D}_t = \{(\mathbf{x}_\tau, y_\tau)\}_{\tau=1}^{t}$ are:

$$\mu_t(\mathbf{x}') \triangleq \boldsymbol{\kappa}_t^{\top}(\mathbf{x}')(K_t + \zeta I)^{-1}\mathbf{y}_t \tag{1}$$ $$\sigma_t^{2}(\mathbf{x}') \triangleq \kappa(\mathbf{x}', \mathbf{x}') - \boldsymbol{\kappa}_t^{\top}(\mathbf{x}')(K_t + \zeta I)^{-1}\boldsymbol{\kappa}_t(\mathbf{x}') \tag{2}$$

Bayesian Experimental Design (BED): Provides a principled framework for selecting experiments that maximize information gain about a latent quantity of interest.

Optimal hyperparameter scaling law: A parametric function $\Pi$ mapping scale parameters (model size $N$, data size $D$) to optimal hyperparameters $\theta^*$:

$$\theta^* = \Pi(N, D)$$

The paper assumes a power-law functional form:

$$\theta^* = c N^{\alpha} D^{\beta} \tag{3}$$

The goal is to estimate the power-law coefficients $c, \alpha, \beta$ efficiently through small-scale experimentation.

Methodology

1. Estimating Power-Law Coefficients from a GP

The power-law in Equation (3) is transformed into a linear form via logarithms:

$$\log \theta^* = \log c + \alpha \log N + \beta \log D \tag{4}$$

This is treated as a Bayesian linear regression problem learning coefficients $\mathbf{w} = (\log c, \alpha, \beta)^{\top}$.

Finding $\theta^*$ for a fixed scale: Thompson sampling over the current GP estimates the expectation of the optimal hyperparameter at scale $i$:

$$\theta_i^* = \mathbb{E}_{f \sim \mathcal{GP}}[\arg\min_{\theta} f(N_i, D_i, \theta)] \approx \frac{1}{K}\sum_{k=1}^{K} \arg\min_{\theta} f_k(N_i, D_i, \theta) \tag{5}$$

Bayesian linear regression: Aggregating across $m$ scales, the target vector $\mathbf{y}$, design matrix $\mathbf{X}$, and noise covariance $\boldsymbol{\Lambda}$ are:

$$\mathbf{y} = [\log \theta_1^*, \dots, \log \theta_m]^{\top}, \quad \mathbf{X} = \begin{bmatrix} 1 & \log N_1 & \log D_1 \\ & \vdots \\ 1 & \log N_m & \log D_m \end{bmatrix}, \quad \boldsymbol{\Lambda} = \mathrm{diag}(\sigma^2(\log \theta_1^*), \ldots, \sigma^2(\log \theta_m^*)) \tag{6}$$

With Gaussian prior $\mathbf{w} \sim \mathcal{N}(\mathbf{0}, \boldsymbol{\Sigma}_0)$, the posterior is $p(\mathbf{w}|\mathbf{y}, \mathbf{X}) = \mathcal{N}(\mu_{\mathbf{w}}, \Sigma_{\mathbf{w}})$:

$$\boldsymbol{\Sigma}_{\mathbf{w}} = \left(\boldsymbol{\Sigma}_0^{-1} + \mathbf{X}^{\top}\boldsymbol{\Lambda}^{-1}\mathbf{X}\right)^{-1} \tag{7}$$ $$\boldsymbol{\mu}_{\mathbf{w}} = \boldsymbol{\Sigma}_{\mathbf{w}} \mathbf{X}^{\top} \boldsymbol{\Lambda}^{-1} \mathbf{y} \tag{8}$$

The differential entropy $\log\det\Sigma_{\mathbf{w}}(\mathcal{GP}(\mathcal{D}))$ quantifies uncertainty in the power-law coefficients.

2. Power-Law Entropy Search (PLES)

The acquisition function selects candidates maximizing uncertainty reduction per unit computational cost:

$$\mathbf{PLES}(x) = \arg\max_{x} \frac{\log\det \boldsymbol{\Sigma}_{\mathbf{w}}(\mathcal{GP}(\mathcal{D})) - \log\det \boldsymbol{\Sigma}_{\mathbf{w}}(\mathcal{GP}(\mathcal{D} \cup x))}{(ND)^d} \tag{9}$$

where $d$ is a cost-cooling factor discouraging large-scale experiments. Setting $d=1$ searches for maximum information gain per unit cost; larger $d$ favors smaller scales. Fantasy observations are drawn from the current GP to compute the updated covariance.

3. Extension to Multiple Hyperparameters

For multiple hyperparameters (e.g., learning rate and batch size), independent power-laws with different coefficients are maintained. The acquisition function is modified to reduce a weighted-sum of differential entropies across multiple scaling laws. Interestingly, experiments show that explicitly modeling multiple scaling laws may be unnecessary—optimizing for one hyperparameter's scaling law can simultaneously recover accurate scaling laws for others.

4. Stopping Criterion

The posterior uncertainty at the held-out scale, $\mathbf{x}_{\max}^{\top}\Sigma_{\mathbf{w}}\mathbf{x}_{\max}$ where $\mathbf{x}_{\max} = (1, \log N_{\max}, \log D_{\max})^{\top}$, provides a natural stopping criterion. When this value is small, the power-law estimate has converged.

Empirical Validation / Results

Experimental Settings

1. Synthetic function: A known function describing how model loss varies with learning rate, batch size, model scale $N$, and data scale $D$.
2. Surrogate GP models: Fitted to real LLM training runs from (Li et al., 2025; Lin et al., 2026), serving as a continuous oracle.
3. Real LLM training runs: Llama-8B models with $N \in [9.4M, 600M]$, $D \in [199.8M, 12B]$, learning rate $\in [0.0001, 0.1]$, trained on a subset of Llama training data.

Baselines Compared

Key Results

Main findings (Figure 4):

• PLES consistently achieves smaller percentage error in estimated optimal hyperparameters at the held-out scale
• PLES converges much faster, reaching the best attainable model loss in less than one-tenth the computational budget of baselines
• Results hold across all three settings (synthetic, surrogate, real LLM)

Scaling law uncertainty analysis (Figure 5):

• Uncertainty decreases with more experimentation budget
• The uncertainty at the held-out scale serves as an effective stopping criterion—at iteration 6, the estimate has already converged to the ground-truth scaling law

Multiple hyperparameters (Figure 6):

• PLES recovers the batch size scaling law despite being designed only to minimize uncertainty about the learning rate scaling law
• Explanation: PLES uncovers regions containing optima for multiple hyperparameters simultaneously

Candidate scale analysis (Figure 7):

• Most candidate scales proposed by PLES are less than 2% of the computational cost of the held-out scale
• This makes the total experimental cost significantly smaller than direct hyperparameter optimization at the largest scale

Computational Complexity

Per-iteration complexity is $\mathcal{O}(C \cdot K \cdot Q)$, where:

• $C$ = number of candidate configurations
• $K$ = number of Thompson samples
• $Q$ = cost of solving the inner $\arg\min_\theta$ problem

Using batched pathwise sampling (BoTorch), the Thompson sampling loop and inner minimizations are fully parallelizable, making overhead negligible compared to training cost savings.

Theoretical and Practical Implications

Theoretical Contributions

1. Novel acquisition function: PLES reframes the problem from single-objective optimization to uncertainty reduction of a functional estimate (the scaling law), which is a fundamentally different paradigm in Bayesian optimization.
2. Convergence guarantee: Thompson sampling from the GP asymptotically recovers the true solution, providing theoretical grounding for the approach.
3. Cost-aware design: The cost-cooling factor $d$ provides a principled way to balance information gain against experimental cost, naturally favoring informative small-scale experiments.

Practical Implications

1. Massive cost reduction: Reducing computational requirements by 10× for scaling law estimation makes it feasible to re-estimate scaling laws as architectures evolve.
2. Accessibility: Smaller labs with limited compute can now estimate scaling laws that were previously only feasible for large organizations.
3. Automatic stopping: The uncertainty-based stopping criterion removes guesswork about when enough experiments have been performed.
4. Multi-hyperparameter recovery: The surprising result that optimizing for one hyperparameter's scaling law recovers others suggests potential further simplifications in practice.

Conclusion

PLES offers a practical and principled framework for scaling law estimation that substantially reduces the computational cost of developing LLMs. The key takeaways are:

1. Efficiency: Less than one-tenth the computational budget of conventional approaches while achieving comparable or better accuracy.
2. Principled uncertainty quantification: Provides confidence intervals on scaling law estimates and a natural stopping criterion.
3. Scalability: Extends naturally to multiple hyperparameters and is computationally efficient in its own overhead.

Limitations and Future Directions

• Assumption of power-law form: The method assumes optimal hyperparameters follow a power-law (Equation 3). While well-supported empirically for studied hyperparameters and architectures, this may not hold universally. Deviations would produce biased coefficient estimates.
• Future work: Extending PLES to accommodate more flexible functional representations while retaining tractable uncertainty quantification is a promising direction.

As model architectures and training recipes continue to evolve—requiring repeated re-estimation of scaling laws—methods like PLES that minimize experimental burden become increasingly valuable.