# Beyond the Log-Linear Frontier: Pre-Asymptotic Analysis of Finite-Budget Neural Scaling Laws

> Finite-budget neural scaling laws deviate from classical power laws due to spectral and domain heterogeneity, with compute-optimal allocation drifting proportionally to exponent variances.

- **Source:** [arXiv](https://arxiv.org/abs/2611.13534)
- **Published:** 2026-11-14
- **Permalink:** https://picx.dev/p/Mg9g8h
- **Whiteboard:** https://picx.dev/p/Mg9g8h/image

## Summary

# Summary of "Beyond the Log-Linear Frontier: Pre-Asymptotic Analysis of Finite-Budget Neural Scaling Laws"

## Summary (Overview)

- **Core Contribution**: This paper develops a rigorous analytical framework for finite-budget neural scaling laws, moving beyond the classical log-linear frontier that assumes separable power laws in model capacity $N$ and dataset size $D$. The authors show that spectral and domain heterogeneity govern finite-scale departures from classical scaling behavior.

- **Three-Axis Analysis**: The paper provides a unified treatment across three scaling axes: (1) capacity scaling via Mellin–Barnes analysis of covariance resolvents, (2) multi-domain data mixing via KKT water-filling duality, and (3) compute-optimal allocation via exponent-variance identities and Schur-complement reductions.

- **Key Theoretical Results**: The authors derive a three-term capacity expansion with competing spectral and regularization residues, a scalar water-filling characterization of optimal domain mixtures with finite-budget bottleneck bounds, and prove that the optimal compute-allocation exponent drifts in proportion to capacity and data exponent variances.

- **Empirical Validation**: Across 45 operator spectra, Pythia (70M–12B), Chinchilla (245 runs), and RegMix (17 domains), the estimators achieve 5.89% holdout MAPE, cut high-compute allocation regret by up to 76.4%, and reduce cross-scale mixture RMSE by 59.4% with zero selection regret.

- **Practical Implications**: The framework explains why multi-power fitting on narrow pilot grids is ill-conditioned (proposing Aitken differencing as a solution), formalizes mid-training data annealing via sequential sampling laws, and separates genuine frontier drift from frozen-hyperparameter bias.

## Introduction and Theoretical Foundation

### Background and Motivation

Classical neural scaling laws model population excess risk as a separable, scale-invariant power law:

$$L_{\text{classical}}(N, D) - E_{\infty} = A_N N^{-\alpha_N} + A_D D^{-\alpha_D} \implies \eta^{\star} := \frac{d \ln N^{\star}(C)}{d \ln C} = \frac{\alpha_D}{\alpha_N + \alpha_D} \tag{1.1}$$

where $\alpha_N, \alpha_D > 0$ are constant capacity and data exponents, and $C(N, D) = 6ND$ approximates the dense-transformer training compute budget. Because the marginal balance exponent $\eta^{\star}$ is independent of $C$, classical theory predicts log-excess risk decays linearly and compute-optimal allocation follows a straight line in log coordinates.

However, empirical learning curves at finite model sizes systematically bend away from a single power law, and reported compute-optimal allocation exponents vary widely: from $\eta^{\star} = 0.73$ (Kaplan et al., 2020) to $\eta^{\star} = 0.50$ (Hofmann et al., 2022) and $\eta^{\star} \in [0.52, 0.58]$ (Bi et al., 2024).

### Theoretical Foundation

The paper builds on a Hilbert-space resolvent framework. Let $(\Omega, \mathcal{F}, \mathbb{P})$ be a complete probability space with a $K$-domain mixture setting. The reference covariance operator $\mathcal{T}$ is defined via:

$$\mathcal{T} := \int_{\mathcal{X}} (\psi(x) \otimes_{\mathcal{H}} \psi(x)) \mu_{\pi}(dx) \tag{2.3}$$

with spectral decomposition $\mathcal{T} = \sum_{m=1}^{\infty} \lambda_m (e_m \otimes_{\mathcal{H}} e_m)$ and regularized resolvent $\mathcal{R}(\mu) := (\mathcal{T} + \mu\mathcal{I})^{-1}$.

**Key structural assumptions**:
- **Assumption 2.1**: Spectral source decay $c_m = cm^{-r}(1 + bm^{-\delta})$ with rational filter law
$$\mathcal{E}_{\text{spec}}(N) := \sum_{m=1}^{\infty} c_m \psi_q(m/M), \quad \psi_q(u) := (1 + u^{-q})^{-2} \tag{2.17}$$
- **Assumption 2.2**: Separable multi-domain excess risk
$$\mathcal{E}_{\text{mix}}(w; D) := \sum_{k=1}^{K} \pi_k B_k (w_k D)^{-\beta_k} = \sum_{k=1}^{K} c_k (w_k D)^{-\beta_k} \tag{2.19}$$
- **Assumption 2.3**: Regularity of the iso-compute risk surface with slice convexity condition $\mathcal{H}_{\text{iso}}(u,v) > 0$

The local Fisher quadratic approximation (Lemma 2.2) connects Hilbert-space spectral analysis to cross-entropy risk via:
$$\mathcal{I}_F(z) := \nabla^2 A(z) = \text{diag}(p(z)) - p(z)p(z)^{\top}, \quad \mathcal{T}_F := \mathbb{E}_{\mu_\pi}[\Psi(X)^* \mathcal{I}_F(z^{\star}(X)) \Psi(X)]$$

## Methodology

### 1. Finite-Capacity Approximation via Mellin–Barnes Analysis

The paper applies Mellin–Barnes contour integration to the spectral risk series. The Mellin kernel of the rational resolvent filter (Lemma 3.1) is:

$$\Psi_q(s) = \frac{1}{q} \Gamma\left(2 + \frac{s}{q}\right) \Gamma\left(-\frac{s}{q}\right) = -\frac{s+q}{q^2} \frac{\pi}{\sin(\pi s/q)} \tag{3.2}$$

**Theorem 3.1** establishes the three-term residue expansion:
$$\mathcal{E}_{\text{spec}}(N) = A_0 N^{-\alpha_{\infty}} + A_g N^{-(\alpha_{\infty}+\Delta_g)} + A_f N^{-(\alpha_{\infty}+\Delta_f)} + R_\rho(N^\kappa) \tag{3.8}$$

where the exponents are:
$$\alpha_{\infty} := \kappa(r-1), \quad \Delta_g := \kappa\delta, \quad \Delta_f := \kappa(2q-r+1), \quad \varrho := \frac{\Delta_f}{\Delta_g} = \frac{2q-r+1}{\delta} \tag{3.1}$$

The key insight is that $A_0 > 0$, $A_g \geq 0$, and $A_f < 0$ (since $\zeta(z) < 0$ on $(-2, 1)$), so competing spectral and regularization modes produce finite-scale curvature with a normalized master slope profile:
$$F_\varrho(x) := \frac{\varrho x^{-1} - x^{-\varrho}}{\varrho - 1} \tag{3.13}$$

### 2. Multi-Domain Mixing via KKT Water-Filling

**Proposition 4.1** shows that KKT duality reduces $K$-domain simplex mixing to a one-dimensional water-filling equation:

$$\sum_{k=1}^{K} \left(\frac{c_k \beta_k D^{-\beta_k}}{\nu}\right)^{\frac{1}{\beta_k+1}} = 1 \iff w_k^{\star}(D) = \left(\frac{c_k \beta_k D^{-\beta_k}}{\nu(D)}\right)^{\frac{1}{\beta_k+1}} \tag{4.2}$$

**Corollary 4.1** establishes the invariant powered share-ratio law:
$$\frac{w_k^{\star}(D)^{\beta_k+1}}{w_1^{\star}(D)^{\beta_1+1}} = \frac{c_k \beta_k}{c_1 \beta_1} D^{-(\beta_k-\beta_1)} = \tilde{D}_k^{-(\beta_k-\beta_1)} \tag{4.4}$$

**Theorem 4.1** provides finite-budget bottleneck bounds with fast-domain mass $\varepsilon(D)$ satisfying:
$$\frac{Z(D)}{1+Z(D)} \leq \varepsilon(D) \leq \min\{Z(D), 1\} \tag{4.8}$$

### 3. Sequential Allocation and Simplex Hessian Duality

**Proposition 4.2** derives the instantaneous sampling distribution:
$$q_k^{\star}(D) := \frac{d n_k^{\star}(D)}{dD} = \frac{\frac{w_k^{\star}(D)}{\beta_k+1}}{\sum_{j=1}^{K} \frac{w_j^{\star}(D)}{\beta_j+1}} \tag{4.13}$$

**Proposition 4.3** establishes the simplex Hessian metric duality:
$$\boldsymbol{D}_H(D)^{-1} = \frac{\mathbb{E}_{\mathbb{P}_D^{\star}}[(\beta_I+1)^{-1}]}{\nu(D)} \text{diag}(q^{\star}(D)) \tag{4.16}$$

### 4. Compute-Optimal Allocation and Exponent Drift

**Theorem 5.1** proves the exponent-variance identity for compute-optimal drift:
$$\eta(t) = \frac{\mathbb{E}_{\mathbb{Q}_t}[\beta]}{\mathbb{E}_{\mathbb{P}_t}[\alpha] + \mathbb{E}_{\mathbb{Q}_t}[\beta]}, \quad \mathcal{K}(t) = \frac{\eta(t)\text{Var}_{\mathbb{P}_t}(\alpha)}{\mathbb{E}_{\mathbb{P}_t}[\alpha]} - \frac{(1-\eta(t))\text{Var}_{\mathbb{Q}_t}(\beta)}{\mathbb{E}_{\mathbb{Q}_t}[\beta]} \tag{5.9}$$

**Theorem 5.2** derives the Schur-complement hyperparameter envelope:
$$\nabla_{rr}^2 \mathcal{E}(\boldsymbol{r}) = \boldsymbol{H}_{rr}(\boldsymbol{r}) - \boldsymbol{\Delta}_S(\boldsymbol{r}), \quad \boldsymbol{\Delta}_S(\boldsymbol{r}) := \boldsymbol{H}_{r\xi}(\boldsymbol{r}) \boldsymbol{H}_{\xi\xi}(\boldsymbol{r})^{-1} \boldsymbol{H}_{\xi r}(\boldsymbol{r}) \succeq 0 \tag{5.19}$$

## Empirical Validation / Results

### Controlled Numerical Experiments

- **Capacity axis**: Across 45 spectral configurations, the normalized slope deviation collapses onto the master curve $F_\varrho(x)$ to machine precision ($9.62 \times 10^{-16}$ maximum deviation). The divided-difference representation (Proposition 3.1) reduces median relative error from 26.93% (single power law) to 0.00005% across the 36-cell factorial grid.

- **Pilot conditioning**: First-order Aitken differencing (Corollary 3.2) restores linear $u_J$ conditioning (slope 1) versus quadratic singular-value decay for unreduced Jacobians.

- **Mixing axis**: Water-filling saves 35.2% of total token budget compared to uniform sampling. The powered share ratio collapses onto $\tilde{D}_k^{-1}$ exactly.

### Pythia Benchmark (70M–12B)

- Three-parameter shifted power law cuts held-out LAMBADA prediction error from 21.98% to 5.89% MAPE on Standard and from 22.13% to 10.22% on Deduped.
- Pilot Jacobian condition number at $p=3$ is well-conditioned ($\kappa_2(\tilde{J}_3) = 32.29$), while $p=5$ induces a 3,026× conditioning blowup ($\kappa_2(\tilde{J}_5) = 9.77 \times 10^4$).

### Chinchilla Benchmark (245 runs)

| Estimator | Fit Sample | Fitted $\hat{\eta}$ | Holdout MAPE (L≤3.15) | High-C Regret (Frontier) | High-C Regret (Surface) |
|---|---|---|---|---|---|
| Power law (raw) | N≤4B (221) | 0.6712 | 1.92% | 0.0909 | 0.0289 |
| Power law (trimmed) | N≤4B, L≤3.15 (205) | 0.5569 | 1.34% | 0.0393 | 0.0139 |
| Log-quadratic | N≤4B, L≤3.15 (205) | 0.5288 | 0.79% | 0.0709 | **0.0068** |
| Full-sample reference | All N, L≤3.15 (228) | 0.4839 | — | 0.0349 | 0.0000 |

The log-quadratic surface cuts high-compute allocation regret by 76.4% (Surface) compared to the raw power law.

### RegMix Benchmark (17 domains)

| Predictor Family (1M Fit) | 1M Test RMSE | Δ% | 1M $\rho_S$ | 60M $\rho_S$ | 1B $\rho_S$ | 1B Selection Regret |
|---|---|---|---|---|---|---|
| 17-domain linear Ridge (baseline) | 0.4539 | — | 0.62 | 0.56 | 0.37 | 0.1156 |
| Single-domain power (Theorem 4.1) | 0.2803 | -38.3% | 0.92 | 0.84 | 0.74 | 0.0000 |
| + 1st-order transfer (Corollary 4.2) | 0.1992 | -56.1% | 0.95 | 0.93 | **0.77** | 0.0000 |
| + Diag. log-quad. | 0.1842 | **-59.4%** | 0.96 | 0.94 | 0.76 | 0.0000 |

## Theoretical and Practical Implications

### Theoretical Significance

1. **Pre-asymptotic scaling theory**: The Mellin–Barnes framework provides a rigorous characterization of when and why learning curves depart from power laws, with explicit residue competition between spectral-geometric zeta poles and regularization filter poles.

2. **Exponent drift mechanism**: The exponent-variance identity (Theorem 5.1) shows that compute-optimal allocation drift is driven by the variance of capacity and data exponents, not by cross-derivative coupling alone (Proposition 5.1).

3. **Identifiability**: The paper rigorously characterizes multi-power ill-conditioning (Proposition 3.3) and provides a well-conditioned estimation procedure via Aitken differencing.

### Practical Implications for Pretraining

- **Pilot design**: Reliable extrapolation requires widening the scale ratio $N_J/N_1$ between smallest and largest pilot models, not adding intermediate checkpoints. First-order Aitken differencing across a geometric octave grid restores conditioning.

- **Data mixing**: The water-filling laws formalize mid-training data annealing: fast-saturating domains should be down-weighted by $(\beta_k+1)^{-1}$ as total tokens grow.

- **Hyperparameter calibration**: Frozen learning-rate schedules bias apparent capacity exponents upward via the Schur-complement penalty $\Delta_S(r_0) \succeq 0$. Separating intrinsic frontier drift from hyperparameter distortion requires calibrating schedules across pilot scales.

- **Inference efficiency**: Over-training smaller models incurs only mild second-order loss penalties within the global curvature sandwich (5.22), supporting deployment-constrained over-training practices.

## Conclusion

This paper provides a comprehensive pre-asymptotic analysis of finite-budget neural scaling laws, unifying capacity scaling, multi-domain mixing, and compute-optimal allocation within a spectral and convex-analytic framework. The key takeaways are:

1. **Finite-scale departures are governed by spectral and domain heterogeneity**: Three-term Mellin–Barnes expansions capture capacity-side curvature, while KKT water-filling captures data-side saturation effects.

2. **Well-conditioned estimation requires structural priors**: Eliminating the loss floor via first-order Aitken differencing restores linear conditioning on narrow pilot grids.

3. **Exponent drift is a variance phenomenon**: The compute-optimal allocation exponent drifts in proportion to capacity and data exponent variances, with direction determined by which side dominates.

4. **Hyperparameter bias is separable**: Schur-complement reduction separates intrinsic scale drift from frozen-hyperparameter bias, enabling calibrated frontier estimation.

**Future directions** include extending the framework to schedule- and optimizer-dependent scaling analyses, incorporating inference-aware over-training regimes, and developing adaptive pilot design strategies that automatically select model order based on conditioning diagnostics.

---

_Markdown view of https://picx.dev/p/Mg9g8h, served by PicX — AI-generated visual whiteboard summaries of research papers._