Summary (Overview)

• Core Contribution: The paper introduces the Skaling law, a generalized neural scaling law that couples model size (N) and training data (D) through a single interaction exponent $k$, addressing the structural flaw of additive scaling laws like Chinchilla that assume N and D act independently on loss.
• Empirical Validation: The Skaling law reduces Mean Absolute Percentage Error (MAPE) by 1.5–3× across interpolation and extrapolation regimes compared to Chinchilla, on two large-scale pretraining grids (Farseer with 404 runs, and a new SK-Grid with 134 runs).
• Efficiency Gain: Paired with a sparse "L-shape" grid sampling strategy restricted to low-compute edges, Skaling achieves full-grid predictive accuracy using ~10× less compute than uniform sweeps.
• Theoretical Insight: Through mixed-derivative analysis, the authors demonstrate empirically that $\partial^2 L / \partial N \partial D \neq 0$, providing direct evidence that model size and data interact—a phenomenon additive laws cannot represent.
• Practical Impact: Skaling predicts a compute-dependent optimal token-to-parameter ratio that differs by up to 100× from Chinchilla's prediction at frontier scales, with Skaling's prediction tracking empirical trends more accurately.

---

Introduction and Theoretical Foundation

Background and Motivation

Modern Large Language Models (LLMs) rely on neural scaling laws (Kaplan et al., 2020; Hofmann et al., 2022; Bi et al., 2024) to predict performance of high-compute runs from low-compute experiments. These laws guide critical decisions about pretraining budgets, architectural dimensions, and resource allocation.

However, the widely used Chinchilla law (Hofmann et al., 2022) models the reducible loss as a sum of independent terms:

$$L(N, D) = \frac{A}{N^\alpha} + \frac{B}{D^\beta} + E$$

This additive structure forces the cross-derivative to vanish identically:

$$\frac{\partial^2 L}{\partial N \partial D} = 0$$

The paper demonstrates that this assumption breaks down at the boundaries of the training grid—where N and D are most imbalanced. Figure 1 shows Chinchilla develops large, oppositely-signed errors (a saddle-shaped residual) toward the corners of the (N, D) grid.

The Kaplan Form and Its Limitations

The earlier Kaplan form (Kaplan et al., 2020) took a coupled approach:

$$L(N, D) = \left[\left(N_c/N\right)^{\alpha_N/\alpha_D} + D_c/D\right]^{\alpha_D}$$

Here the outer exponent $\alpha_D$ plays the role of Skaling's $k$, but Kaplan ties the inner terms through the ratio $\alpha_N/\alpha_D$, losing independence of per-axis decay rates.

---

Methodology

The Skaling Law

The Skaling law bridges Chinchilla and Kaplan by retaining Chinchilla's interpretable base terms while introducing a single free outer exponent $k$:

$$L(N, D) = \left(\frac{A}{N^\alpha} + \frac{B}{D^\beta}\right)^k + E \tag{3}$$

Key properties:

• At $k = 1$, Skaling recovers the additive Chinchilla law
• For $k \neq 1$, it reinstates a Kaplan-style coupling with non-zero cross-derivative
• For $k > 0$, the function remains strictly decreasing in both N and D
• It preserves Chinchilla's closed-form compute-optimal allocation formula

Derivative Diagnostics

The authors probe the loss surface using Moving Least Squares (MLS) estimation to compute real-space derivatives:

$$\frac{\partial L}{\partial N} = \frac{L}{N}\frac{\partial \ln L}{\partial \ln N}, \quad \frac{\partial L}{\partial D} = \frac{L}{D}\frac{\partial \ln L}{\partial \ln D}$$

First-order diagnostic (log-linear):

$$\ln\left|\frac{\partial L}{\partial N}\right| = \alpha_N \ln N + \gamma_N \ln D + c_N, \quad \ln\left|\frac{\partial L}{\partial D}\right| = \gamma_D \ln N + \alpha_D \ln D + c_D \tag{1}$$

Mixed derivative test (decisive for interaction):

$$\ln\left|\frac{\partial^2 L}{\partial N \partial D}\right| = a \ln N + b \ln D + c \tag{2}$$

L-Shape Sampling Strategy

The L-shape strategy exploits the asymptotic structure of the loss:

$$\lim_{D \to \infty} L(N, D) = \left(\frac{A}{N^\alpha}\right)^k + E$$ $$\lim_{N \to \infty} L(N, D) = \left(\frac{B}{D^\beta}\right)^k + E$$

Rather than filling the entire (N, D) grid, the L-shape strategy:

• Sweeps data volume D exclusively for the smallest models (D-band) to fit $(B, \beta)$
• Sweeps model size N exclusively on shortest training horizons (N-band) to fit $(A, \alpha)$

This concentrates compute on the cheap edges of the grid.

Evaluation Protocol

Cross-validation with five evaluation regimes:

• Interpolation: Random held-out points within grid boundaries
• Extrapolation N: Larger model sizes beyond training set
• Extrapolation D: Larger data volumes
• Far Extrapolation: Both N and D beyond grid boundaries

MAPE metric:

$$\text{MAPE}(\mathcal{S}) = \frac{100}{|\mathcal{S}|}\sum_{i \in \mathcal{S}}\frac{|\hat{L}_i - L_i|}{L_i} [\%] \tag{4}$$

Iso-Ratio Compute Extrapolation

To mirror frontier lab practice (e.g., DeepSeek), runs are grouped into iso-ratio slices of constant $D/N$. The K highest-compute points per slice are held out, and laws are fit on the pooled low-compute remainder.

---

Empirical Validation / Results

Datasets

Key Results (Table 1)

Full grid results (Farseer, $5.0\times10^{22}$ FLOPs):

L-shape grid results (Farseer, $5.1\times10^{21}$ FLOPs — ~10× less compute):

Fitted Parameters (Table 2)

The coupling exponent $k$ is consistently sub-unit ($k \approx 0.31$–$0.45$), confirming the data supports a coupled surface rather than the additive case $k = 1$:

Iso-Ratio Compute Extrapolation (Table 3)

Skaling achieves pooled MAPE of $0.60\pm0.27\%$ — a 3.9× reduction over Chinchilla ($2.34\pm1.11\%$) and better than Farseer ($0.80\pm0.38\%$):

Allocation Frontier

On Farseer data, the token-to-parameter ratio decreases with compute:

• Skaling's analytic exponent: −0.11
• Numerical gradient estimate: −0.14 to −0.15
• Chinchilla's prediction: +0.03 (near-flat, wrong sign)

At one order of magnitude beyond the data, the two prescriptions differ by ~10× in recommended token-to-parameter ratio.

---

Theoretical and Practical Implications

Theoretical Significance

1. Direct evidence of N–D interaction: The non-zero mixed derivative $\partial^2 L / \partial N \partial D$ (with $a \approx b \approx -1.1$) provides rigorous empirical proof that model size and data interact synergistically—scaling both together lowers loss more than scaling either alone.

2. Minimal parameter extension: Adding just one parameter ($k$) to Chinchilla captures the interaction that a nine-parameter Farseer law handles less accurately, demonstrating the importance of inductive bias over raw parameter count.

3. Sub-additivity: The fitted $k < 1$ indicates a concave outer map, meaning the loss decays more slowly at large scales than additive laws predict—explaining why additive laws overestimate loss at large compute.

Practical Implications

1. Compute-efficient profiling: The L-shape strategy reduces profiling compute by ~10×, making principled scaling predictions feasible for smaller labs with constrained budgets.

2. Corrected compute allocation: Skaling's prediction that the optimal token-to-parameter ratio changes with compute (rather than staying constant as Chinchilla implies) has direct implications for frontier model design decisions.

3. Dataset-specific direction: The direction of the allocation trend ($\alpha < \beta$ vs. $\alpha > \beta$) is dataset-specific, meaning labs must fit Skaling on their own data rather than relying on universal rules.

---

Conclusion

The Skaling law resolves a fundamental flaw in additive scaling laws by introducing a single coupling exponent $k$ that captures the empirically verified interaction between model size and training data. Key takeaways:

1. Boundary bias eliminated: Skaling removes the systematic under- and overestimation of loss at the imbalanced extremes of the training grid.

2. Sparse profiling works: The L-shape sampling strategy, anchored on low-compute edges, recovers full-grid accuracy at ~10× less compute—reliable scaling prediction no longer requires massive dense sweeps.

3. Better allocation decisions: Skaling's compute-dependent optimal token-to-parameter ratio tracks empirical trends more accurately than Chinchilla's near-flat prediction.

4. Generalizable framework: The authors expect similar coupling dynamics to apply along other scaling axes (e.g., data mixtures, repeated data), presenting a clear direction for future work.

Future directions: Extending the coupling framework to other scaling dimensions, investigating why the coupling strength varies across datasets (from $k \approx 0.31$ on SK-Grid to $k \approx 0.77$–$0.90$ on other measurements), and developing efficient strategies to identify dataset-specific coupling strengths.