# Skaling: Chinchilla’s Exponents Meet Kaplan’s Coupling

> Skaling law couples model size and data via one interaction exponent, cutting loss-prediction error 1.5–3x over Chinchilla while requiring roughly 10x less tuning compute.

- **Source:** [arXiv](https://arxiv.org/abs/2608.07222)
- **Published:** 2026-08-17
- **Permalink:** https://picx.dev/p/Fk1zpP
- **Whiteboard:** https://picx.dev/p/Fk1zpP/image

## Summary

## Summary (Overview)

- **Core Contribution**: The paper introduces the **Skaling law**, a generalized neural scaling law that couples model size (N) and training data (D) through a single interaction exponent $k$, addressing the structural flaw of additive scaling laws like Chinchilla that assume N and D act independently on loss.
- **Empirical Validation**: The Skaling law reduces Mean Absolute Percentage Error (MAPE) by **1.5–3×** across interpolation and extrapolation regimes compared to Chinchilla, on two large-scale pretraining grids (Farseer with 404 runs, and a new SK-Grid with 134 runs).
- **Efficiency Gain**: Paired with a sparse "L-shape" grid sampling strategy restricted to low-compute edges, Skaling achieves full-grid predictive accuracy using **~10× less compute** than uniform sweeps.
- **Theoretical Insight**: Through mixed-derivative analysis, the authors demonstrate empirically that $\partial^2 L / \partial N \partial D \neq 0$, providing direct evidence that model size and data interact—a phenomenon additive laws cannot represent.
- **Practical Impact**: Skaling predicts a compute-dependent optimal token-to-parameter ratio that differs by up to **100×** from Chinchilla's prediction at frontier scales, with Skaling's prediction tracking empirical trends more accurately.

---

## Introduction and Theoretical Foundation

### Background and Motivation

Modern Large Language Models (LLMs) rely on **neural scaling laws** (Kaplan et al., 2020; Hofmann et al., 2022; Bi et al., 2024) to predict performance of high-compute runs from low-compute experiments. These laws guide critical decisions about pretraining budgets, architectural dimensions, and resource allocation.

However, the widely used **Chinchilla law** (Hofmann et al., 2022) models the reducible loss as a sum of independent terms:

$$L(N, D) = \frac{A}{N^\alpha} + \frac{B}{D^\beta} + E$$

This additive structure forces the cross-derivative to vanish identically:

$$\frac{\partial^2 L}{\partial N \partial D} = 0$$

The paper demonstrates that this assumption breaks down at the **boundaries of the training grid**—where N and D are most imbalanced. Figure 1 shows Chinchilla develops large, oppositely-signed errors (a saddle-shaped residual) toward the corners of the (N, D) grid.

### The Kaplan Form and Its Limitations

The earlier Kaplan form (Kaplan et al., 2020) took a coupled approach:

$$L(N, D) = \left[\left(N_c/N\right)^{\alpha_N/\alpha_D} + D_c/D\right]^{\alpha_D}$$

Here the outer exponent $\alpha_D$ plays the role of Skaling's $k$, but Kaplan ties the inner terms through the ratio $\alpha_N/\alpha_D$, losing independence of per-axis decay rates.

---

## Methodology

### The Skaling Law

The Skaling law bridges Chinchilla and Kaplan by retaining Chinchilla's interpretable base terms while introducing a single free outer exponent $k$:

$$L(N, D) = \left(\frac{A}{N^\alpha} + \frac{B}{D^\beta}\right)^k + E \tag{3}$$

**Key properties:**
- At $k = 1$, Skaling recovers the additive Chinchilla law
- For $k \neq 1$, it reinstates a Kaplan-style coupling with non-zero cross-derivative
- For $k > 0$, the function remains strictly decreasing in both N and D
- It preserves Chinchilla's closed-form compute-optimal allocation formula

### Derivative Diagnostics

The authors probe the loss surface using **Moving Least Squares (MLS)** estimation to compute real-space derivatives:

$$\frac{\partial L}{\partial N} = \frac{L}{N}\frac{\partial \ln L}{\partial \ln N}, \quad \frac{\partial L}{\partial D} = \frac{L}{D}\frac{\partial \ln L}{\partial \ln D}$$

**First-order diagnostic** (log-linear):

$$\ln\left|\frac{\partial L}{\partial N}\right| = \alpha_N \ln N + \gamma_N \ln D + c_N, \quad \ln\left|\frac{\partial L}{\partial D}\right| = \gamma_D \ln N + \alpha_D \ln D + c_D \tag{1}$$

**Mixed derivative test** (decisive for interaction):

$$\ln\left|\frac{\partial^2 L}{\partial N \partial D}\right| = a \ln N + b \ln D + c \tag{2}$$

### L-Shape Sampling Strategy

The L-shape strategy exploits the asymptotic structure of the loss:

$$\lim_{D \to \infty} L(N, D) = \left(\frac{A}{N^\alpha}\right)^k + E$$

$$\lim_{N \to \infty} L(N, D) = \left(\frac{B}{D^\beta}\right)^k + E$$

Rather than filling the entire (N, D) grid, the L-shape strategy:
- Sweeps data volume D exclusively for the smallest models (D-band) to fit $(B, \beta)$
- Sweeps model size N exclusively on shortest training horizons (N-band) to fit $(A, \alpha)$

This concentrates compute on the cheap edges of the grid.

### Evaluation Protocol

Cross-validation with five evaluation regimes:
- **Interpolation**: Random held-out points within grid boundaries
- **Extrapolation N**: Larger model sizes beyond training set
- **Extrapolation D**: Larger data volumes
- **Far Extrapolation**: Both N and D beyond grid boundaries

**MAPE metric**:

$$\text{MAPE}(\mathcal{S}) = \frac{100}{|\mathcal{S}|}\sum_{i \in \mathcal{S}}\frac{|\hat{L}_i - L_i|}{L_i} [\%] \tag{4}$$

### Iso-Ratio Compute Extrapolation

To mirror frontier lab practice (e.g., DeepSeek), runs are grouped into iso-ratio slices of constant $D/N$. The K highest-compute points per slice are held out, and laws are fit on the pooled low-compute remainder.

---

## Empirical Validation / Results

### Datasets

| Dataset | Configurations | Model Sizes | Data Budgets | Compute Range (FLOPs) |
|---------|---------------|-------------|--------------|----------------------|
| **Farseer** | 404 | 100M–6.4B | 1B–512B tokens | $1.6\times10^{18}$ to $4.1\times10^{21}$ |
| **SK-Grid** | 134 | 134M–4.9B | 316M–316B tokens | $9.0\times10^{16}$ to $9.9\times10^{20}$ |

### Key Results (Table 1)

**Full grid results (Farseer, $5.0\times10^{22}$ FLOPs):**

| Law | $R^2$ | Interp. | Ext. N | Ext. D | Far |
|-----|-------|---------|--------|--------|-----|
| Chinchilla | 0.995 | $0.77\pm0.04$ | $1.48\pm0.03$ | $1.98\pm0.08$ | $2.46\pm0.19$ |
| Farseer | 0.982 | $1.73\pm0.45$ | $2.37\pm0.08$ | $4.13\pm1.36$ | $2.43\pm1.93$ |
| **Skaling** | **0.998** | **$0.41\pm0.05$** | **$0.47\pm0.03$** | **$0.88\pm0.06$** | **$2.31\pm0.18$** |

**L-shape grid results (Farseer, $5.1\times10^{21}$ FLOPs — ~10× less compute):**

| Law | $R^2$ | Interp. | Ext. N | Ext. D | Far |
|-----|-------|---------|--------|--------|-----|
| Chinchilla | 0.954 | $2.51\pm0.07$ | $4.32\pm0.13$ | $3.29\pm0.11$ | $9.82\pm0.48$ |
| Farseer | 0.974 | $1.81\pm1.23$ | $2.07\pm1.72$ | $2.52\pm1.95$ | $2.37\pm1.33$ |
| **Skaling** | **0.995** | **$0.85\pm0.10$** | **$0.89\pm0.23$** | **$1.35\pm0.20$** | **$1.51\pm0.67$** |

### Fitted Parameters (Table 2)

The coupling exponent $k$ is consistently **sub-unit** ($k \approx 0.31$–$0.45$), confirming the data supports a coupled surface rather than the additive case $k = 1$:

| Setup | Law | $\alpha$ | $\beta$ | $k$ | $E$ |
|-------|-----|----------|---------|-----|-----|
| Farseer, full | Chinchilla | $0.27\pm0.01$ | $0.24\pm0.00$ | 1 | $0.45\pm0.01$ |
| Farseer, full | **Skaling** | $0.32\pm0.01$ | $0.39\pm0.00$ | **$0.41\pm0.01$** | $0.03\pm0.02$ |
| SK-Grid, full | Chinchilla | $0.34\pm0.01$ | $0.31\pm0.01$ | 1 | $1.75\pm0.02$ |
| SK-Grid, full | **Skaling** | $0.73\pm0.01$ | $0.63\pm0.01$ | **$0.31\pm0.02$** | $1.14\pm0.06$ |

### Iso-Ratio Compute Extrapolation (Table 3)

Skaling achieves pooled MAPE of **$0.60\pm0.27\%$** — a **3.9× reduction** over Chinchilla ($2.34\pm1.11\%$) and better than Farseer ($0.80\pm0.38\%$):

| Law | Undertrained MAPE | Optimal MAPE | Overtrained MAPE | All MAPE |
|-----|------------------|--------------|------------------|----------|
| Power law (per-ratio) | $1.32\pm0.87$ | **$0.77\pm0.39$** | $0.86\pm0.75$ | $0.99\pm0.69$ |
| Chinchilla | $1.52\pm0.80$ | $3.47\pm0.65$ | $1.94\pm0.66$ | $2.34\pm1.11$ |
| Farseer | $0.46\pm0.20$ | $1.20\pm0.20$ | $0.73\pm0.23$ | $0.80\pm0.38$ |
| **Skaling** | **$0.45\pm0.21$** | $0.88\pm0.15$ | **$0.42\pm0.15$** | **$0.60\pm0.27$** |

### Allocation Frontier

On Farseer data, the token-to-parameter ratio decreases with compute:
- Skaling's analytic exponent: **−0.11**
- Numerical gradient estimate: **−0.14 to −0.15**
- Chinchilla's prediction: **+0.03** (near-flat, wrong sign)

At one order of magnitude beyond the data, the two prescriptions differ by **~10×** in recommended token-to-parameter ratio.

---

## Theoretical and Practical Implications

### Theoretical Significance

1. **Direct evidence of N–D interaction**: The non-zero mixed derivative $\partial^2 L / \partial N \partial D$ (with $a \approx b \approx -1.1$) provides rigorous empirical proof that model size and data interact synergistically—scaling both together lowers loss more than scaling either alone.

2. **Minimal parameter extension**: Adding just one parameter ($k$) to Chinchilla captures the interaction that a nine-parameter Farseer law handles less accurately, demonstrating the importance of **inductive bias** over raw parameter count.

3. **Sub-additivity**: The fitted $k < 1$ indicates a concave outer map, meaning the loss decays more slowly at large scales than additive laws predict—explaining why additive laws overestimate loss at large compute.

### Practical Implications

1. **Compute-efficient profiling**: The L-shape strategy reduces profiling compute by **~10×**, making principled scaling predictions feasible for smaller labs with constrained budgets.

2. **Corrected compute allocation**: Skaling's prediction that the optimal token-to-parameter ratio changes with compute (rather than staying constant as Chinchilla implies) has direct implications for frontier model design decisions.

3. **Dataset-specific direction**: The direction of the allocation trend ($\alpha < \beta$ vs. $\alpha > \beta$) is dataset-specific, meaning labs must fit Skaling on their own data rather than relying on universal rules.

---

## Conclusion

The Skaling law resolves a fundamental flaw in additive scaling laws by introducing a single coupling exponent $k$ that captures the empirically verified interaction between model size and training data. Key takeaways:

1. **Boundary bias eliminated**: Skaling removes the systematic under- and overestimation of loss at the imbalanced extremes of the training grid.

2. **Sparse profiling works**: The L-shape sampling strategy, anchored on low-compute edges, recovers full-grid accuracy at ~10× less compute—reliable scaling prediction no longer requires massive dense sweeps.

3. **Better allocation decisions**: Skaling's compute-dependent optimal token-to-parameter ratio tracks empirical trends more accurately than Chinchilla's near-flat prediction.

4. **Generalizable framework**: The authors expect similar coupling dynamics to apply along other scaling axes (e.g., data mixtures, repeated data), presenting a clear direction for future work.

**Future directions**: Extending the coupling framework to other scaling dimensions, investigating why the coupling strength varies across datasets (from $k \approx 0.31$ on SK-Grid to $k \approx 0.77$–$0.90$ on other measurements), and developing efficient strategies to identify dataset-specific coupling strengths.

---

_Markdown view of https://picx.dev/p/Fk1zpP, served by PicX — AI-generated visual whiteboard summaries of research papers._