# Tokens-per-Parameter Coverage Is Critical for Robust LLM Scaling Law Extrapolation

> viewaling-law fits from collinear token-per-parameter designs are fundamentally ill-conditioned, so non-collinear training grids are essential for robust extrapolation.

- **Source:** [arXiv](https://arxiv.org/abs/2605.08541)
- **Published:** 2026-08-17
- **Permalink:** https://picx.dev/p/xqiFHS
- **Whiteboard:** https://picx.dev/p/xqiFHS/image

## Summary

# Summary of "Tokens-per-Parameter Coverage Is Critical for Robust LLM Scaling Law Extrapolation"

## Summary (Overview)

- **Core finding**: Scaling-law fits trained on collinear designs (fixed tokens-per-parameter ratio, $D = kN$) are inherently ill-conditioned when the exponents governing model size ($N$) and data size ($D$) are nearly equal, making scale coefficients practically unidentifiable.

- **Key theoretical result**: The condition number of the Gauss-Newton least-squares problem grows as $\Theta(\varepsilon^{-2})$ where $\varepsilon$ is the gap between the $N$- and $D$-exponents. This is proven for four scaling-law formalisms: Chinchilla, repeated-data, Kaplan, and Droppo-Elibol.

- **Practical consequence**: Non-collinear designs outperform collinear ones on held-out data with a **97.3% win rate** across four laws, five corpora, and multiple floating-point precision modes (including BF16).

- **Constructive contribution**: A closed-form TPP-diversity threshold (Proposition 2) that is necessary and sufficient for well-conditioned estimation, plus an identified reduced model that recovers what single-ray designs can constrain.

- **Root cause**: The degeneracy is in Jacobian geometry, not the loss function—any smooth estimation objective whose curvature involves the Jacobian inherits the same ill-conditioning.

## Introduction and Theoretical Foundation

### Background and Motivation

Scaling laws approximate language model loss as a power-law function of parameter count $N$ and token count $D$. The seminal Chinchilla study (Hoffmann et al., 2022) proposed $D \approx 20N$ for compute-optimal training. However, the authors identify a critical methodological flaw:

> "This ratio is a prescription for how to allocate compute when training a single model, not a design for the experimental grid from which scaling laws are derived."

Fixed-TPP grids are widely used for scaling-law estimation despite this distinction, appearing in Cerebras-GPT, OLMo, and other major efforts. When all training runs lie on a single ray $D = kN$ in the $(N, D)$ plane, the predictor variables become collinear.

### Theoretical Foundation

The standard Chinchilla scaling law is:

$$
L(N, D) = A N^{-\alpha} + B D^{-\beta} + E, \tag{1}
$$

where $A, B, \alpha, \beta, E \in \mathbb{R}_{>0}$ and $N, D \in \mathbb{R}_{>0}$.

On a single ray ($K=1$), substituting $D = kN$ collapses the two power-law terms when $\beta = \alpha$:

**Definition 2 (Reduced Chinchilla model under fixed TPP):**

$$
L(N; \psi, \alpha, E) = (A + B k^{-\alpha}) N^{-\alpha} + E = \psi N^{-\alpha} + E, \tag{2}
$$

where $\psi := A + B k^{-\alpha}$.

**Definition 3 (Exponent gap):** The law-specific exponent gap $\varepsilon$ is defined as:
- Chinchilla/repeated-data: $\varepsilon := |\alpha - \beta|$
- Kaplan: $\varepsilon := |\alpha_D - \alpha_N|$
- Droppo-Elibol: $\varepsilon := |\gamma_N - \gamma_D|$ where $\gamma_N := \alpha_N/\alpha$, $\gamma_D := \alpha_D/\alpha$

## Methodology

### Gauss-Newton (GN) Framework

The authors fit by nonlinear least squares, minimizing $S(\boldsymbol{\theta}) := \frac{1}{2}\|\mathbf{r}(\boldsymbol{\theta})\|^2$ where residuals $r_i := L_i - \hat{L}(N_i, D_i; \boldsymbol{\theta})$ stack into $\mathbf{r}(\boldsymbol{\theta})$, with Jacobian $J \in \mathbb{R}^{m \times p}$ given by $J_{ij} = \partial r_i / \partial \theta_j$.

**Table 1: Jacobian columns under D = kN** (showing collinearity structure)

| Law ($\hat{L}$) | Collinear pair | $j_1$ | $j_2$ | $\varepsilon$ |
|---|---|---|---|---|
| $AN^{-\alpha} + BD^{-\beta} + E$ | (A,B) | $N^{-\alpha}$ | $k^{-\beta}N^{-\alpha}N^{\varepsilon}$ | $|\alpha - \beta|$ |
| $AN'^{-\alpha} + BD'^{-\beta} + E$ | (A,B) | $N'^{-\alpha}$ | $k^{-\beta}(D'/N')^{-\beta}N'^{-\alpha}N^{\varepsilon}$ | $|\alpha - \beta|$ |
| $[(N_c/N)^{\alpha_N/\alpha_D} + D_c/D]^{\alpha_D}$ | ($N_c$, $D_c$) | $\propto N^{-\alpha_D}$ | $\propto N^{-\alpha_D}N^{\varepsilon}$ | $|\alpha_D - \alpha_N|$ |
| $[(N_C/N)^{\alpha_N} + (D_C/D)^{\alpha_D} + E^{1/\alpha}]^\alpha$ | ($N_C$, $D_C$) | $\propto N^{-\gamma_N}$ | $\propto N^{-\gamma_N}N^{\varepsilon}$ | $|\gamma_N - \gamma_D|$ |

### Key Theoretical Results

**Proposition 1 (Full-matrix conditioning):** If two Jacobian columns satisfy $\mathbf{j}_b = c\mathbf{j}_a + \boldsymbol{\delta}$ with $\|\boldsymbol{\delta}\| = O(\varepsilon)$, then as $\varepsilon \to 0$:

$$
\kappa(J^T J) = \Theta(\varepsilon^{-2}). \tag{3}
$$

**Corollary 1 (Confidence interval inflation):** Under i.i.d. Gaussian noise on a single ray $D = kN$:

$$
\frac{\mathrm{CI}_{0.95}(A)}{\mathrm{CI}_{0.95}(\psi)} = \Theta(\varepsilon^{-1}). \tag{4}
$$

**Proposition 2 (TPP diversity threshold):** For any $\kappa_{\text{target}} > 0$ and ordered training ratios $k_1 \leq \cdots \leq k_K$:

$$
\kappa(J^T J) \leq \kappa_{\text{target}} \iff V_K \geq \tau_K, \tag{5}
$$

where $V_K$ is the second central moment of $\{k_\ell^{-\beta_{\text{eff}}}\}_{\ell=1}^K$ and $\beta_{\text{eff}}$ is the law-specific data-size exponent.

**Table 2: Statistical identifiability under single-TPP designs**

| Law | Exponents | $\varepsilon$ | $\varepsilon^{-1}$ | $\varepsilon^{-2}$ | $\kappa_{A,B}$ | CI inflation |
|---|---|---|---|---|---|---|
| Chinchilla | $\alpha \approx 0.34, \beta \approx 0.28$ | 0.06 | 17 | 278 | $10^3-10^4$ | $\gtrsim 17\times$ |
| Kaplan | $\alpha_N \approx 0.076, \alpha_D \approx 0.095$ | 0.019 | 53 | $\sim 2.8\times 10^3$ | $10^4-10^5$ | $\gtrsim 53\times$ |
| Droppo-Elibol | $\gamma_N \approx \gamma_D$ | $\sim 0.05-0.10$ | 10-20 | 100-400 | $10^3-10^4$ | $\gtrsim 10-20\times$ |

## Empirical Validation / Results

### Experimental Setup

- **Models**: LLaMA-style transformers, 5.04–76.5M parameters across 14 log-spaced configurations
- **Corpora**: C4, RedPajama, Wikipedia, Cosmopedia, peS2o
- **Designs**: Collinear (CO) with $k \in \{1, 1.5, 1.9, 2, 2.5, 2.7, 3, 3.3, 3.5, 4, 4.5, 5\}$ vs. Non-collinear (NC) with $14 \times 12$ grid spanning a two-dimensional region
- **Fitting**: Nonlinear least squares with 100 random restarts and differential-evolution polish, 30 independent optimizer seeds

### Aggregate Results

**Table 3: $R^2$ and RMSE summary across training designs**

| Split | Design | $R^2$ Mean | 95% CI | RMSE Mean | 95% CI |
|---|---|---|---|---|---|
| Train ($\mathcal{D}$) | CO | 0.9848 | [0.985, 0.985] | 0.1737 | [0.173, 0.174] |
| Train ($\mathcal{D}$) | NC | 0.9526 | [0.952, 0.953] | 0.2023 | [0.202, 0.203] |
| Holdout ($\mathcal{H}$) | CO | 0.8370 | [0.832, 0.842] | 0.2373 | [0.233, 0.241] |
| Holdout ($\mathcal{H}$) | NC | 0.9319 | [0.930, 0.934] | 0.1561 | [0.154, 0.158] |

**Key observation**: CO overfits training data (higher $R^2$), but NC generalizes dramatically better on holdout data—improving $R^2$ from 0.837 to 0.932 and cutting RMSE from 0.237 to 0.156.

**Table 4: Win rate breakdown (NC vs. CO), overall 97.3% (1,460/1,500)**

| Dataset | Win Rate | Scaling Law | Win Rate | Epoch | Win Rate |
|---|---|---|---|---|---|
| C4 | 93.0% | Chinchilla | 97.6% | first | 99.8% |
| Cosmopedia | 94.7% | Droppo-Elibol | 98.0% | second | 97.6% |
| peS2o | 99.3% | Kaplan | 95.6% | final | 93.8% |
| RedPajama | 100.0% | Repeated-Data | 100.0% | | |
| Wikipedia | 99.7% | | | | |

### Precision Invariance

**Table 5: BF16 mixed-precision results**

| Split | Design | $R^2$ Mean | RMSE Mean |
|---|---|---|---|
| Train ($\mathcal{D}$) | CO | 0.9896 | 0.1854 |
| Train ($\mathcal{D}$) | NC | 0.9907 | 0.1457 |
| Holdout ($\mathcal{H}$) | CO | 0.9412 | 0.2546 |
| Holdout ($\mathcal{H}$) | NC | 0.9657 | 0.1949 |

The BF16 win rate (98.7%, CI: [96.6%, 99.5%]) is comparable to full-precision results, confirming the effect is precision-invariant.

### Budget-Matched Subset Enumeration

**Table 6: Regime A NC-win rate (%) by law and dataset**

| Law | C4 | RedPajama | Wikipedia | Cosmopedia | peS2o | Dataset Avg. |
|---|---|---|---|---|---|---|
| Chinchilla | 77.1 | 78.6 | 88.0 | 51.8 | 47.1 | 68.5 [67.3, 69.8] |
| Droppo-Elibol | 62.2 | 67.2 | 65.9 | 45.5 | 46.2 | 57.4 [56.1, 58.7] |
| Kaplan | 76.1 | 79.9 | 72.9 | 50.9 | 50.3 | 66.0 [64.1, 67.9] |

The effect is strongest on web-crawled corpora (77–88% for Chinchilla) and systematically smaller on specialized corpora (Cosmopedia, peS2o).

## Theoretical and Practical Implications

### Theoretical Implications

1. **Jacobian geometry is the root cause**: The degeneracy is not an artifact of the loss function—any smooth estimation objective whose curvature involves the Jacobian inherits the same ill-conditioning.

2. **Explains replication gaps**: The Kaplan-Chinchilla discrepancy (Besiroglu et al., 2024; Porian et al., 2025) can be reframed as a predictable consequence of $\Theta(\varepsilon^{-2})$ ill-conditioning: $A$'s confidence intervals inflate $\geq 17\times$ at Chinchilla exponents and $\geq 53\times$ at Kaplan exponents.

3. **Transferability**: Because the degeneracy is purely Jacobian-geometric, the analysis transfers to generative vision, time-series, and scientific ML applications.

### Practical Implications

1. **Design diagnostic**: Proposition 2's condition (Eq. 5) can be evaluated before training using a literature estimate of $\beta_{\text{eff}}$, serving as an a-priori design diagnostic.

2. **Reduced model reporting**: When the design is collinear, the reduced parameterization (Definition 2) is the appropriate reporting target—along $D = kN$ rays, only the combined effect of $N$ and $D$ is well constrained.

3. **IsoFLOP prediction**: IsoFLOP curves cut across the training ray and probe the sloppy direction that collinear fitting leaves unconstrained, making them an informative diagnostic.

4. **Scaling 12 TPP ratios doesn't rescue collinearity**: The condition number $\kappa \propto \varepsilon^{-2}$ depends on the exponent gap, not the number of rays.

## Conclusion

### Main Takeaways

- Single-TPP fits do not support coefficient-level interpretation; claims about the relative importance of model size vs. data volume require training that spans a two-dimensional region of $(N, D)$.

- Collinear designs produce "deceptively confident fits that fail to generalize"—overfitting training data while degrading sharply off the training ray.

- Non-collinear designs are strictly preferred for scaling-law extrapolation, with a 97.3% win rate across all experimental conditions.

### Limitations

- Theorem 1 gives an ordering, not a quantitative gap
- Corollary 1 assumes i.i.d. Gaussian residuals (temporal correlation would understate, not invert, the pathology)
- Small-scale models (5.04–76.5M parameters), single architecture, English-only text
- Budget-matched validation softens to near-50% on specialized corpora (Cosmopedia, peS2o)

### Future Directions

- Downstream-task and cross-domain extensions that inherit the Jacobian argument
- Testing the framework on larger-scale models and additional architectures
- Extending the analysis to data-mixture weights and interaction terms
- Exploring isoFLOP-grid designs as a natural way to avoid the collinearity pitfall

The paper concludes that the degeneracy is fundamental and design-level: **any fixed-TPP experimental design will produce ill-conditioned scaling-law estimates**, and researchers should either use non-collinear designs or report the reduced model (Definition 2) when collinearity is unavoidable.

---

_Markdown view of https://picx.dev/p/xqiFHS, served by PicX — AI-generated visual whiteboard summaries of research papers._