Summary of "Tokens-per-Parameter Coverage Is Critical for Robust LLM Scaling Law Extrapolation"

Summary (Overview)

• Core finding: Scaling-law fits trained on collinear designs (fixed tokens-per-parameter ratio, $D = kN$) are inherently ill-conditioned when the exponents governing model size ($N$) and data size ($D$) are nearly equal, making scale coefficients practically unidentifiable.

• Key theoretical result: The condition number of the Gauss-Newton least-squares problem grows as $\Theta(\varepsilon^{-2})$ where $\varepsilon$ is the gap between the $N$- and $D$-exponents. This is proven for four scaling-law formalisms: Chinchilla, repeated-data, Kaplan, and Droppo-Elibol.

• Practical consequence: Non-collinear designs outperform collinear ones on held-out data with a 97.3% win rate across four laws, five corpora, and multiple floating-point precision modes (including BF16).

• Constructive contribution: A closed-form TPP-diversity threshold (Proposition 2) that is necessary and sufficient for well-conditioned estimation, plus an identified reduced model that recovers what single-ray designs can constrain.

• Root cause: The degeneracy is in Jacobian geometry, not the loss function—any smooth estimation objective whose curvature involves the Jacobian inherits the same ill-conditioning.

Introduction and Theoretical Foundation

Background and Motivation

Scaling laws approximate language model loss as a power-law function of parameter count $N$ and token count $D$. The seminal Chinchilla study (Hoffmann et al., 2022) proposed $D \approx 20N$ for compute-optimal training. However, the authors identify a critical methodological flaw:

"This ratio is a prescription for how to allocate compute when training a single model, not a design for the experimental grid from which scaling laws are derived."

Fixed-TPP grids are widely used for scaling-law estimation despite this distinction, appearing in Cerebras-GPT, OLMo, and other major efforts. When all training runs lie on a single ray $D = kN$ in the $(N, D)$ plane, the predictor variables become collinear.

Theoretical Foundation

The standard Chinchilla scaling law is:

$$L(N, D) = A N^{-\alpha} + B D^{-\beta} + E, \tag{1}$$

where $A, B, \alpha, \beta, E \in \mathbb{R}_{>0}$ and $N, D \in \mathbb{R}_{>0}$.

On a single ray ($K=1$), substituting $D = kN$ collapses the two power-law terms when $\beta = \alpha$:

Definition 2 (Reduced Chinchilla model under fixed TPP):

$$L(N; \psi, \alpha, E) = (A + B k^{-\alpha}) N^{-\alpha} + E = \psi N^{-\alpha} + E, \tag{2}$$

where $\psi := A + B k^{-\alpha}$.

Definition 3 (Exponent gap): The law-specific exponent gap $\varepsilon$ is defined as:

• Chinchilla/repeated-data: $\varepsilon := |\alpha - \beta|$
• Kaplan: $\varepsilon := |\alpha_D - \alpha_N|$
• Droppo-Elibol: $\varepsilon := |\gamma_N - \gamma_D|$ where $\gamma_N := \alpha_N/\alpha$, $\gamma_D := \alpha_D/\alpha$

Methodology

Gauss-Newton (GN) Framework

The authors fit by nonlinear least squares, minimizing $S(\boldsymbol{\theta}) := \frac{1}{2}\|\mathbf{r}(\boldsymbol{\theta})\|^2$ where residuals $r_i := L_i - \hat{L}(N_i, D_i; \boldsymbol{\theta})$ stack into $\mathbf{r}(\boldsymbol{\theta})$, with Jacobian $J \in \mathbb{R}^{m \times p}$ given by $J_{ij} = \partial r_i / \partial \theta_j$.

Table 1: Jacobian columns under D = kN (showing collinearity structure)

Key Theoretical Results

Proposition 1 (Full-matrix conditioning): If two Jacobian columns satisfy $\mathbf{j}_b = c\mathbf{j}_a + \boldsymbol{\delta}$ with $\|\boldsymbol{\delta}\| = O(\varepsilon)$, then as $\varepsilon \to 0$:

$$\kappa(J^T J) = \Theta(\varepsilon^{-2}). \tag{3}$$

Corollary 1 (Confidence interval inflation): Under i.i.d. Gaussian noise on a single ray $D = kN$:

$$\frac{\mathrm{CI}_{0.95}(A)}{\mathrm{CI}_{0.95}(\psi)} = \Theta(\varepsilon^{-1}). \tag{4}$$

Proposition 2 (TPP diversity threshold): For any $\kappa_{\text{target}} > 0$ and ordered training ratios $k_1 \leq \cdots \leq k_K$:

$$\kappa(J^T J) \leq \kappa_{\text{target}} \iff V_K \geq \tau_K, \tag{5}$$

where $V_K$ is the second central moment of $\{k_\ell^{-\beta_{\text{eff}}}\}_{\ell=1}^K$ and $\beta_{\text{eff}}$ is the law-specific data-size exponent.

Table 2: Statistical identifiability under single-TPP designs

Empirical Validation / Results

Experimental Setup

• Models: LLaMA-style transformers, 5.04–76.5M parameters across 14 log-spaced configurations
• Corpora: C4, RedPajama, Wikipedia, Cosmopedia, peS2o
• Designs: Collinear (CO) with $k \in \{1, 1.5, 1.9, 2, 2.5, 2.7, 3, 3.3, 3.5, 4, 4.5, 5\}$ vs. Non-collinear (NC) with $14 \times 12$ grid spanning a two-dimensional region
• Fitting: Nonlinear least squares with 100 random restarts and differential-evolution polish, 30 independent optimizer seeds

Aggregate Results

Table 3: $R^2$ and RMSE summary across training designs

Key observation: CO overfits training data (higher $R^2$), but NC generalizes dramatically better on holdout data—improving $R^2$ from 0.837 to 0.932 and cutting RMSE from 0.237 to 0.156.

Table 4: Win rate breakdown (NC vs. CO), overall 97.3% (1,460/1,500)

Precision Invariance

Table 5: BF16 mixed-precision results

The BF16 win rate (98.7%, CI: [96.6%, 99.5%]) is comparable to full-precision results, confirming the effect is precision-invariant.

Budget-Matched Subset Enumeration

Table 6: Regime A NC-win rate (%) by law and dataset

The effect is strongest on web-crawled corpora (77–88% for Chinchilla) and systematically smaller on specialized corpora (Cosmopedia, peS2o).

Theoretical and Practical Implications

Theoretical Implications

1. Jacobian geometry is the root cause: The degeneracy is not an artifact of the loss function—any smooth estimation objective whose curvature involves the Jacobian inherits the same ill-conditioning.

2. Explains replication gaps: The Kaplan-Chinchilla discrepancy (Besiroglu et al., 2024; Porian et al., 2025) can be reframed as a predictable consequence of $\Theta(\varepsilon^{-2})$ ill-conditioning: $A$'s confidence intervals inflate $\geq 17\times$ at Chinchilla exponents and $\geq 53\times$ at Kaplan exponents.

3. Transferability: Because the degeneracy is purely Jacobian-geometric, the analysis transfers to generative vision, time-series, and scientific ML applications.

Practical Implications

1. Design diagnostic: Proposition 2's condition (Eq. 5) can be evaluated before training using a literature estimate of $\beta_{\text{eff}}$, serving as an a-priori design diagnostic.

2. Reduced model reporting: When the design is collinear, the reduced parameterization (Definition 2) is the appropriate reporting target—along $D = kN$ rays, only the combined effect of $N$ and $D$ is well constrained.

3. IsoFLOP prediction: IsoFLOP curves cut across the training ray and probe the sloppy direction that collinear fitting leaves unconstrained, making them an informative diagnostic.

4. Scaling 12 TPP ratios doesn't rescue collinearity: The condition number $\kappa \propto \varepsilon^{-2}$ depends on the exponent gap, not the number of rays.

Conclusion

Main Takeaways

• Single-TPP fits do not support coefficient-level interpretation; claims about the relative importance of model size vs. data volume require training that spans a two-dimensional region of $(N, D)$.

• Collinear designs produce "deceptively confident fits that fail to generalize"—overfitting training data while degrading sharply off the training ray.

• Non-collinear designs are strictly preferred for scaling-law extrapolation, with a 97.3% win rate across all experimental conditions.

Limitations

• Theorem 1 gives an ordering, not a quantitative gap
• Corollary 1 assumes i.i.d. Gaussian residuals (temporal correlation would understate, not invert, the pathology)
• Small-scale models (5.04–76.5M parameters), single architecture, English-only text
• Budget-matched validation softens to near-50% on specialized corpora (Cosmopedia, peS2o)

Future Directions

• Downstream-task and cross-domain extensions that inherit the Jacobian argument
• Testing the framework on larger-scale models and additional architectures
• Extending the analysis to data-mixture weights and interaction terms
• Exploring isoFLOP-grid designs as a natural way to avoid the collinearity pitfall

The paper concludes that the degeneracy is fundamental and design-level: any fixed-TPP experimental design will produce ill-conditioned scaling-law estimates, and researchers should either use non-collinear designs or report the reduced model (Definition 2) when collinearity is unavoidable.