Problems with Chinchilla Approach 2: Systematic Biases in IsoFLOP Parabola Fits

Summary (Overview)

• Identifies systematic biases in Chinchilla Approach 2: The paper demonstrates that the parabolic approximation used in Approach 2 introduces structural biases in compute-optimal allocation estimates, even on noise-free synthetic data with perfect experimental conditions.
• Quantifies real-world costs: Applied to published Llama 3 IsoFLOP data at frontier compute scales ($3.8 × 10²⁵ FLOPs), these biases imply a parameter underallocation corresponding to 6.5$1.4M in unnecessary compute at 50% H100 MFU.
• Identifies three error sources: IsoFLOP sampling grid width (Taylor approximation accuracy), uncentered IsoFLOP sampling, and loss surface asymmetry ($\alpha \neq \beta$).
• Proposes VPNLS (Variable Projection with Non-negative Least Squares): A new fitting method that exploits the partially linear structure of the Chinchilla loss surface, reducing optimization to a well-conditioned 2D search over scaling exponents.
• Shows Approach 3 concerns are unfounded: Claims about Approach 3 being data-inefficient, numerically unstable, prone to local minima, and hard to implement are each shown to be addressable or incorrect.

Introduction and Theoretical Foundation

The Chinchilla paper (Hoffmann et al., 2022) introduced three approaches to scaling law estimation. Approach 2 has seen the broadest adoption, used by DeepMind, Meta, DeepSeek, Microsoft, Amazon, Waymo, and Arc Institute. The method's appeal lies in its putative stability and data efficiency relative to full nonlinear optimization.

The Chinchilla loss surface models model performance as:

$$L(N, D) = E + \frac{A}{N^{\alpha}} + \frac{B}{D^{\beta}} \tag{1}$$

where $N$ is the number of parameters, $D$ is the number of training tokens, $E$ is the irreducible loss, and $A, B, \alpha, \beta$ capture how quickly performance improves with scale.

Given a compute budget $C \approx 6ND$, the optimal allocation satisfies:

$$N^{*} \propto C^{a} \quad \text{where} \quad a = \frac{\beta}{\alpha + \beta} \tag{2}$$ $$D^{*} \propto C^{b} \quad \text{where} \quad b = \frac{\alpha}{\alpha + \beta} \tag{3}$$

Approach 2 proceeds in three steps:

1. Sample IsoFLOP contours for each compute budget $C$
2. Fit parabolas: $L = p(\log N)^2 + q(\log N) + r$, extract minimum $N^*$
3. Fit power laws: regress $\log N^*$ against $\log C$ to recover exponent $a$

Approach 3 fits all five parameters $(E, A, B, \alpha, \beta)$ simultaneously:

$$\min_{E, A, B, \alpha, \beta} \sum_{i} \left(L_i - \hat{L}(N_i, D_i)\right)^2 \tag{4}$$

with the prediction computed via LogSumExp reparameterization:

$$\log \hat{L}(N, D) = \operatorname{logsumexp}\left(e, a - \alpha \log N, b - \beta \log D\right) \tag{5}$$

Methodology

The study employs four complementary approaches:

1. Noise-free synthetic simulations: IsoFLOP experiments generated from known loss surfaces with controlled conditions (symmetric $\alpha = \beta$, asymmetric $\alpha \neq \beta$), varying sampling grid widths and center offsets.

2. Closed-form analytical derivation: The intercept error from parabola vertex shift is derived in closed form:

   $$\text{Intercept error} = 10^{\delta w} - 1 \tag{6}$$

   where $\delta w = f(\alpha, \beta, W, n)$ depends only on surface exponents and sampling grid, not on compute budget.

3. Noisy simulations with empirical noise models: Gaussian noise calibrated to residuals from 6 real IsoFLOP experiments, filtered through an 8-step quality control pipeline.

4. Empirical validation on Llama 3 data: Digitized IsoFLOP data from Figure 2 of the Llama 3 paper, fit with both Approach 2 and two Approach 3 variants.

Key simulation settings: Five compute budgets spanning $10^{17}$ to $10^{21}$ FLOPs, 15 points per IsoFLOP curve, four sampling grid widths (XS: ±2×, Small: ±4×, Large: ±8×, XL: ±16×).

VPNLS algorithm: Exploits partial linearity of the loss surface—for fixed $(\alpha, \beta)$, remaining parameters $(E, A, B)$ are solved exactly via least squares. The nonlinear search reduces to a well-conditioned 2D landscape over $(\alpha, \beta)$ with condition number $\kappa \approx 11$ vs $\kappa \approx 3.5 \times 10^{11}$ for the full 5D problem.

Empirical Validation / Results

Misallocation Costs at Scale

Deadweight Compute Loss (DCL) — excess FLOPs consumed relative to optimal allocation:

| Surface | D* true | D* inf. | N* true | N* inf. | ΔLoss (×10³) | DCL % | DCL $| b/a | |---|---|---|---|---|---|---|---| | Llama 3 (empirical) | 10.1T | 16.1T | 628B | 393B | +0.3 | 6.5$1.4M | 0.99 | | Llama 3 (simulated) | 10.2T | 16.1T | 622B | 393B | +0.3 | 6.3% | $1.3M | 0.99 | | Sparse-NMM | 176.5T | 272.3T | 36B | 23B | +0.9 | 10.1$2.1M | 1.91 | | SODA | 28.8T | 42.1T | 220B | 151B | +0.2 | 8.0% | $1.7M | 1.56 | | Chinchilla (simulated) | 29.7T | 33.5T | 213B | 189B | +0.1 | 0.4$95K | 1.21 | | Llama 3 (QC-filtered) | 10.2T | 10.8T | 622B | 584B | +0.0 | 0.1% | $25K | 0.99 |

Key Simulation Findings

• Symmetric surfaces ($\alpha = \beta$): Approach 2 recovers parameters to machine precision (relative error ~$10^{-10}\%$)
• Asymmetric surfaces: Intercepts systematically biased (Chinchilla: −4.1%, Asymmetric: −8.5% at ±16× grid), while exponents remain accurate
• Off-center sampling: Produces positive bias (overestimation of $D^*$), with narrower grids amplifying the error
• Drifting centers: Distorts both exponents and intercepts, producing the most severe misallocations
• Compounding errors: When asymmetry and off-center sampling act in the same direction, errors reach +35% on the Asymmetric surface

Method Comparison (Noisy Data)

Pooled across noise levels, budget counts, and dataset sizes:

• Approach 2: Largest maximum errors (716% on a, 239% on b)
• Naive Approach 3: Extreme errors (296% on a) without proper initialization
• Canonical Approach 3 (LSE + grid init): Max errors reduced to 44% on a
• VPNLS: Smallest maximum errors (34% on a, 11% on b) with comparable typical accuracy

Data Efficiency

Under ideal conditions (symmetric surface, perfectly centered sampling), Approach 2 shows ~8× higher variance than Approach 3 or VPNLS in recovering exponents.

Theoretical and Practical Implications

1. Structural nature of biases: The Approach 2 biases are inherent to the method's parabolic approximation, not statistical artifacts—they persist on noise-free data and compound with realistic noise.

2. Cost at production scale: At Llama 3 compute scale, Approach 2's bias translates to ~6.5% wasted compute ($1.4M at 50% H100 utilization). For multimodal surfaces with higher asymmetry, costs grow to 8-10% of budget.

3. Practical guidance: Practitioners using Approach 2 should recognize that scaling exponent estimates carry a systematic bias that grows with loss surface asymmetry, sampling center offsets, and sampling grid width.

4. VPNLS as a replacement: Offers equivalent or better accuracy than Approach 3 with a simpler optimization structure, while eliminating the parabolic approximation errors of Approach 2. Its 2D search space makes dense grid search practical and enables analytically differentiable gradients.

Conclusion

The Approach 2 biases documented in this paper are structural rather than statistical, existing on noise-free data with perfect experimental conditions. Three sources of error—sampling grid width, uncentered sampling, and loss surface asymmetry—can compound in practice, with published IsoFLOP curves showing clear signs of these pathologies.

Key takeaways:

• Approach 2's parabolic approximation introduces systematic biases that translate to meaningful compute waste at frontier scales
• Approach 3 eliminates these biases when properly implemented with grid initialization and LSE reparameterization
• VPNLS provides a simpler, more stable alternative that exploits the linear structure of the loss surface
• The structural advantages of VPNLS extend naturally to richer loss surface formulations (data repetition, MoE sparsity, data quality, etc.)

Limitations: The study assumes the Chinchilla loss surface model is correct, and the qualitative characterization of published study errors is not quantified precisely.

Future directions: VPNLS may serve as a more convenient replacement for Approach 2 or a more scalable alternative for adaptations of Approach 3 to richer scaling law formulations.