Ambiguous Fits: The Rashomon Effect and Challenges in Scaling Law Estimation

Summary (Overview)

• Core Problem: The final stage of neural scaling law estimation—fitting a parametric curve to observed (scale, loss) data—is treated as routine, yet standard heuristic optimization (Chinchilla Approach 3 with multi-start L-BFGS) suffers from two critical failures: no guarantee of near-optimality and obscuring of fundamentally different parameter choices that fit equally well.

• Rashomon Effect Demonstrated: The paper shows that scaling law fitting is subject to a pronounced Rashomon effect (Breiman, 2001): the sublevel sets of near-minimizers span broad parameter regions that achieve near-optimal fitting loss on observed data while yielding widely diverging predictions at larger scales. On real scaling ladders, fits with equal loss predict compute efficiency differences between 14× and 146× at target losses beyond training range.

• Certified Optimization Framework: The authors develop a deterministic branch-and-bound algorithm that provides three guarantees: (a) provably returns parameters within $\varepsilon$ of global minimum, (b) computes an arbitrarily tight cover of the $\varepsilon$-sublevel set, and (c) computes certified prediction envelopes bounding the range of predictions across all near-optimal fits.

• Key Theoretical Insight: A monotone reparameterization of the power-law family enables cheap uniform lower and upper bounds on fitting loss over parameter boxes, making branch-and-bound global optimization tractable.

• Practical Impact: Certified suboptimality measurements reveal that Chinchilla Approach 3 fits exceed the minimum fitting loss by between 2% and 180% across four real scaling ladders, even with dense initialization grids.

Introduction and Theoretical Foundation

Background

Neural scaling laws describe empirical power-law relationships between training resources (compute, data, parameters) and model performance (validation loss). They are essential tools for frontier AI model development, shaping architecture decisions and compute allocations. The estimation pipeline consists of four stages:

1. Scaling recipe design: selecting architectures, hyperparameters, and data mixtures
2. Training: training models across scales
3. Evaluation: measuring performance on target metrics
4. Scaling law fitting: applying a fitting algorithm to (scale, loss) pairs

The paper focuses on the final stage, which is typically treated as routine.

The Chinchilla Approach 3

The dominant fitting methodology parameterizes scaling laws as a sum of power laws:

$$L(x; \theta) = E + \sum_{i=1}^{k} A_i \cdot x_i^{-\alpha_i}, \quad \theta = (E, A_1, \alpha_1, \ldots, A_k, \alpha_k) \tag{1}$$

The fitting loss is the log-residual Huber loss:

$$L_{\mathrm{fit}}(\theta; \mathcal{D}) = \frac{1}{N_{\mathrm{obs}}} \sum_{n=1}^{N_{\mathrm{obs}}} \operatorname{Huber}_{10^{-3}}\left(\log L(x^{(n)}; \theta) - \log y^{(n)}\right)$$

where $\operatorname{Huber}_\delta(z) = \frac{1}{2}z^2$ if $|z| \le \delta$ and $\delta(|z| - \frac{\delta}{2})$ otherwise.

The optimizer initializes from a grid: $e = \log E \in \{-1, -0.5, \ldots, 1\}$, $a_i = \log A_i \in \{0, 5, \ldots, 25\}$, $\alpha_i \in \{0, 0.5, \ldots, 2\}$, runs L-BFGS from each point, and returns the best solution.

Two Fundamental Problems

Problem 1: No optimality guarantee. The fitting loss is non-convex, so local optimization from a grid does not guarantee finding near-global minima. The gap between the returned fit and the true optimum is unknown.

Problem 2: The Rashomon Effect. Heuristic optimizers return a single fit, obscuring that many near-optimal parameter configurations exist with virtually identical fitting loss but substantially different extrapolations. The sublevel set:

$$S_\varepsilon = \{\theta \in \Theta : L_{\mathrm{fit}}(\theta) - \min_{\theta' \in \Theta} L_{\mathrm{fit}}(\theta') \leq \varepsilon\}$$

can be broad, and which element is returned depends on incidental choices (initialization grid, solver settings).

Methodology

Monotone Reparameterization

The key theoretical contribution is a coordinate transformation that makes scaling law predictions monotone in parameters. For each input coordinate, define:

$$s_i(x) = \log \frac{x_i}{x_i^{\mathrm{min}}}, \qquad C_i = A_i (x_i^{\mathrm{min}})^{-\alpha_i}, \qquad r_i = e^{-\alpha_i} \tag{2}$$

Writing $u = (E, C_1, \ldots, C_k, r_1, \ldots, r_k)$ gives the equivalent form:

$$L(x; u) = E + \sum_{i=1}^{k} C_i r_i^{s_i(x)} \tag{3}$$

This is a bijective reparameterization ($u = \phi(\theta)$) whose inverse leaves $E$ unchanged and recovers $\alpha_i = -\log r_i$ and $A_i = C_i (x_i^{\mathrm{min}})^{\alpha_i}$.

Lemma 2 (Monotone prediction map and corner enclosure): Since $s_i(x^{(n)}) \geq 0$, the maps $u \mapsto L(x^{(n)}; u)$ are coordinatewise nondecreasing on the search domain $\mathcal{U}$. Consequently, for every parameter box $B = [B^\bot, B^\top] \subseteq \mathcal{U}$:

$$\log L(x^{(n)}; B^\bot) \leq \log L(x^{(n)}; u) \leq \log L(x^{(n)}; B^\top) \quad \forall u \in B \tag{6}$$

Box Loss Bounds

For a box $B$, define corner prediction intervals:

$$p_n^-(B) = \log L(x^{(n)}; B^\bot), \qquad p_n^+(B) = \log L(x^{(n)}; B^\top), \qquad I_n(B) = [p_n^-(B), p_n^+(B)] \tag{7}$$

The lower bound uses the unavoidable residual $d_n(B) = \operatorname{dist}(t_n, I_n(B))$:

$$\underline{L}_{\mathrm{fit}}(B) = \frac{1}{N_{\mathrm{obs}}} \sum_{n=1}^{N_{\mathrm{obs}}} \operatorname{Huber}_\delta(d_n(B)) \tag{9}$$

The upper bound exploits convexity of Huber loss (maximum attained at an endpoint):

$$\overline{L}_{\mathrm{fit}}(B) = \frac{1}{N_{\mathrm{obs}}} \sum_{n=1}^{N_{\mathrm{obs}}} \max\left\{\operatorname{Huber}_\delta(p_n^-(B) - t_n), \operatorname{Huber}_\delta(p_n^+(B) - t_n)\right\} \tag{10}$$

These satisfy $\underline{L}_{\mathrm{fit}}(B) \leq L_{\mathrm{fit}}(u) \leq \overline{L}_{\mathrm{fit}}(B)$ for all $u \in B$, and the gap converges to zero as the box diameter shrinks.

Three Certified Algorithms

Algorithm 1 (Certified fitting search): Branch-and-bound that maintains a priority queue of active regions ordered by loss lower bound. Splits boxes with the lowest lower bound, prunes regions whose lower bound exceeds the best candidate loss, and terminates when the incumbent is within $\varepsilon$ of the smallest remaining lower bound.

Algorithm 2 (Certified sublevel set cover): Two-phase algorithm that first certifies the global minimum to within $\varepsilon/3$, then constructs a box cover $\mathcal{B}$ satisfying:

$$S_{\epsilon/2}^{\phi} \subseteq \bigcup_{B \in \mathcal{B}} B \subseteq S_{\epsilon}^{\phi}$$

Algorithm 3 (Certified prediction envelope): Computes upper and lower bounds $[\underline{L}_{x^*}, \overline{L}_{x^*}]$ on predictions at target scale $x^*$ such that:

$$\inf \mathcal{L}_{x^*, \epsilon+\eta} \leq \underline{L}_{x^*} \leq \inf \mathcal{L}_{x^*, \epsilon}, \quad \text{and} \quad \sup \mathcal{L}_{x^*, \epsilon} \leq \overline{L}_{x^*} \leq \sup \mathcal{L}_{x^*, \epsilon+\eta}$$

The algorithm prioritizes boxes by maximum possible prediction at $x^*$, prunes boxes disjoint from $S_\epsilon^\phi$, and terminates when the highest-priority box is certified within $S_{\epsilon+\eta}^\phi$.

Theorem 1 (Main Result)

The three algorithms provide:

• (a) Certified fitting: returns $\hat{\theta} \in S_\varepsilon$ in finite iterations
• (b) Certified sublevel set cover: returns region $C$ with $S_{\varepsilon/2} \subseteq C \subseteq S_\varepsilon$
• (c) Certified prediction envelope: returns interval $[\underline{L}_{x^*}, \overline{L}_{x^*}]$ containing the prediction range for $S_\varepsilon$ and contained within that of $S_{\varepsilon+\eta}$

Empirical Validation / Results

Rashomon Effect on Real Scaling Ladders

The paper analyzes four real scaling ladders from the literature (Besiroglu et al., 2024; Dubey et al., 2024; Held et al., 2026; Krajewski et al., 2025), fitting the five-parameter law $L(N, D) = E + A_1 N^{-\alpha_1} + A_2 D^{-\alpha_2}$.

Key findings:

• With the standard $5^5$ grid initialization, fits in the sublevel set yield compute efficiency differences between 14× and 146× for a target loss
• Increasing grid density ($K = 2, 5, 8$) generally narrows envelopes but does not guarantee improvement (e.g., on Llama 3, the $2^5$ grid attains lower loss than $5^5$)
• Certified bounds show $5^5$ fits exceed minimum fitting loss by 2% to 180% across ladders

Synthetic Demonstration

On synthetic data, two power-law fits certified within 1% of optimum predict a 74.6× difference in compute needed to reach a given loss at 300× the training range.

Certified Suboptimality Measurement

Using Algorithm 1, the paper measures certified suboptimality of Chinchilla Approach 3 solutions versus branch-and-bound (Figure 4). Key results:

Runtime Analysis

• Worst-case bounds: $O(\varepsilon^{-d})$ iterations for fixed dimension $d = 2k+1$
• Algorithm 3 is fast for three-parameter laws (under a minute per endpoint at 1% tolerance)
• Five-parameter laws at tight tolerances currently exceed compute budget

Theoretical and Practical Implications

Theoretical Contributions

1. Characterization of non-identifiability: The paper rigorously demonstrates that scaling law fitting fundamentally suffers from the Rashomon effect—the $\varepsilon$-sublevel set contains parameter configurations with nearly identical fitting loss but wildly divergent extrapolations.

2. Novel monotone reparameterization: The transformation in Equation (2) provides a principled way to obtain box loss bounds, enabling deterministic global optimization for a non-convex problem that was previously only addressed heuristically.

3. Certified optimization framework: The branch-and-bound framework draws on interval analysis (Hansen and Walster, 2003) and Set Inversion Via Interval Analysis (SIVIA) (Jaulin and Walter, 1993), adapting these tools to the scaling law context.

Practical Implications

1. Scaling law forecasts are uncertain: Relying on a single fit, even one with good fitting loss, is unreliable. The prediction envelopes quantify the uncertainty inherent in the data—a crucial consideration for expensive training decisions.

2. Evaluation of fit quality: The certified bounds provide a principled way to measure suboptimality of any fitting algorithm, enabling apples-to-apples comparison.

3. Improved fitting practice: Practitioners can use Algorithm 1 for guaranteed near-optimal fits and Algorithm 3 for certified prediction envelopes at target scales.

4. Broader applicability: The monotone reparameterization approach can extend to other power-law families and many residual loss functions.

Conclusion

Main Takeaways

The paper demonstrates that the seemingly routine fitting stage of scaling law estimation introduces ambiguities that substantially undermine the reliability of scaling forecasts. Standard fitting procedures:

• Offer no guarantee of returning nearly optimal solutions
• Obscure the broad family of near-optimal fits with divergent extrapolations

The proposed branch-and-bound framework replaces heuristic local search with certified optimization, providing:

• Provable near-optimal fits
• Characterized sublevel sets of near-minimizers
• Certified prediction envelopes at unobserved scales

Future Directions

1. Computational scaling: More efficient splitting rules, tighter loss bounds, and parallel implementations for five-parameter laws at tight tolerances.

2. Randomness in observations: Each scaling ladder observation is a single random realization (one seed); accounting for this randomness in fitting and uncertainty quantification is an open problem.

3. Choice of objective: The log-residual Huber loss is standard but not clearly optimal; understanding how objective choice affects sublevel set structure is important.

4. Loss landscape geometry: Identifying which features of a scaling ladder make sublevel sets broad could inform experimental design.

5. Full pipeline optimization: The width of prediction envelopes is governed by all stages of the scaling law estimation pipeline—the design of the scaling ladder, training procedures, and evaluation metrics—not just the fitting stage.