Summary (Overview)

• SLDBench v2 extends the original SLDBench benchmark by introducing active scaling law discovery, where language-model agents must decide which experiments to purchase (at their recorded costs) and construct predictive laws from partial evidence, rather than receiving all experimental results upfront.
• The benchmark expands from 8 to 127 scenarios (3,376,938 prediction records), with 30 challenging tasks selected from 14 sources (339,092 records), spanning language-model pretraining, downstream performance, and vision–language transfer.
• Across 8 frontier agent systems (720 runs per setting), the Active setting dramatically separates systems: median $R^2$ drops from 0.72–0.75 (Passive) to 0.24–0.57 (Active), with prediction error rising 1.5–4.7× depending on the system.
• Agents consistently favor cheap experiments, spending only 8–44% of full-pool cost on average; low-cost acquisition can yield strong predictions, but cheap purchases often leave experimental conditions needed for extrapolation unobserved.
• The paper identifies experimental planning as a key open challenge for AI-for-AI research, showing that acquisition cost alone does not determine prediction quality.

Introduction and Theoretical Foundation

Scaling laws guide foundation-model development by predicting how performance changes with model size, training data, and computation (Kaplan et al., 2020; Hoffmann et al., 2022). Their practical value lies in extrapolation: experiments at smaller scales inform decisions about larger training runs, such as how to split compute between model size and data.

However, a reliable law requires experiments of its own. Researchers must choose which configurations to measure and how much compute to spend before predictions can guide investment. The paper motivates the problem as follows:

"Efficient scaling law discovery therefore depends on both the quality of a law and the cost of the evidence behind it."

The theoretical foundation builds on the Chinchilla scaling law for pretraining loss:

$$L(N, D) = E_g + A_g N^{-\alpha_g} + B_g D^{-\beta_g}$$

where $N$ is parameter count, $D$ is training tokens, and $g$ indexes experimental groups. This is one hypothesis; the agent must decide which variables enter the law, which parameters are shared across groups, and how the law is fitted.

Key distinction from prior work:

• SLDBench and symbolic regression fix the data and ask for the law
• Bayesian experimental design and SL² fix the law family and ask for the data
• SLDBench v2 asks for both, while fixing the target, test set, and prices

Methodology

Active Formulation

Active SLD withholds targets and exposes a priced catalogue $U = \{(i, x_i, g_i, c_i)\}_{i=1}^n$, where $c_i > 0$ is the cost of running experiment $i$ (typically training FLOPs). The agent starts with $S_0 = \emptyset$ and at round $t$ holds revealed history $H_t = \{(i, y_i) : i \in S_t\}$, either stopping or buying a new batch:

$$S_{t+1} = S_t \cup B_t, \quad C(S) = \sum_{i \in S} c_i, \quad \rho(S) = \frac{C(S)}{C(U)} \in [0, 1]$$

At stopping round $\tau$, the agent submits a law fitted to $H_\tau$. A solution couples three decisions: an acquisition policy, a stopping rule, and a law. Passive SLD is the special case $S_0 = U$ and $\rho = 1$.

Task Construction (Three-Stage Funnel)

1. Collection: 127 scenarios with 3,376,938 prediction records from public papers, repositories, and training logs
2. Active eligibility: Requires positive prices, $|U| \geq 40$, $c_{\text{max}}/c_{\text{min}} \geq 20$, at least 8 distinct prices, and median test price exceeding median candidate price — 60 scenarios qualify
3. Difficulty screen: A reference power-law fit must score $R^2 \leq 0.8$ in both settings — 30 tasks retained

Reference Law

The reference fit uses a shared additive power-law family with bounded multi-start least squares:

$$f_g(x) = E_g + \sum_{j=1}^{d} A_{g,j} \exp[-\alpha_{g,j} \log(1 + |x_j|)]$$

Execution Environment

Each task is packaged in the Harbor format with paired containers:

• Agent container: unprivileged, no network, 2 CPUs, 4 GB memory, 3,600s limit; holds instruction and priced catalogue (no target column)
• Verifier container: checks ledger signatures, runs the law unprivileged under 420s limit, scores on hidden test set

Experiments are replayed historically: selecting an experiment reveals its recorded outcome and charges its price via a root-only oracle with HMAC-signed ledger.

Metrics

Quality and cost are reported separately:

$$\text{NMSE}_T(f) = \frac{\sum_{i \in T} (y_i - f(x_i; \theta_{g_i}))^2}{\sum_{i \in T} (y_i - \bar{y}_T)^2}, \quad R^2 = 1 - \text{NMSE}_T(f)$$

The reward is unclipped $R^2$, and $\rho(S_\tau)$ is reported alongside but never subtracted (no fixed exchange rate between error and FLOPs is imposed).

Empirical Validation / Results

Main Results

Key findings:

• Passive medians span only 0.030 (0.723–0.753); Active medians span 0.334 (0.235–0.570)
• Passive ranking weakly predicts Active ranking (Spearman $r_s = 0.21$, $n = 8$)
• Every system loses quality: source-aggregated error multiplier between 1.49 and 4.71
• Pooled failure share ($R^2 \leq 0$ or no score) rises from 11.4% to 31.7%
• Runs that buy the full pool match Passive counterparts — the loss stems from what is bought, not sequential access

What Agents Buy

• Cheap first, expensive later: For every system, median share of candidates bought exceeds median share of cost; price ratios range 0.18–0.90 of catalogue mean
• Coverage gaps: 89.3% of Active runs observe every group, but 16.7% lose independent parameter variation, and 10.7% omit at least one group
• Cost does not determine quality: Claude Opus 5.5 achieves highest Active median at 4.01% cost; Hy4-Preview spends little (3.44%) but raises failure share most (+34.4 pp)

Non-Agent Rules

Under a fixed law, at $\rho \leq 5\%$:

• Random and price-spread selection beat cheapest-first on 63–77% of tasks
• Cheapest-per-group (reaching every group via cheapest candidates) is weakest: at $\rho = 2\%$, median $R^2$ is −1.456 vs. −0.005 for price-spread

"Under a fixed law, cheap points in every group add little information; variation in price within a group, and hence in scale, is what identifies the law, and it is exactly what costs the most."

Case Study (T22)

DeepSeek V4 Pro buys 78 points for 4.407% of pool cost, fitting $\hat{y}_g = E_g + A_g D^{-0.05}$ with shared exponent. The 14 sparsely sampled groups contribute 96.4% of test squared error, with extrapolation up to 1,000× the largest purchased sample count — the failure is an untested assumption (exponent transfer), not a shortage of points.

Theoretical and Practical Implications

Theoretical implications:

• Active SLD extends the objective from accurate extrapolation on fixed data to jointly balancing extrapolation quality and experimental cost
• The coupling between acquisition and law choice makes the problem fundamentally harder: "Two candidate laws that agree on the purchased points can diverge at the test scale, and only some purchases, often the expensive ones, can separate them"
• The benchmark provides an auditable record of which experiments agents chose and in what order, enabling analysis of acquisition behavior

Practical implications:

• The benchmark represents at least $1.09 \times 10^{23}$ FLOPs of source-reported training compute (from 2 C3 sources alone)
• Active SLD is implemented via historical experiment replay, allowing expensive decisions to be evaluated repeatedly without rerunning training
• The paired Passive/Active settings provide a controlled comparison: prediction target held fixed, only access to observations varies
• The Harbor task format means any command-line agent can be evaluated without adaptation

Conclusion

Main takeaways:

1. Systems nearly indistinguishable when all data are given separate widely when they must buy it
2. The Active loss is a heavier failure tail, not a uniform shift, and arises from what is bought
3. Agents favor cheap experiments, which can leave conditions needed for extrapolation unobserved
4. Acquisition cost alone does not determine prediction quality — some low-cost agents excel while others fail
5. Under a fixed law, variation in scale within each group matters more than reaching every group

Limitations:

• Replay restricts agents to experiments that were actually run; they cannot propose new configurations
• Prices mix reported FLOPs, derived estimates, and proxies
• The difficulty screen uses one reference law, favoring tasks on which it fails
• Cost is reported rather than traded off against quality (two axes, not one ranking)

Future direction: Agents that select experiments to discriminate between candidate laws, rather than by price, are identified as a natural next step.