Summary of "InfoLaw: Information-Based Scaling Law for Data-Constrained LLM Training"

Summary (Overview)

This paper proposes InfoLaw, a novel scaling law that predicts LLM performance under data-constrained settings where high-quality data is scarce and repetition is necessary. The approach models the "information" a model can extract from training data (accounting for data quality, repetition, and compute), and fits a parametric law that accurately extrapolates to unseen mixture recipes, larger model scales, and higher overtraining ratios.

---

Summary (Key Points)

• Problem: High-quality data is limited; training LLMs requires repeating data, which hurts performance. Existing scaling laws fail under data repetition and varied data quality mixtures.
• Proposed solution—InfoLaw: A new scaling law based on a novel Info metric that quantifies the useful information a model extracts from a repeated, quality-weighted data mixture:
  • Info = Σ_d f_d log(K) (1 - e^{-λ(N)R_d/log(K)})
  • where f_d = quality density of bucket d (fitted), R_d = repetition factor, K = total tokens, N = model FLOPs/token, and λ(N) = learning rate constant (fitted).
• Validation: Trained on 252M–1.2B models, the law extrapolates reliably to:
  • Unseen data recipes (MLQ, MHQ, random mixtures)
  • Larger scales (up to 7B models, 25× overtraining)
  • Maximum absolute error 0.96%, mean error 0.15% in validation loss.
• Practical utility: Enables efficient data-mixture selection for new training runs without expensive hyperparameter sweeps. Identified a 2.5B model recipe (HQ heavy, ~50% HQ + ~50% bucket-1) that outperforms random baselines.

---

Methodology

Layerwise Mixture (LayerMix):

• Data is split into 6 quality buckets (e.g., high-quality (HQ) to low-quality (LQ)).
• w_d defines the token proportion of bucket d in the training set.
• B_d defines the source budget per bucket.
• Training set size per bucket: K_d = w_d K, source size S_d = B_d S.
• Repetition factor: R_d = K_d / min(K_d, S_d).

Information model:

• The model derives value from data via a saturating exponential form (information extraction decays with repetition).
• λ(N) is a fitted function of model size (found to follow a log law, λ = 0.14 ln(N) + 0.018 in FP32 normalized units).
• f_d = quality density of each quality bucket (fitted).
• The law bridges compute C = K·N, data composition w, model size N, and repetition R_d.

Validation results:

• Loss prediction on unseen recipes (including 25 random mixtures): mean abs error 0.15%, max error 0.96%.
• Extrapolates from 1.2B models to 7B models (425B tokens) and to 25× overtraining (640B tokens on 1.2B model).
• Accurately ranks candidate recipes (e.g., for a 2.5B model) without extra training.
• Shows InfoLaw is better than Chinchilla optimal recipe under data-constrained regime (e.g., recommend 650B tokens for 2.5B, vs optimal-token 500B).
• High-quality data becomes more important as overtraining increases (optimal recipe emphasizes HQ data with repetition).

---

Summary (Details)

Notation

---

Summary (Detailed Structure)

1. Motivation & Problem

• LLM performance follows power laws in N and D, but this breaks when data is repeated.
• Traditional approaches fit separate scaling curves for each data mixture, requiring retraining for every new recipe—an expensive and impractical requirement.
• Need for a law that predicts loss for arbitrary mixtures of quality-bucketed data with repetition.

2. Method

• LayerMix sampling: Bucket data into 6 quality levels (HQ heavy, MLQ mid-low, etc.), each with a repetition factor.

• Key insight: Data quality influences when performance saturates; repetition produces diminishing returns modeled by an exponential term (1 - e^{-λR}).

• Fitting:

  • f_d (quality density) fit per bucket
  • λ(N) fit as λ = a·ln(N) + b (with a=0.140, b=0.018)
  • Optimization using Spearman correlation to find quality densities and λ.

• Key results:

  • Predicts validation loss on unseen recipes with 0.15% mean / 0.96% max absolute error.
  • Extrapolates from 252M–1.2B to 7B models (up to 25× overtraining).
  • Enables data recipe selection without extra training runs (e.g., identified superior mixtures for 2.5B model).

---

Summary (Core Claims & Experimental Evidence)

• Claim 1 (Conventional scaling laws fail under repetition): Standard L–C power laws fit poorly when data mixtures involve repetition and quality differences; a "repeat-and-upweight" approach shows significant slope breaks in loss–compute curves across data recipes.
• Claim 2 (Information is a unified predictor): The Info metric (instead of raw tokens or compute) collapses loss curves across different mixture recipes onto a single scaling law.
• Claim 3 (Extrapolation fails gracefully): With parameters fit on the 252M–1.2B set, the law predicts:
  • Unseen recipes (MLQ, MHQ and 25 random weight sets)
  • Larger models (1.5B–7B)
  • Higher overtraining (25× instead of 3.6×)
  • Combined extrapolation — all with <0.96% max absolute error relative to loss.

---

Summary (Key Details)

• Dataset curation: Data is partitioned into quality buckets (e.g., HQ/MLQ/MQ/LQ) using a "LayerMix" procedure. Each bucket has target sampling weight w_d, unique tokens M_d, and repetition factor R_d.

• Loss–Compute view:

  • Under overtraining (m=3.6), loss is plotted against C_m = N·K (Chinchilla-style compute), where N = non-embedding FLOPs/token and K = tokens.
  • Under excessive repetition, loss deviates from the standard power law; InfoLaw accounts for the diminishing learning benefit from repeated tokens.

• Info definition:

  • I_{d,total} = f_d M_d log(K) (1 - e^{-λ(N)R_d/log(K)})
  • Info = Σ_d I_{d,total}
  • f_d adjusts for information density per bucket; fitted via Spearman correlation ρ_s maximization.
  • λ(N) = a log(N) + b captures the model's "learning rate" as a function of model size.

• Results: InfoLaw achieves extremely accurate predictions:

  • Interpolation on unseen mixtures: accurate.
  • Extrapolation to 7B, 640B tokens: max error 0.96% in loss.
  • Extrapolation to 25× overtraining: reliable.

---

Summary (Methodology)

1. Partition the training data into quality buckets d with proportions w_d.
2. For each bucket: estimate information content I_d = f_d log(K), which scales logarithmically with total tokens.
3. Model forgetting/learning saturation from repetition with (1 - e^{-λ(N)R_d/log(K)}).
4. Total Information is a sum over buckets.
5. Fit f_d and λ(N) to minimize Spearman correlation between predicted info and observed loss on small models.
6. Use λ(N) = a log(N) + b to predict λ for unseen model sizes.

---

Summary (Training & Extrapolation Details)

• Trained over the full HQ, MQ, LQ, MLQ recipe set with multi-seed evaluations.
• No special tokenization/data filtering differences across buckets—only curation-based quality splits.
• Overtraining ratios: baseline m = 3.6×; high repetition test m' = 25×.
• Training data: same public LLM pretraining corpus for both small-scale fits and 7B/1.3B verifications.
• Validation losses computed on downstream language modeling benchmarks.

---

Conclusion (Paper's Main Message)

InfoLaw provides a principled, data-quality-aware scaling law that connects compute, model scale, data repetition, and quality mixtures into a single predictive law. It accurately predicts loss on unseen recipes and scales, including over-trained regimes, enabling efficient data recipe selection without expensive grid sweeps.

---

Summary (Representation of Key Formulas)

Repetition-aware Information

$$I_{i,\text{total}}(t, \lambda(N), K) = f_i \log(K) \left(1 - e^{-\lambda(N) R_i / \log(K)}\right)$$

Total Information

$$\mathcal{I} = \sum_{d} f_d \log(K) (1 - e^{-\lambda(N) R_d / \log(K)})$$

• f_d = quality density of bucket d (fitted)
• R_d = w_d K / M_d (repetition factor)
• λ(N) = a log N + b (fitted)
• The law then predicts: L = L(Info) via fitted power law.

---

Summary (Key Findings)

1. Information-theoretic scaling: Loss collapses when plotted against an Information metric, not raw token count, under mixed-quality data with repetition.
2. Parametric parsimony: One global law (Info vs Loss) fits all recipes, model sizes, and overtraining levels.
3. Fitted constants: a* = 0.140, b* = 0.018 (for λ(N)); f_d learned for quality densities.
4. Extrapolation is robust:
   • Unseen mixtures (random 25 recipes) → <0.15% mean error
   • Up to 25× overtraining → still accurate
   • Combined shifts (both mixture + model size + training tokens) → accurate to ≤0.96% max error
5. Data-efficiency: recipe selection via InfoLaw reduced search to 4 (instead of hundreds) of candidate runs, arriving at a better recipe than 4 random baselines.

---

Summary (Theoretical Insights & Implications)

• Measures learning as a saturating exponential in repetition level R, normalized by log(K).
• Shows that data quality and data repetition can be collapsed into a single scalar (Info) for scaling-law purposes.
• Demonstrates that Chinchilla-based scaling laws overestimate optimal data-compute tradeoffs when repetition is present.
• Repetition amortizes the value of data → reusing high-quality data can still be optimal up to a (compute-dependent) threshold; InfoLaw lets you choose that threshold automatically.
• The framework is flexible and can guide practical data selection for large-scale training runs without full-law refits per recipe.
• The model assumes a fixed λ(N) per model size; decoupling λ from N is left as future work. All results are on English data in the pretraining setting with recurrent repeats (same data repeated until end).

---

Summary (Keywords)

scaling laws, large language models, data quality, data repetition, overtraining, compute-optimal scaling, data-constrained training, LLM data selection, power law, neural scaling laws, downstream evaluation