# InfoLaw: Information Scaling Laws for Large Language Models with Quality-Weighted Mixture Data and Repetition

> InfoLaw predicts LLM loss under data repetition and quality mixtures with under 1% error, enabling efficient data recipe selection without expensive hyperparameter sweeps.

- **Source:** [arXiv](https://arxiv.org/abs/2605.02364)
- **Published:** 2026-08-17
- **Permalink:** https://picx.dev/p/Xtt9iH
- **Whiteboard:** https://picx.dev/p/Xtt9iH/image

## Summary

# Summary of "InfoLaw: Information-Based Scaling Law for Data-Constrained LLM Training"

## Summary (Overview)

This paper proposes **InfoLaw**, a novel scaling law that predicts LLM performance under data-constrained settings where high-quality data is scarce and repetition is necessary. The approach models the "information" a model can extract from training data (accounting for data quality, repetition, and compute), and fits a parametric law that accurately extrapolates to unseen mixture recipes, larger model scales, and higher overtraining ratios.

---

## Summary (Key Points)

- **Problem**: High-quality data is limited; training LLMs requires repeating data, which hurts performance. Existing scaling laws fail under data repetition and varied data quality mixtures.
- **Proposed solution—InfoLaw**: A new scaling law based on a novel `Info` metric that quantifies the useful information a model extracts from a repeated, quality-weighted data mixture:
  - `Info = Σ_d f_d log(K) (1 - e^{-λ(N)R_d/log(K)})`
  - where `f_d` = quality density of bucket d (fitted), `R_d` = repetition factor, `K` = total tokens, `N` = model FLOPs/token, and `λ(N)` = learning rate constant (fitted).
- **Validation**: Trained on 252M–1.2B models, the law extrapolates reliably to:
  - Unseen data recipes (MLQ, MHQ, random mixtures)
  - Larger scales (up to 7B models, 25× overtraining)
  - Maximum absolute error **0.96%**, mean error **0.15%** in validation loss.
- **Practical utility**: Enables efficient data-mixture selection for new training runs without expensive hyperparameter sweeps. Identified a 2.5B model recipe (HQ heavy, ~50% HQ + ~50% bucket-1) that outperforms random baselines.

---

## Methodology

**Layerwise Mixture (LayerMix)**:
- Data is split into 6 quality buckets (e.g., high-quality (HQ) to low-quality (LQ)).
- `w_d` defines the token proportion of bucket *d* in the training set.
- `B_d` defines the source budget per bucket.
- Training set size per bucket: `K_d = w_d K`, source size `S_d = B_d S`.
- Repetition factor: `R_d = K_d / min(K_d, S_d)`.

**Information model**:
- The model derives value from data via a saturating exponential form (information extraction decays with repetition).
- `λ(N)` is a fitted function of model size (found to follow a log law, `λ = 0.14 ln(N) + 0.018` in FP32 normalized units).
- `f_d` = quality density of each quality bucket (fitted).
- The law bridges compute `C = K·N`, data composition `w`, model size `N`, and repetition `R_d`.

**Validation results**:
- Loss prediction on unseen recipes (including 25 random mixtures): mean abs error 0.15%, max error 0.96%.
- Extrapolates from 1.2B models to 7B models (425B tokens) and to 25× overtraining (640B tokens on 1.2B model).
- Accurately ranks candidate recipes (e.g., for a 2.5B model) without extra training.
- Shows InfoLaw is better than Chinchilla optimal recipe under data-constrained regime (e.g., recommend 650B tokens for 2.5B, vs optimal-token 500B).
- High-quality data becomes more important as overtraining increases (optimal recipe emphasizes HQ data with repetition).

---

## Summary (Details)

### Notation

| Symbol | Meaning |
|---|---|
| `K` | Total training tokens |
| `N` | Model non-embedding FLOPs per token |
| `S` | Source corpus tokens before sampling |
| `d` | Quality bucket (higher d = higher quality) |
| `w_d` | Target sampling weight for bucket d |
| `B_d` | Source fraction of bucket d in source corpus |
| `M_d = min(w_d K, B_d S)` | Unique tokens from bucket d |
| `R_d = w_d K / M_d` | Repetition factor for bucket d |
| `f_d` | Fitted quality density of bucket d |
| `λ(N)` | Fitted learning parameter as a function of N |
| `m` | Overtraining factor (e.g., 3.6, 25) |
| `C` | Compute = N·K |

---

## Summary (Detailed Structure)

### 1. Motivation & Problem
- LLM performance follows power laws in `N` and `D`, but this breaks when data is repeated.
- Traditional approaches fit separate scaling curves for each data mixture, requiring retraining for every new recipe—an expensive and impractical requirement.
- Need for a law that predicts loss for *arbitrary* mixtures of quality-bucketed data with repetition.

### 2. Method
- **LayerMix sampling**: Bucket data into 6 quality levels (HQ heavy, MLQ mid-low, etc.), each with a repetition factor.
- **Key insight**: Data quality influences *when* performance saturates; repetition produces diminishing returns modeled by an exponential term `(1 - e^{-λR})`.
- **Fitting**:
  - `f_d` (quality density) fit per bucket
  - `λ(N)` fit as `λ = a·ln(N) + b` (with a=0.140, b=0.018)
  - Optimization using Spearman correlation to find quality densities and λ.

- **Key results**:
  - Predicts validation loss on unseen recipes with **0.15% mean / 0.96% max absolute error**.
  - Extrapolates from 252M–1.2B to 7B models (up to 25× overtraining).
  - Enables data recipe selection without extra training runs (e.g., identified superior mixtures for 2.5B model).

---

## Summary (Core Claims & Experimental Evidence)

- **Claim 1 (Conventional scaling laws fail under repetition)**: Standard L–C power laws fit poorly when data mixtures involve repetition and quality differences; a "repeat-and-upweight" approach shows significant slope breaks in loss–compute curves across data recipes.
- **Claim 2 (Information is a unified predictor)**: The `Info` metric (instead of raw tokens or compute) collapses loss curves across different mixture recipes onto a single scaling law.
- **Claim 3 (Extrapolation fails gracefully)**: With parameters fit on the 252M–1.2B set, the law predicts:
  - Unseen recipes (MLQ, MHQ and 25 random weight sets)
  - Larger models (1.5B–7B)
  - Higher overtraining (25× instead of 3.6×)
  - Combined extrapolation — all with <0.96% max absolute error relative to loss.

---

## Summary (Key Details)

- **Dataset curation**: Data is partitioned into quality buckets (e.g., HQ/MLQ/MQ/LQ) using a "LayerMix" procedure. Each bucket has target sampling weight `w_d`, unique tokens `M_d`, and repetition factor `R_d`.
- **Loss–Compute view**:
  - Under overtraining (`m=3.6`), loss is plotted against `C_m = N·K` (Chinchilla-style compute), where `N` = non-embedding FLOPs/token and `K` = tokens.
  - Under excessive repetition, loss deviates from the standard power law; InfoLaw accounts for the diminishing learning benefit from repeated tokens.

- **Info definition**:
  - `I_{d,total} = f_d M_d log(K) (1 - e^{-λ(N)R_d/log(K)})`
  - `Info = Σ_d I_{d,total}`
  - `f_d` adjusts for information density per bucket; fitted via Spearman correlation `ρ_s` maximization.
  - `λ(N) = a log(N) + b` captures the model's "learning rate" as a function of model size.

- **Results**: InfoLaw achieves extremely accurate predictions:
  - Interpolation on unseen mixtures: accurate.
  - Extrapolation to 7B, 640B tokens: max error 0.96% in loss.
  - Extrapolation to 25× overtraining: reliable.

---

## Summary (Methodology)

1. Partition the training data into quality buckets **d** with proportions `w_d`.
2. For each bucket: estimate information content `I_d = f_d log(K)`, which scales logarithmically with total tokens.
3. Model forgetting/learning saturation from repetition with `(1 - e^{-λ(N)R_d/log(K)})`.
4. Total Information is a sum over buckets.
5. Fit `f_d` and `λ(N)` to minimize Spearman correlation between predicted info and observed loss on small models.
6. Use `λ(N) = a log(N) + b` to predict λ for unseen model sizes.

---

## Summary (Training & Extrapolation Details)

- Trained over the full `HQ, MQ, LQ, MLQ` recipe set with multi-seed evaluations.
- No special tokenization/data filtering differences across buckets—only curation-based quality splits.
- Overtraining ratios: baseline `m = 3.6×`; high repetition test `m' = 25×`.
- **Training data**: same public LLM pretraining corpus for both small-scale fits and 7B/1.3B verifications.
- **Validation losses** computed on downstream language modeling benchmarks.

---

## Conclusion (Paper's Main Message)

> **InfoLaw provides a principled, data-quality-aware scaling law that connects compute, model scale, data repetition, and quality mixtures into a single predictive law. It accurately predicts loss on unseen recipes and scales, including over-trained regimes, enabling efficient data recipe selection without expensive grid sweeps.**

---

## Summary (Representation of Key Formulas)

### Repetition-aware Information

$$
I_{i,\text{total}}(t, \lambda(N), K) = f_i \log(K) \left(1 - e^{-\lambda(N) R_i / \log(K)}\right)
$$

### Total Information

$$
\mathcal{I} = \sum_{d} f_d \log(K) (1 - e^{-\lambda(N) R_d / \log(K)})
$$

- `f_d` = quality density of bucket d (fitted)
- `R_d = w_d K / M_d` (repetition factor)
- `λ(N) = a log N + b` (fitted)
- The law then predicts: `L = L(Info)` via fitted power law.

---

## Summary (Key Findings)

1. **Information-theoretic scaling**: Loss collapses when plotted against an Information metric, not raw token count, under mixed-quality data with repetition.
2. **Parametric parsimony**: One global law (Info vs Loss) fits all recipes, model sizes, and overtraining levels.
3. **Fitted constants**: `a* = 0.140`, `b* = 0.018` (for λ(N)); `f_d` learned for quality densities.
4. **Extrapolation is robust**:
   - Unseen mixtures (random 25 recipes) → <0.15% mean error
   - Up to 25× overtraining → still accurate
   - Combined shifts (both mixture + model size + training tokens) → accurate to ≤0.96% max error
5. **Data-efficiency**: recipe selection via InfoLaw reduced search to 4 (instead of hundreds) of candidate runs, arriving at a better recipe than 4 random baselines.

---

## Summary (Theoretical Insights & Implications)

- Measures learning as a saturating exponential in repetition level `R`, normalized by `log(K)`.
- Shows that **data quality** and data **repetition** can be collapsed into a single scalar (Info) for scaling-law purposes.
- Demonstrates that **Chinchilla-based scaling laws overestimate** optimal data-compute tradeoffs when repetition is present.
- Repetition amortizes the value of data → reusing high-quality data can still be optimal up to a (compute-dependent) threshold; InfoLaw lets you *choose* that threshold automatically.
- The framework is flexible and can guide practical data selection for large-scale training runs without full-law refits per recipe.
- The model assumes a fixed λ(N) per model size; decoupling λ from N is left as future work. All results are on English data in the pretraining setting with recurrent repeats (same data repeated until end).

---

## Summary (Keywords)

`scaling laws`, `large language models`, `data quality`, `data repetition`, `overtraining`, `compute-optimal scaling`, `data-constrained training`, `LLM data selection`, `power law`, `neural scaling laws`, `downstream evaluation`

---

_Markdown view of https://picx.dev/p/Xtt9iH, served by PicX — AI-generated visual whiteboard summaries of research papers._