# Scaling Laws for Multimodal Data Mixtures

> A unified multimodal scaling law reveals audio favors parameters over data, vision and text mutually interfere, and non-text repetition saturates after roughly two epochs.

- **Source:** [arXiv](https://arxiv.org/abs/2610.14423)
- **Published:** 2026-10-17
- **Permalink:** https://picx.dev/p/OxzCP6
- **Whiteboard:** https://picx.dev/p/OxzCP6/image

## Summary

# Scaling Laws for Multimodal Data Mixtures

## Summary (Overview)

- **Unified scaling-law framework**: The authors derive a multimodal scaling law for text, vision, and audio jointly trained in Mixture-of-Experts (MoE) architectures, extending the standard Chinchilla form to capture cross-modal transfer, capacity sharing, and repetition saturation.

- **268 controlled experiments**: The framework is fitted on MoE models ranging from 457M to 8.3B parameters, trained on up to 150B tokens, totaling roughly 300k GH200-hours of compute.

- **Key finding—modality-specific compute optima**: Text and vision favor balanced scaling of parameters and tokens, while audio is model-heavy, favoring parameter scaling over additional tokens.

- **Key finding—asymmetric cross-modal interference**: Vision strongly interferes with audio ($v_{vision \to audio} = -0.795$), while audio is effectively inert with respect to text and vision; text and vision mutually interfere with each other.

- **Key finding—repetition saturation**: Both vision and audio show sharply diminishing returns beyond roughly two passes over the same data, with repetition caps of $R^*_{vision} = 1.00$ and $R^*_{audio} = 1.76$.

## Introduction and Theoretical Foundation

Frontier AI systems are increasingly natively multimodal, jointly pretrained on text, vision, and audio. However, the scientific understanding of multimodal data mixtures—how much of each modality to include and at what interference cost—remains largely opaque. The authors identify several key challenges:

1. **Architectural convergence toward MoE**: MoEs scale capacity through sparse activation, decoupling parameter count from compute, and permit modality-specific expert allocation that partially insulates modalities from one another.

2. **Data imbalance**: Non-text modalities (vision, audio) are often trained in data-constrained regimes, requiring repeated passes over the same examples. While text repetition tolerance is well characterized, the saturation point for repeated vision or audio data was previously unknown.

3. **Optimization complexity**: Modalities have distinct feature representations, follow heterogeneous training dynamics, and compete for finite parameter capacity and routing budget.

The theoretical foundation builds on the standard Chinchilla scaling law [19]:

$$
L_m(N, D_m) = E_m + \frac{A_m}{N^{\alpha_m}} + \frac{B_m}{D_m^{\beta_m}}\tag{1}
$$

where $m \in \mathcal{M} = \{\text{text}, \text{vision}, \text{audio}\}$, $N$ is total parameter count, $D_m$ is the number of sampled training tokens from modality $m$, $E_m$ is the irreducible loss, and $A_m$, $B_m$, $\alpha_m$, $\beta_m$ are fitted constants.

## Methodology

### Effective Capacity with Mixture-Dependent Sharing

The authors replace $N$ with a target-specific effective capacity:

$$
N_{\text{eff}}^{(m)} = N \exp(c_m(1 - h_m))\tag{2}
$$

where $h_m = D_m / \sum_{\ell \in \mathcal{M}} D_\ell$ is the mixture weight of modality $m$, and $c_m$ is a signed log-capacity coefficient. If $c_m < 0$, non-target data dilutes effective capacity; if $c_m > 0$, shared capacity is beneficial.

### Effective Target-Modality Data

The effective data term combines two corrections:

**Repetition-aware factor** (diminishing returns on repeated data):

$$
\phi_m(r_m) = 1 + R_m^{\star}\left(1 - \exp\left(-\frac{r_m - 1}{R_m^{\star}}\right)\right)\tag{4}
$$

where $r_m = D_m / D_{\text{unq}}^{(m)} \geq 1$ is the repetition ratio and $R_m^{\star} > 0$ is a modality-specific repetition scale. As $r_m \to \infty$, $\phi_m(r_m) \to 1 + R_m^{\star}$.

**Directional cross-modal transfer** (log-scale to ensure positivity):

$$
D_m^t = D_m \exp\left(\sum_{k \neq m} v_{k \to m} h_k\right)\tag{6}
$$

where $v_{k \to m}$ is the directional transfer coefficient (positive = synergy, negative = interference).

### Unified Multimodal Scaling Law

Combining all components:

$$
L_m = E_m + \frac{A_m}{\left[N \exp(c_m(1 - h_m))\right]^{\alpha_m}} + \frac{B_m}{\left[D_{\text{unq}}^{(m)} \phi_m(r_m) \exp\left(\sum_{k \neq m} v_{k \to m} h_k\right)\right]^{\beta_m}}\tag{10}
$$

This reduces to the standard unimodal law when $h_m = 1$, $h_k = 0$ for $k \neq m$, and $r_m = 1$.

### Experimental Setup

- **Models**: Six MoE scales (457M to 8.3B total parameters), 32 layers, 16 routable experts (2 selected per token), GQA attention, SwiGLU activation, QK-Norm.
- **Tokenizers**: Mistral-Nemo (text, 131K vocab), Emu3.5 (images, 131K vocab), WavTokenizer (audio, 4K vocab, 40 tokens/sec).
- **Datasets**: Apertus-style text, Chameleon-style vision, Voxtral-style audio mixtures.
- **Training grid**: Unimodal baselines, text-vision and text-audio mixtures (5B–25B non-text tokens), repetition ablations (1×–5×), vision-audio and three-way mixtures.
- **Fitting**: Joint optimization of all coefficients via Huber loss on log validation loss with L-BFGS.

## Empirical Validation / Results

### Unimodal Scaling Laws

**Table 1: Fitted scaling-law exponents**

| Modality / Reference | $E_m$ | $\alpha_m$ | $\beta_m$ | $N_{opt} \propto C^a$ | $D_{opt} \propto C^b$ |
|---|---|---|---|---|---|
| Text | 1.91 | 0.34 | 0.34 | 0.50 | 0.50 |
| Vision | 5.67 | 0.50 | 0.51 | 0.50 | 0.50 |
| Audio | 4.36 | 0.32 | 0.52 | 0.62 | 0.38 |
| Chinchilla (text-dense) | 1.69 | 0.34 | 0.28 | 0.46 | 0.54 |
| TGL (text-MoE) | - | - | - | 0.51 | 0.49 |
| NMM (vision-dense) | 1.90 | 0.30 | 0.34 | 0.53 | 0.47 |
| SODA (audio-dense) | - | - | - | 0.37 | 0.58 |

Key observations:
- Vision has the highest scaling exponents ($\alpha_{vis} = 0.50$, $\beta_{vis} = 0.51$), about 50% above text values.
- Audio has asymmetric exponents ($\alpha_{aud} = 0.32$, $\beta_{aud} = 0.52$), tilting compute-optimal allocation toward model size ($N_{opt} \propto C^{0.62}$).
- Text matches Chinchilla's balanced split ($a = b = 0.50$).

### Cross-Modal Transfer and Interference

The fitted transfer coefficients reveal strong asymmetry:

- **Audio → Text**: $v_{audio \to text} = 0.146$ (not stable; effectively inert)
- **Audio → Vision**: $v_{audio \to vision} = 0.130$ (not stable; effectively inert)
- **Vision → Text**: $v_{vision \to text} = -0.558$ (interference)
- **Text → Vision**: $v_{text \to vision} = -0.583$ (interference)
- **Vision → Audio**: $v_{vision \to audio} = -0.795$ (strong interference)
- **Text → Audio**: $v_{text \to audio} = -0.102$ (mild interference)

### Repetition Saturation

- $R^*_{vision} = 1.00$: repetition caps at ~2.0 fresh-data-equivalent epochs
- $R^*_{audio} = 1.76$: repetition caps at ~2.76 fresh-data-equivalent epochs

Marginal worth of successive epochs (fraction of one unique-data epoch):
- Vision: 2nd epoch = 63%, 3rd = 23%, 5th = 3%
- Audio: 2nd epoch = 63%, 3rd = 43%, 5th = 14%

### Capacity Dilution

All capacity coefficients are negative:
- $c_{text} = -0.46$ (least sensitive)
- $c_{vision} = -0.85$
- $c_{audio} = -1.87$ (most sensitive, ~4× text and ~2× vision)

For a target at 10% mixture weight ($1 - h_m = 0.9$): effective capacity is ≈ 0.66N for text, 0.47N for vision, and only 0.19N for audio.

### Out-of-Distribution Validation

The fitted law was validated on three held-out runs:
1. **Val Scratch**: 12.6B-A3.8B MoE, 200B tokens (185B text, 7.5B vision, 7.5B audio)
2. **Val non-text-heavy CPT**: Continual pretraining on 50B tokens (10B text, 20B vision, 20B audio)
3. **Val text-heavy CPT**: CPT on 50B tokens (40B text, 5B vision, 5B audio)

Results: Pearson correlation $r = 0.999$; 8 of 9 predictions within 2.6% of observed loss; 6 of 9 within 1.1%.

## Theoretical and Practical Implications

### Theoretical Contributions

1. **Unified framework**: The proposed law generalizes prior bimodal interaction laws to directional, unbalanced, and repetition-aware mixtures, reducing to standard Chinchilla form in the unimodal limit.

2. **Log-linear parameterization**: Modeling transfer on the log-effective-data scale (Eq. 6) guarantees positivity even under severe interference, addressing a limitation of additive effective-data models.

3. **Modality-specific scaling regimes**: The framework reveals that different modalities occupy different regions of the compute-optimal frontier, challenging the assumption of a universal scaling recipe.

### Practical Implications

1. **Data mixture design**: The framework enables practitioners to derive context-specific optimal mixtures. Example: for a 120B-A29B MoE on 15T tokens with equal modality importance, the optimal mixture uses only 9.7T text tokens, repeating 1T vision tokens 2.6× and 500B audio tokens 5.3×.

2. **Compute allocation**: Audio's model-heavy regime ($a = 0.62$) suggests spending additional compute on parameters rather than audio tokens.

3. **Repetition policy**: Beyond ~2 epochs, compute is better spent on unique data or larger models for vision and audio.

4. **Modality addition cost estimation**: The framework can estimate the cost of adding a modality, revealing when scarce data should be repeated versus replaced with model scaling.

## Conclusion

The authors introduce a unified scaling-law framework for native multimodal MoE pretraining that captures how data amount, model capacity, data repetition, and cross-modal interactions shape per-modality loss. Key takeaways:

1. **Compute-optimal allocation is modality-specific**: text and vision favor balanced Chinchilla-style scaling; audio favors parameters over tokens.
2. **Cross-modal transfer is asymmetric**: audio is inert; text and vision interfere with each other and with audio.
3. **Non-text repetition saturates quickly**: both vision and audio plateau near two fresh-data-equivalent epochs.
4. **Audio is most capacity-sensitive**: roughly 4× more sensitive than text and 2× more than vision to capacity sharing.

**Future directions**: Validation at substantially larger scales (≫10B active parameters, >1T tokens), extension to additional modalities (e.g., long-form video), mechanistic analysis of audio's low interference, and exploration of bidirectional capacity sharing with more training runs.

**Limitations**: The fitted coefficients depend on tokenizer choices, loss objectives, and MoE architectural hyperparameters; the work is best viewed as a framework for fitting context-specific laws rather than a universal prescription.

---

_Markdown view of https://picx.dev/p/OxzCP6, served by PicX — AI-generated visual whiteboard summaries of research papers._