# Scaling Laws for Vision-Language Fusion

> A unified scaling law shows early fusion preserves text performance while late fusion's vision advantage decays with model size, making early fusion preferable at scale.

- **Source:** [arXiv](https://arxiv.org/abs/2611.22351)
- **Published:** 2026-11-21
- **Permalink:** https://picx.dev/p/JUQza5
- **Whiteboard:** https://picx.dev/p/JUQza5/image

## Summary

## Summary (Overview)

- **Unified scaling law for vision-language fusion**: The paper introduces a novel scaling law that jointly characterizes both early and late fusion paradigms in multimodal pre-training, capturing how cross-modal interactions and distribution shifts evolve with fusion timing.
- **Key decomposition**: Validation loss is decomposed into a base learning trajectory (where different data types contribute at distinct exchange rates) and an additive distribution shift governed by competition between modalities.
- **Scale-dependent trade-off**: Early fusion preserves text performance while late fusion favors vision performance; however, the vision advantage of late fusion decays as $N^{-0.5}$ with model size, making early fusion increasingly attractive at scale.
- **Novel evaluation protocol**: The authors develop a rewriting strategy to evaluate base VLMs without instruction-following capabilities, converting multiple-choice benchmarks into log-likelihood ranking tasks to avoid answer-prior collapse.
- **Practical finding**: Vision benchmark gains do not track caption loss improvements, while text benchmarks closely mirror text loss — reinforcing early fusion as the preferred recipe at scale.

## Introduction and Theoretical Foundation

Vision-language models (VLMs) have become central foundation models, traditionally built via **late fusion** — augmenting a pretrained LLM with a visual encoder and continuing pre-training on multimodal data. More recently, **early fusion** (native multimodal pre-training) has gained traction by mixing all modalities from the start. However, the fundamental trade-offs between these paradigms remain poorly understood.

The authors identify a critical gap: no unified scaling law exists that can characterize both early and late fusion within a single formulation. Existing approaches treat these paradigms in isolation, failing to capture cross-modal interactions and distribution shifts as fusion timing varies. This leaves practitioners with heuristic-driven decisions about allocating expensive pre-training compute.

The core theoretical insight is that studying fusion timings under a fixed data budget reduces to modeling **distribution shift** governed by the post-fusion mixture ratio $R$ (the multimodal token fraction in continued pre-training data). The framework spans a continuous spectrum: small-shift regimes (image-text interleaved documents) resemble standard LLM continued pre-training, while large-shift regimes (image-caption data) exhibit sharply diminished cross-modal transfer.

## Methodology

### Experimental Setup

- **Architecture**: Gemma-3-style LLM backbone (189M to 3.5B parameters) augmented with a SigLIP2 visual encoder
- **Data sources**: FineWeb-Edu (text-only PT), OBELICS and OmniCorpus (interleaved), ReLAION (image-caption)
- **Control variables**: $D_{pt}$ and $D_{cpt}$ (PT and CPT token counts), $R$ (multimodal token fraction), $N$ (model size)
- **Observables**: Cross-entropy validation loss $L_m$ for modalities $m \in \{txt, int, cap\}$

### The Unified Scaling Law

The central formulation decomposes validation loss into a base learning trajectory and a distribution shift term:

$$L = L_0 + A \mathcal{D}_{base}^{-\alpha} + \Delta L(D_{cpt}) \tag{1}$$

The base term incorporates cross-modal exchange rates $w_m \in [0, \infty)$:

$$\mathcal{D}_{base,cap/int} = w_{cap/int} D_{pt} + (R + w_{cap/int}(1-R)) D_{cpt}$$
$$\mathcal{D}_{base,txt} = D_{pt} + (w_{txt} R + (1-R)) D_{cpt} \tag{2}$$

The shift term uses a fractional competition between modalities:

$$\rho_m(R) = \frac{k_m R}{k_m R + (1-R)} \tag{3}$$

with a saturation function borrowed from chemical kinetics:

$$\Phi(D_{cpt}) = 1 - \left(1 + (kR + 1-R) \frac{D_{cpt}}{\tau}\right)^{-\beta} \tag{4}$$

The final unified scaling law assembles all components:

$$L = L_0 + A \mathcal{D}_{base}^{-\alpha} + \sigma B_0 \rho(R) \Phi(D_{cpt}) \tag{5}$$

where $\sigma_{cap} = \sigma_{int} = -1$, $\sigma_{txt} = +1$.

### Model Size Extension

The law extends to model size with an additional Chinchilla-style term and scale-dependent shift amplitude:

$$\mathcal{L} = \mathcal{L}_0 + A \mathcal{D}_{base}^{-\alpha} + \sigma B_0 N^{-\gamma} \rho(R) \Phi(D_{cpt}) + C N^{-\eta} \tag{6}$$

## Empirical Validation / Results

### Key Fitted Parameters (Table 1)

| Parameter | Interleaved $L_{int}$ | Interleaved $L_{txt}$ | Caption $L_{cap}$ | Caption $L_{txt}$ |
|-----------|----------------------|----------------------|-------------------|-------------------|
| $w$ | 1.34 | 0.765 | 0.013 | 0.438 |
| $B_0$ | 0.416 | 0.182 | 0.501 | 0.741 |
| $\beta$ | 0.170 | 0.299 | 0.353 | 0.463 |
| Fit RMSE | 0.0022 | 0.0018 | 0.0099 | 0.0017 |

**Critical findings from parameters:**
- $w_{int} \approx 1.34$: text and interleaved tokens are near-substitutable (low-shift regime)
- $w_{cap} \approx 0.013$: caption learning progresses mainly on caption tokens — text tokens contribute almost nothing
- $\gamma_{cap} = 0.5$: shift amplitude decays as $N^{-0.5}$ for captions
- $\gamma_{txt} = 0$: text shift amplitude is size-independent

### Fusion Timing Trade-offs (Table 3, abbreviated)

| Scale | Schedule | $D_{pt}$ | R | Caption ↓ | Text ↓ | MME ↑ | GQA ↑ |
|-------|----------|---------|---|-----------|--------|-------|-------|
| 300M | S0 (Early) | 0 | 0.45 | 1.1488 | 3.4095 | 0.6701 | 0.4103 |
| 300M | S5 (Late) | 32k | 1.00 | 0.9902 | 6.3135 | 0.6770 | 0.3869 |
| 1B | S0 (Early) | 0 | 0.45 | 0.9659 | 3.0258 | 0.7064 | 0.4134 |
| 1B | S5 (Late) | 32k | 1.00 | 0.8705 | 5.8537 | 0.6839 | 0.4209 |

**Key observations:**
- Caption loss gap between early/late fusion narrows 40% (0.159 → 0.095) from 300M to 1B
- Text loss gap changes only 2.6% (2.904 → 2.828)
- Text benchmarks collapse at late fusion (R=1.0), with LAMBADA perplexity exploding by orders of magnitude
- Vision benchmarks show inconsistent correlations with caption loss across scales

## Theoretical and Practical Implications

### Theoretical Contributions

1. **Unified framework**: The scaling law bridges early and late fusion within a single formulation, with the base term unifying modalities via exchange rates and the shift term capturing cross-modal competition.

2. **Scale-dependent trade-off**: The mathematical structure reveals that $\gamma_{cap} = 0.5$ while $\gamma_{txt} = 0$, meaning the late fusion vision advantage shrinks as $N^{-0.5}$ while its text penalty persists. This provides theoretical justification for early fusion at scale.

3. **Token-equivalent analysis**: The vanishing loss gap translates to a vanishing token-equivalent gap — early fusion needs ~39B fewer tokens at 300M scale but only ~23B at 1B to match late fusion's vision performance.

### Practical Implications

- **Training recipe guidance**: Under fixed budgets, optimal fusion timing depends on priorities: late fusion minimizes vision loss ($D_{pt}^* = T_{txt}$), early fusion minimizes text loss ($D_{pt}^* = 0$)
- **Benchmark design**: The proposed rewriting strategy enables clean evaluation of base VLMs without instruction-following confounders, converting MCQ benchmarks to log-likelihood ranking tasks
- **Vision benchmark caveat**: The finding that vision benchmark scores don't track caption loss suggests VQA requires joint language-vision optimization — optimizing one while degrading the other may be counterproductive

## Conclusion

The paper establishes the first unified scaling law for vision-language fusion, decomposing learning dynamics into base modality progress and competitive distribution shift. The framework reveals that:

1. **Early fusion** minimizes text loss while **late fusion** minimizes vision loss
2. The late fusion vision advantage **decays with model size** ($N^{-0.5}$) while its text penalty persists
3. Vision benchmark gains **do not translate** from caption loss improvements
4. **Early fusion is the preferred recipe** at scale, as it preserves language capabilities without sacrificing downstream vision performance

**Future directions** include:
- Scaling to larger models and broader modalities
- Adapting the framework to other data sources and tokenization schemes
- Exploring regimes beyond the practical boundaries (e.g., extreme caption ratios causing language collapse)

The authors emphasize that fitted parameters should be treated as a framework for practitioners to characterize their own training runs rather than rigid universal constants.

---

_Markdown view of https://picx.dev/p/JUQza5, served by PicX — AI-generated visual whiteboard summaries of research papers._
