Summary (Overview)

  • Unified scaling law for vision-language fusion: The paper introduces a novel scaling law that jointly characterizes both early and late fusion paradigms in multimodal pre-training, capturing how cross-modal interactions and distribution shifts evolve with fusion timing.
  • Key decomposition: Validation loss is decomposed into a base learning trajectory (where different data types contribute at distinct exchange rates) and an additive distribution shift governed by competition between modalities.
  • Scale-dependent trade-off: Early fusion preserves text performance while late fusion favors vision performance; however, the vision advantage of late fusion decays as N−0.5N^{-0.5} with model size, making early fusion increasingly attractive at scale.
  • Novel evaluation protocol: The authors develop a rewriting strategy to evaluate base VLMs without instruction-following capabilities, converting multiple-choice benchmarks into log-likelihood ranking tasks to avoid answer-prior collapse.
  • Practical finding: Vision benchmark gains do not track caption loss improvements, while text benchmarks closely mirror text loss — reinforcing early fusion as the preferred recipe at scale.

Introduction and Theoretical Foundation

Vision-language models (VLMs) have become central foundation models, traditionally built via late fusion — augmenting a pretrained LLM with a visual encoder and continuing pre-training on multimodal data. More recently, early fusion (native multimodal pre-training) has gained traction by mixing all modalities from the start. However, the fundamental trade-offs between these paradigms remain poorly understood.

The authors identify a critical gap: no unified scaling law exists that can characterize both early and late fusion within a single formulation. Existing approaches treat these paradigms in isolation, failing to capture cross-modal interactions and distribution shifts as fusion timing varies. This leaves practitioners with heuristic-driven decisions about allocating expensive pre-training compute.

The core theoretical insight is that studying fusion timings under a fixed data budget reduces to modeling distribution shift governed by the post-fusion mixture ratio RR (the multimodal token fraction in continued pre-training data). The framework spans a continuous spectrum: small-shift regimes (image-text interleaved documents) resemble standard LLM continued pre-training, while large-shift regimes (image-caption data) exhibit sharply diminished cross-modal transfer.

Methodology

Experimental Setup

  • Architecture: Gemma-3-style LLM backbone (189M to 3.5B parameters) augmented with a SigLIP2 visual encoder
  • Data sources: FineWeb-Edu (text-only PT), OBELICS and OmniCorpus (interleaved), ReLAION (image-caption)
  • Control variables: DptD_{pt} and DcptD_{cpt} (PT and CPT token counts), RR (multimodal token fraction), NN (model size)
  • Observables: Cross-entropy validation loss LmL_m for modalities m∈{txt,int,cap}m \in \{txt, int, cap\}

The Unified Scaling Law

The central formulation decomposes validation loss into a base learning trajectory and a distribution shift term:

L=L0+ADbase−α+ΔL(Dcpt)(1)L = L_0 + A \mathcal{D}_{base}^{-\alpha} + \Delta L(D_{cpt}) \tag{1}

The base term incorporates cross-modal exchange rates wm∈[0,∞)w_m \in [0, \infty):

Dbase,cap/int=wcap/intDpt+(R+wcap/int(1−R))Dcpt\mathcal{D}_{base,cap/int} = w_{cap/int} D_{pt} + (R + w_{cap/int}(1-R)) D_{cpt} Dbase,txt=Dpt+(wtxtR+(1−R))Dcpt(2)\mathcal{D}_{base,txt} = D_{pt} + (w_{txt} R + (1-R)) D_{cpt} \tag{2}

The shift term uses a fractional competition between modalities:

ρm(R)=kmRkmR+(1−R)(3)\rho_m(R) = \frac{k_m R}{k_m R + (1-R)} \tag{3}

with a saturation function borrowed from chemical kinetics:

Φ(Dcpt)=1−(1+(kR+1−R)Dcptτ)−β(4)\Phi(D_{cpt}) = 1 - \left(1 + (kR + 1-R) \frac{D_{cpt}}{\tau}\right)^{-\beta} \tag{4}

The final unified scaling law assembles all components:

L=L0+ADbase−α+σB0ρ(R)Φ(Dcpt)(5)L = L_0 + A \mathcal{D}_{base}^{-\alpha} + \sigma B_0 \rho(R) \Phi(D_{cpt}) \tag{5}

where σcap=σint=−1\sigma_{cap} = \sigma_{int} = -1, σtxt=+1\sigma_{txt} = +1.

Model Size Extension

The law extends to model size with an additional Chinchilla-style term and scale-dependent shift amplitude:

L=L0+ADbase−α+σB0N−γρ(R)Φ(Dcpt)+CN−η(6)\mathcal{L} = \mathcal{L}_0 + A \mathcal{D}_{base}^{-\alpha} + \sigma B_0 N^{-\gamma} \rho(R) \Phi(D_{cpt}) + C N^{-\eta} \tag{6}

Empirical Validation / Results

Key Fitted Parameters (Table 1)

ParameterInterleaved LintL_{int}Interleaved LtxtL_{txt}Caption LcapL_{cap}Caption LtxtL_{txt}
ww1.340.7650.0130.438
B0B_00.4160.1820.5010.741
β\beta0.1700.2990.3530.463
Fit RMSE0.00220.00180.00990.0017

Critical findings from parameters:

  • wint≈1.34w_{int} \approx 1.34: text and interleaved tokens are near-substitutable (low-shift regime)
  • wcap≈0.013w_{cap} \approx 0.013: caption learning progresses mainly on caption tokens — text tokens contribute almost nothing
  • γcap=0.5\gamma_{cap} = 0.5: shift amplitude decays as N−0.5N^{-0.5} for captions
  • γtxt=0\gamma_{txt} = 0: text shift amplitude is size-independent

Fusion Timing Trade-offs (Table 3, abbreviated)

ScaleScheduleDptD_{pt}RCaption ↓Text ↓MME ↑GQA ↑
300MS0 (Early)00.451.14883.40950.67010.4103
300MS5 (Late)32k1.000.99026.31350.67700.3869
1BS0 (Early)00.450.96593.02580.70640.4134
1BS5 (Late)32k1.000.87055.85370.68390.4209

Key observations:

  • Caption loss gap between early/late fusion narrows 40% (0.159 → 0.095) from 300M to 1B
  • Text loss gap changes only 2.6% (2.904 → 2.828)
  • Text benchmarks collapse at late fusion (R=1.0), with LAMBADA perplexity exploding by orders of magnitude
  • Vision benchmarks show inconsistent correlations with caption loss across scales

Theoretical and Practical Implications

Theoretical Contributions

  1. Unified framework: The scaling law bridges early and late fusion within a single formulation, with the base term unifying modalities via exchange rates and the shift term capturing cross-modal competition.

  2. Scale-dependent trade-off: The mathematical structure reveals that γcap=0.5\gamma_{cap} = 0.5 while γtxt=0\gamma_{txt} = 0, meaning the late fusion vision advantage shrinks as N−0.5N^{-0.5} while its text penalty persists. This provides theoretical justification for early fusion at scale.

  3. Token-equivalent analysis: The vanishing loss gap translates to a vanishing token-equivalent gap — early fusion needs ~39B fewer tokens at 300M scale but only ~23B at 1B to match late fusion's vision performance.

Practical Implications

  • Training recipe guidance: Under fixed budgets, optimal fusion timing depends on priorities: late fusion minimizes vision loss (Dpt∗=TtxtD_{pt}^* = T_{txt}), early fusion minimizes text loss (Dpt∗=0D_{pt}^* = 0)
  • Benchmark design: The proposed rewriting strategy enables clean evaluation of base VLMs without instruction-following confounders, converting MCQ benchmarks to log-likelihood ranking tasks
  • Vision benchmark caveat: The finding that vision benchmark scores don't track caption loss suggests VQA requires joint language-vision optimization — optimizing one while degrading the other may be counterproductive

Conclusion

The paper establishes the first unified scaling law for vision-language fusion, decomposing learning dynamics into base modality progress and competitive distribution shift. The framework reveals that:

  1. Early fusion minimizes text loss while late fusion minimizes vision loss
  2. The late fusion vision advantage decays with model size (N−0.5N^{-0.5}) while its text penalty persists
  3. Vision benchmark gains do not translate from caption loss improvements
  4. Early fusion is the preferred recipe at scale, as it preserves language capabilities without sacrificing downstream vision performance

Future directions include:

  • Scaling to larger models and broader modalities
  • Adapting the framework to other data sources and tokenization schemes
  • Exploring regimes beyond the practical boundaries (e.g., extreme caption ratios causing language collapse)

The authors emphasize that fitted parameters should be treated as a framework for practitioners to characterize their own training runs rather than rigid universal constants.

Related papers