Summary (Overview)
- Unified scaling law for vision-language fusion: The paper introduces a novel scaling law that jointly characterizes both early and late fusion paradigms in multimodal pre-training, capturing how cross-modal interactions and distribution shifts evolve with fusion timing.
- Key decomposition: Validation loss is decomposed into a base learning trajectory (where different data types contribute at distinct exchange rates) and an additive distribution shift governed by competition between modalities.
- Scale-dependent trade-off: Early fusion preserves text performance while late fusion favors vision performance; however, the vision advantage of late fusion decays as with model size, making early fusion increasingly attractive at scale.
- Novel evaluation protocol: The authors develop a rewriting strategy to evaluate base VLMs without instruction-following capabilities, converting multiple-choice benchmarks into log-likelihood ranking tasks to avoid answer-prior collapse.
- Practical finding: Vision benchmark gains do not track caption loss improvements, while text benchmarks closely mirror text loss — reinforcing early fusion as the preferred recipe at scale.
Introduction and Theoretical Foundation
Vision-language models (VLMs) have become central foundation models, traditionally built via late fusion — augmenting a pretrained LLM with a visual encoder and continuing pre-training on multimodal data. More recently, early fusion (native multimodal pre-training) has gained traction by mixing all modalities from the start. However, the fundamental trade-offs between these paradigms remain poorly understood.
The authors identify a critical gap: no unified scaling law exists that can characterize both early and late fusion within a single formulation. Existing approaches treat these paradigms in isolation, failing to capture cross-modal interactions and distribution shifts as fusion timing varies. This leaves practitioners with heuristic-driven decisions about allocating expensive pre-training compute.
The core theoretical insight is that studying fusion timings under a fixed data budget reduces to modeling distribution shift governed by the post-fusion mixture ratio (the multimodal token fraction in continued pre-training data). The framework spans a continuous spectrum: small-shift regimes (image-text interleaved documents) resemble standard LLM continued pre-training, while large-shift regimes (image-caption data) exhibit sharply diminished cross-modal transfer.
Methodology
Experimental Setup
- Architecture: Gemma-3-style LLM backbone (189M to 3.5B parameters) augmented with a SigLIP2 visual encoder
- Data sources: FineWeb-Edu (text-only PT), OBELICS and OmniCorpus (interleaved), ReLAION (image-caption)
- Control variables: and (PT and CPT token counts), (multimodal token fraction), (model size)
- Observables: Cross-entropy validation loss for modalities
The Unified Scaling Law
The central formulation decomposes validation loss into a base learning trajectory and a distribution shift term:
The base term incorporates cross-modal exchange rates :
The shift term uses a fractional competition between modalities:
with a saturation function borrowed from chemical kinetics:
The final unified scaling law assembles all components:
where , .
Model Size Extension
The law extends to model size with an additional Chinchilla-style term and scale-dependent shift amplitude:
Empirical Validation / Results
Key Fitted Parameters (Table 1)
| Parameter | Interleaved | Interleaved | Caption | Caption |
|---|---|---|---|---|
| 1.34 | 0.765 | 0.013 | 0.438 | |
| 0.416 | 0.182 | 0.501 | 0.741 | |
| 0.170 | 0.299 | 0.353 | 0.463 | |
| Fit RMSE | 0.0022 | 0.0018 | 0.0099 | 0.0017 |
Critical findings from parameters:
- : text and interleaved tokens are near-substitutable (low-shift regime)
- : caption learning progresses mainly on caption tokens — text tokens contribute almost nothing
- : shift amplitude decays as for captions
- : text shift amplitude is size-independent
Fusion Timing Trade-offs (Table 3, abbreviated)
| Scale | Schedule | R | Caption ↓ | Text ↓ | MME ↑ | GQA ↑ | |
|---|---|---|---|---|---|---|---|
| 300M | S0 (Early) | 0 | 0.45 | 1.1488 | 3.4095 | 0.6701 | 0.4103 |
| 300M | S5 (Late) | 32k | 1.00 | 0.9902 | 6.3135 | 0.6770 | 0.3869 |
| 1B | S0 (Early) | 0 | 0.45 | 0.9659 | 3.0258 | 0.7064 | 0.4134 |
| 1B | S5 (Late) | 32k | 1.00 | 0.8705 | 5.8537 | 0.6839 | 0.4209 |
Key observations:
- Caption loss gap between early/late fusion narrows 40% (0.159 → 0.095) from 300M to 1B
- Text loss gap changes only 2.6% (2.904 → 2.828)
- Text benchmarks collapse at late fusion (R=1.0), with LAMBADA perplexity exploding by orders of magnitude
- Vision benchmarks show inconsistent correlations with caption loss across scales
Theoretical and Practical Implications
Theoretical Contributions
-
Unified framework: The scaling law bridges early and late fusion within a single formulation, with the base term unifying modalities via exchange rates and the shift term capturing cross-modal competition.
-
Scale-dependent trade-off: The mathematical structure reveals that while , meaning the late fusion vision advantage shrinks as while its text penalty persists. This provides theoretical justification for early fusion at scale.
-
Token-equivalent analysis: The vanishing loss gap translates to a vanishing token-equivalent gap — early fusion needs ~39B fewer tokens at 300M scale but only ~23B at 1B to match late fusion's vision performance.
Practical Implications
- Training recipe guidance: Under fixed budgets, optimal fusion timing depends on priorities: late fusion minimizes vision loss (), early fusion minimizes text loss ()
- Benchmark design: The proposed rewriting strategy enables clean evaluation of base VLMs without instruction-following confounders, converting MCQ benchmarks to log-likelihood ranking tasks
- Vision benchmark caveat: The finding that vision benchmark scores don't track caption loss suggests VQA requires joint language-vision optimization — optimizing one while degrading the other may be counterproductive
Conclusion
The paper establishes the first unified scaling law for vision-language fusion, decomposing learning dynamics into base modality progress and competitive distribution shift. The framework reveals that:
- Early fusion minimizes text loss while late fusion minimizes vision loss
- The late fusion vision advantage decays with model size () while its text penalty persists
- Vision benchmark gains do not translate from caption loss improvements
- Early fusion is the preferred recipe at scale, as it preserves language capabilities without sacrificing downstream vision performance
Future directions include:
- Scaling to larger models and broader modalities
- Adapting the framework to other data sources and tokenization schemes
- Exploring regimes beyond the practical boundaries (e.g., extreme caption ratios causing language collapse)
The authors emphasize that fitted parameters should be treated as a framework for practitioners to characterize their own training runs rather than rigid universal constants.
Related papers
- InfuseLoop: Cross-Trajectory Latent Injection for Scalable Parallel Looped Transformers
InfuseLoop injects cross-trajectory latent states into parallel looped transformers, eliminating training spikes and enabling quality gains at 8 loops with minimal overhead.
- Grokking Can Be Steered: Selecting Desirable Solutions By Choice Of Learning Rule
The learning rule, not the task, determines neural circuit size in grokked networks, enabling deliberate trade-offs between robustness, compressibility, and transfer speed.
- Rethinking Fixed-Horizon Latent Reasoning
Adaptive latent inference that stops reasoning when progress stalls improves accuracy by up to 7.2 points while cutting computation by 29%, as completing fixed latent horizons can degrade performance.