# Towards Physics of Multimodal Pretraining: Knowledge Flow, Modality Synergy, Early Unification, and Recipes

> This paper establishes a causal "physics" of multimodal pretraining, showing language universally boosts vision, vision aids generation, but generation minimally transfers back, with asymmetric data mixes enabling efficient scaling.

- **Source:** [arXiv](https://arxiv.org/abs/2608.05000)
- **Published:** 2026-08-07
- **Permalink:** https://picx.dev/p/12bbbP

## Summary

## Summary (Overview)

- This paper presents a systematic, bottom-up empirical study of multimodal pretraining, aiming to replace heuristics with evidence about how modalities interact during unified training.
- The authors identify four key insights: **(i) Asymmetric Knowledge Flow**—language acts as a universal booster, visual understanding strongly aids generation, but generation offers minimal backward transfer; **(ii) Synergy vs. Competition**—determined by data complexity and architectural choices (shared attention/norm + split FFNs); **(iii) Early Unification**—joint training from the start is critical, revealing a "vision laziness" phenomenon; **(iv) Recipes**—highly asymmetric data mixes (e.g., 70/25/5) enable efficient scaling.
- Findings are validated on both synthetic (CLEVR) and real-world data (DCLM, SSTK), and scaled to 13.5B MoE models trained on 2T tokens.

## Introduction and Theoretical Foundation

The field of foundation models is moving from unimodal language models to natively unified multimodal models that treat **visual understanding** and **visual generation** as simultaneous, first-class objectives. However, the design space is vast and largely navigated by heuristics. Most existing approaches retrofit pretrained LLMs/MLLMs, treating vision as a post-hoc module, which obscures how vision should actively participate in foundational pretraining.

This paper aims to establish a **"physics" of multimodal pretraining**—the fundamental mechanisms governing modality interaction, knowledge transfer, and training dynamics. The theoretical foundation draws on:
- **Transfusion framework**: unifies discrete next-token prediction for text and continuous flow matching for visual generation.
- **Platonic Representation Hypothesis**: suggests a mature language manifold may capture modality-agnostic world structure that bootstraps visual learning.
- The authors systematically isolate variables via controlled experiments to convert correlational findings into causal statements about modality transfer.

## Methodology

**Training Protocol**: Decoder-only Llama-3-like models (default 1.5B, 2.3B with split FFNs) trained on DCLM (language) and SSTK (image-text pairs). Uses SwiGLU, RoPE (θ=500,000), pre-RMSNorm, GQA, QK-norm, FlashAttention, and x-prediction for flow matching (velocity $v = (x_0 - x_t)/(1-t)$).

**Four visual tokenization configurations**:
1. **RAE (default)**: Frozen SigLIP-2 ViT-400m/14 encoder, flow matching in SigLIP latent space.
2. **Raw Pixels**: No pretrained encoder; patchified 224×224 images via 14×14 convolution.
3. **CLIP + VAE**: SigLIP-2 for understanding, SD3 VAE latent space for generation.
4. **AR (UniTok)**: Autoregressive discrete codes with cross-entropy objective.

**Evaluation**: 11 language benchmarks (ARC, BoolQ, HellaSwag, etc.), 16 VQA benchmarks across four axes (General, Knowledge, OCR & Chart, Vision-Centric), and generation metrics (GenEval, DPG-Bench, CLIP similarity, diffusion loss).

**Controlled experiments**:
- **Knowledge flow**: Systematic mixtures with fixed base allocations and swept ratios.
- **Complexity progressions**: Seven rungs of visual complexity (backgrounds → noise → OCR → SSTK) and three rungs of language complexity (simple → nonsense → DCLM).
- **Architecture ablations**: dense, split_ffn, split_ffn_attn, split_ffn_norm, split_all.
- **Timing sweeps**: Pure-language warm-up from 0 to 800B tokens before unified training.
- **Synthetic CLEVR testbed**: Concept ablation across color, shape, spatial relation, size, and count.

## Empirical Validation / Results

**Finding 1: Asymmetric Knowledge Flow (Real-world)**
- Language data universally improves both visual understanding and generation (monotonic gains in all axes).
- Visual understanding data significantly benefits generation but mildly degrades pure language performance.
- Visual generation yields neutral effects on language and understanding.

**Finding 2: Concept-Dependent Transfer (CLEVR)**
- **Low-level concepts** (color, shape) strictly fail zero-shot transfer in both directions.
- **Structural concepts** (relation, size, count) show asymmetric transfer: understanding → generation succeeds, generation → understanding largely fails.
- **Latent priors**: Generation provides strong latent priors for understanding (Δ mean acc +0.133 to +0.273), but understanding provides minimal priors for generation (Δ ≈ 0).

**Finding 3: Synergy vs. Competition**
- **Simple tasks act as cross-modal boosters**: Backgrounds improve language ppl by −0.211; simple language improves vision loss by −0.0153.
- **Complex tasks induce capacity competition**: Video/SSTK degrade text ppl by +0.052/+0.075.
- **Architecture**: Dense models suffer severe competition (ppl +0.272). Split-FFN mitigates competition while preserving synergy (ppl −0.211, vision loss −0.0168). Shared attention/norm drives synergy; full isolation matches unimodal baselines.
- **Generalization**: Synergy occurs across all four vision tokenizer designs (RAE, Raw Pixels, CLIP+VAE, AR).

**Finding 4 & 5: Early Unification & Vision Laziness**
- Delaying vision integration (up to 800B language tokens) monotonically degrades all visual capabilities while providing diminishing language returns.
- Sequential training (all six orderings) fails to match joint training; replay buffers only partially mitigate.
- **Vision laziness**: Four mechanistic measurements (training-time activation L2, embedding norms, inference-time activations, attention fraction on image tokens) all show that longer language warm-up reduces the vision pathway's commitment and integration.

**Finding 6: Recipes & Scaling**

| Model | Lang PPL ↓ | Lang Acc ↑ | VU Avg ↑ | DPG ↑ | GenEval ↑ | DiffLoss ↓ |
|---|---|---|---|---|---|---|
| Balanced Recipe | 11.97 | 52.86 | 41.42 | 0.676 | 0.467 | 0.261 |
| Dense Model | 12.14 | 52.03 | 40.49 | 0.667 | 0.459 | 0.266 |
| Late-Fusion | 12.25 | 51.78 | 40.66 | 0.672 | 0.471 | 0.269 |
| **Full** | **11.67** | **54.31** | **43.08** | **0.689** | **0.482** | 0.272 |

The optimal data mix from grid search is **L70/U25/G5** (Language/Understanding/Generation), achieving strong generation with only 5% of tokens allocated to generation.

## Theoretical and Practical Implications

- **Theoretical**: Provides causal evidence for asymmetric knowledge flow, resolving debates about whether generation helps understanding (it does, but only as latent priors for dense-feature tasks). Supports decoupled architectural designs (MoT/MoE) while showing shared attention/norm is critical for synergy.
- **Practical**: Derives ready-to-use recipes: (1) heavily skew data toward language/understanding (~70/25/5), (2) use MoE with modality-specific FFNs plus shared attention/norm, (3) unify modalities from scratch. These recipes achieve strong performance with only 5% of the compute budget for generation.
- **Explanatory**: "Vision laziness" mechanistically explains why many multimodal LLMs appear "blind"—their vision pathways are under-optimized due to late alignment, over-relying on language priors.

## Conclusion

This paper provides a principled foundation for understanding and scaling multimodal pretraining through four core findings: asymmetric knowledge flow, task-complexity-dependent synergy, the necessity of early unification, and efficient asymmetric data recipes. The authors validate these insights at scale (13.5B MoE, 2T tokens), demonstrating that strong language and understanding priors can effectively bootstrap visual generation with minimal generative tokens. Future directions include exploring the benefits of late-stage generation up-weighting, further scaling validation, and extending the physics framework to additional modalities (audio, video).

---

_Markdown view of https://picx.dev/p/12bbbP, served by PicX — AI-generated visual whiteboard summaries of research papers._