# Scalable Visual Pretraining for Language Intelligence

> Visual Pretraining from raw document images outperforms text-only pretraining on scientific reasoning with 75% fewer tokens.

- **Source:** [arXiv](https://arxiv.org/abs/2607.09657)
- **Published:** 2026-07-14
- **Permalink:** https://picx.dev/p/0L1czI
- **Whiteboard:** https://picx.dev/p/0L1czI/image

## Summary

## Summary (Overview)

*   **Proposes Visual Pretraining (VP):** a framework where large language models (LLMs) learn directly from raw visual documents (e.g., PDF pages) via next visual latent prediction, bypassing the conventional text-extraction pipeline.
*   **Key finding:** VP consistently outperforms matched text-only pretraining (TP) on scientific reasoning benchmarks (e.g., GPQA, AIME, MMLU-Pro) across multiple LLM backbones (Qwen, Llama) and model sizes.
*   **Efficiency:** VP achieves these gains while using only ~25% of the token budget of text pretraining (20B visual tokens vs. 80B text tokens), and scales favorably with more training data.
*   **Cross-modal emergence:** Without any paired image-text supervision, VP improves cross-modal alignment and multimodal reasoning benchmarks (e.g., ChartQAPro, MathVista), demonstrating that visual self-supervision transfers to language tasks.
*   **Visual grounding:** The advantage of VP grows with the visual-structure density of pages (figures, equations, tables), confirming that the benefit stems from modeling visual content that text extraction loses.

## Introduction and Theoretical Foundation

The paper challenges the prevailing assumption that language intelligence must be learned from plain text. While cognitive science shows that humans use visual cues (diagrams, layouts, mathematical notation) for reasoning, existing pretraining pipelines discard these cues by converting visually rich documents (e.g., scientific PDFs) into linear text via OCR or parsing. This conversion is inherently lossy for figures, equations, tables, and spatial layouts. The **Platonic Representation Hypothesis** [10] further suggests that representations from different modalities converge toward shared abstractions, implying that raw visual documents contain richer structure than their textualized forms.

Existing multimodal models either treat vision as a conditioning context (not integrated into the predictive objective) or rely on image-text pairs for supervision. **Visual Pretraining (VP)** instead uses the raw visual document as the prediction target itself, training the shared autoregressive backbone to predict the next visual latent in a frozen visual feature space. This approach preserves the native visual structure and avoids the need for any text extraction or paired supervision.

## Methodology

**Sparse Document Representation:**  
Given a rendered document page $\mathcal{I}$, a frozen vision tower $E_v$ extracts a sequence of visual features:

$$
\mathcal{Z} = E_v(\mathcal{I}) = (z_1, \dots, z_N). \tag{1}
$$

Blank regions (margins, whitespace) are removed using patch-level statistics (pixel variance, luminance). The retained foreground features are ordered in raster scan to form a sparse sequence:

$$
\mathcal{U} = \text{Raster}\{z_i : m_i = 1\} = (u_1, \dots, u_L), \quad L \ll N. \tag{2}
$$

Each foreground feature is projected into the LLM’s hidden space via a learned linear projection, and position indices are reassigned according to raster order.

**Next Visual Latent Prediction:**  
The sparse visual sequence is fed into the LLM under a causal attention mask. At position $t$, an output projection head maps the LLM hidden state back to the frozen visual-latent space to produce a prediction $\hat{\mathbf{z}}_{t+1}$. The training objective is a contrastive (InfoNCE) loss:

$$
\mathcal{L}_{\text{VP}} = -\frac{1}{|\mathcal{B}|} \sum_{i \in \mathcal{B}} \log p_{ii}, \quad p_{ij} = \frac{\exp(\text{sim}(\hat{\mathbf{z}}_i, \mathbf{z}_j) / \tau)}{\sum_{k \in \mathcal{B}} \exp(\text{sim}(\hat{\mathbf{z}}_i, \mathbf{z}_k) / \tau)}. \tag{3}
$$

Other visual features in the batch serve as negatives. This encourages the model to predict the correct next document feature while distinguishing it from other patches.

**Joint Text and Visual Pretraining:**  
The final training objective combines standard text next-token prediction ($\mathcal{L}_{\text{CE}}$) with the visual latent prediction:

$$
\mathcal{L} = \lambda_{\text{text}} \mathcal{L}_{\text{CE}} + \lambda_{\text{vis}} \mathcal{L}_{\text{VP}}. \tag{4}
$$

Text and visual examples are interleaved during training. The LLM, visual input projection, and prediction head are updated, while the vision encoder remains frozen. Multiple foreground sequences are packed into fixed-length contexts with cross-sample attention masking.

**Training Setup:**  
The same starting checkpoint, non-PDF text corpus, optimization recipe, and SFT stage are used for both VP and TP. The only difference is the representation of the additional scientific-PDF corpus:  
- **TP:** MinerU2.5-parsed text (~80B tokens)  
- **VP:** rendered page images, filtered into sparse foreground visual sequences (~20B tokens)  

Both use the same underlying PDF documents; the token count reflects the representation compactness.

## Empirical Validation / Results

**Effectiveness – Scientific Reasoning (Table 1):**  
VP consistently improves over the matched TP baseline across all backbones and text-only reasoning benchmarks. Gains are largest on GPQA (up to +3.22 points) and MMLU-Pro (up to +2.1 points), while HLE shows only marginal improvements (up to +0.97 points), indicating that VP benefits knowledge-grounded reasoning rather than purely multi-step reasoning.

**Table 1: VP improves text-only scientific reasoning under matched document sources.** Scores are reported on text-only reasoning benchmarks (pass@1 for MMLU-Pro and HLE, pass@8 for GPQA, average over 32 runs for AIME-25).

| Method | MMLU-Pro | GPQA | AIME | HLE | MMLU-Pro | GPQA | AIME | HLE |
|--------|----------|------|------|-----|----------|------|------|-----|
| **Qwen 3.5** | | | | | **Qwen 3** | | | |
| Base | 82.31 | 77.84 | 81.56 | 14.39 | 81.21 | 75.06 | 75.44 | 10.59 |
| Text Pretraining | 83.91 | 76.24 | 89.58 | 15.70 | 81.52 | 74.94 | 74.99 | 11.39 |
| **VP (Ours)** | **85.09** | **79.29** | **90.21** | **16.67** | **81.94** | **77.08** | **76.98** | **11.77** |
| **Llama 3.2 Vision** | | | | | **Llama 3.1** | | | |
| Base | 47.24 | 27.46 | 8.02 | 6.41 | 59.48 | 39.71 | 24.17 | 6.75 |
| Text Pretraining | 50.60 | 30.24 | 13.75 | 6.08 | 60.64 | 43.88 | 23.65 | 6.79 |
| **VP (Ours)** | **51.52** | **33.08** | **18.54** | **7.00** | **62.77** | **47.10** | **24.27** | **7.17** |

**Scalability (Figure 2):**  
- VP reaches a lower final SFT loss compared to TP, despite similar CPT loss.  
- Normalized gains over TP increase with training tokens: 1.27× on MMLU-Pro, 2.02× on GPQA, 2.88× on AIME-25 (Figure 2b).  
- Visual feature cosine similarity (proxy for prediction quality) correlates strongly with downstream gains (Figure 2d).  
- Gains are largest on high-visual-structure-density pages (Figure 2c), confirming visual grounding.

**Efficiency (Figure 3):**  
- VP uses 20B visual tokens vs. 80B text tokens for the same PDF corpus.  
- The best trade-off is achieved with 8,192 visual tokens per batch (1× setting).  
- Lowering rendering resolution (e.g., 560px) preserves strong performance, outperforming TP.

**Cross-Modality (Table 2):**  
Without any paired supervision, VP improves cross-modal alignment (centroid separation drops from 1.665 to 0.661; cosine similarity rises from 0.631 to 0.907) and multimodal benchmark performance. Gains are largest on visually heavy benchmarks (e.g., +5.4 on ChartQAPro for both backbones).

**Table 2: VP improves cross-modal alignment and multimodal transfer without labeled multimodal pretraining data.**  
(a) Alignment metrics on 100 held-out scientific document image–text pairs before and after VP.  
(b) Multimodal benchmark performance (pass@1) for native multimodal models.

| (a) Cross-modal alignment | Metric | Original | VP (Ours) | Δ |
|---------------------------|--------|----------|------------|----|
| Global | Centroid Sep. ↓ | 1.665 | 0.661 | -1.004 |
| | Cosine Sim. ↑ | 0.631 | 0.907 | +0.276 |
| Structural | Linear CKA ↑ | 0.657 | 0.745 | +0.088 |
| Local | Mutual k-NN@1 ↑ | 0.140 | 0.310 | +0.170 |
| | Mutual k-NN@5 ↑ | 0.288 | 0.420 | +0.132 |
| | Mutual k-NN@10 ↑ | 0.395 | 0.496 | +0.101 |

| (b) Multimodal benchmarks | Method | MMMU-Pro | SFE | ChartQAPro | MathVista |
|---------------------------|--------|----------|-----|------------|-----------|
| **Qwen 3.5** | Base | 71.39 | 53.41 | 57.99 | 84.30 |
| | Text Pretraining | 72.14 | 53.09 | 56.42 | 85.50 |
| | **VP (Ours)** | **73.87** | **56.57** | **61.80** | **86.70** |
| **Llama 3.2 Vision** | Base | 28.55 | 28.86 | 20.95 | 39.80 |
| | Text Pretraining | 28.21 | 29.36 | 22.23 | 39.60 |
| | **VP (Ours)** | **29.19** | **31.08** | **27.67** | **44.40** |

**Reasoning Cues:** A qualitative attention analysis shows that VP’s visual-token reasoning attends to the same semantic regions (question constraints, intermediate steps) as text-based reasoning, suggesting that VP exposes reasoning-relevant visual evidence to the shared LLM.

## Theoretical and Practical Implications

*   **Theoretical:** VP challenges the text-only assumption of language model pretraining, providing evidence that visual structure (layouts, equations, figures) encodes knowledge that is lost when textualized. The results support the Platonic Representation Hypothesis: visual and textual representations converge when trained jointly, benefiting downstream reasoning.
*   **Practical:** VP is a scalable, efficient alternative to text pretraining for visually rich domains (e.g., scientific documents). It uses only 25% of the token budget, requires no image-text pairs or OCR, and can be integrated into existing pretraining pipelines as a complementary objective. VP also naturally improves multimodal capabilities without multimodal supervision.
*   **Limitations:** VP is not independent of language pretraining—it relies on a text-pretrained backbone and a frozen vision encoder. The study focuses on high-knowledge-density scientific PDFs; transfer to natural images or video remains open.

## Conclusion

VP demonstrates that foundation models can learn effectively from raw visual documents via next visual latent prediction, outperforming text-only pretraining on scientific reasoning, scaling efficiently with data, and improving cross-modal alignment without paired supervision. The findings suggest that visual pretraining provides a complementary pathway to text pretraining, particularly for visually native corpora. Future work may explore better coordination of visual and text decoding, extension to broader visual corpora, and the potential for a primarily visual pretraining paradigm with lightweight text alignment.

---

_Markdown view of https://picx.dev/p/0L1czI, served by PicX — AI-generated visual whiteboard summaries of research papers._