New Frontiers in Document Corpora: Vision-Native Pretraining
Highlights of This Issue
On the document extraction quality axis, this issue presents a radical option that could reshape corpus construction decisions: instead of parsing PDFs into text and feeding it to the model, directly use rendered page images as training signals. Some work demonstrates that training a shared autoregressive backbone in a vision-native form during the continued pretraining phase can surpass text-based pretraining with a 25% token budget on the same corpus, while improving cross-modal alignment (Scalable Visual Pretraining for Language Intelligence). This extends the idea from Issue 2, "The Announcement Carries the Cue," which treated "how documents are written into the corpus" as a training variable, but pushes the answer from "how structural cues are encoded" to "skipping text extraction altogether." On the same axis, production-grade difficulty-aware curation (Closing Cost-Quality Gap in Document VLMs) provides empirical evidence that "naively generated samples have near-zero signal, and unfiltered open data actually drags down the baseline," offering a difficulty criterion for mid-training data selection.
Data mixture optimization continues to evolve beyond static regression, but paths are diverging. Some work formalizes the mixture problem as causal inference, using proxy runs to estimate conditional average treatment effects and extrapolate to unseen data pools (CausalMix); another work decouples "what to mix" into two orthogonal subproblems: inter-class proportions and intra-class convex allocation, and provides an attributable dataset admission protocol (DecoupleMix). These two approaches respond to the dynamic mixing problem after RegMix-D in Issue 3 from causal deconfounding and hierarchical decoupling perspectives, respectively.
The transfer of "post-training ceiling" as a criterion continues to deepen this issue. Some work proposes a "learnability" signal orthogonal to snapshot solvability, using lightweight probe estimators to construct static sampling priors for RL post-training (Beyond Solvability: Task Learnability as a Static Prior for LLM RL Post-Training); on the synthetic data side, the "capability polarization" mechanism advances collapse diagnosis from the aggregate level to skill granularity (Learning from Synthetic Data without Model Collapse in Iterative Instruction Tuning); and a systematic review organizes post-training reasoning data into a verifier-anchored taxonomy and attribution framework (A Primer in Post-Training Reasoning Data), providing a methodological reference for this criterion transfer.
Open Questions
- Does the 25% token efficiency of vision-native pretraining hold at larger scales and under post-training metrics? Current evidence uses post-SFT benchmarks as the criterion; if measured by RL trainability or pass@k, whether the benefit of "skipping text extraction" still dominates remains undetermined.
- CausalMix's causal deconfounding is effective in the SFT setting, but in pretraining scenarios, proxy signals are noisier and the target domain is broader—can the causal framework transfer to pretraining mixtures, or is it only applicable to post-training data selection?
- For "learnability" as a prior for RL data selection, what is the trade-off between estimation cost and gain? Current evidence is limited to reasoning tasks with binary rewards; does the signal remain stable when extended to continuous rewards and general tasks?
- Can the skill-granularity "capability polarization" diagnosis (KITE) generalize from SFT to the RL phase? Whether the synthetic data excess threshold is consistent across the two training paradigms has not been cross-phase compared.
Papers in this issue
Visual Pretraining from raw document images outperforms text-only pretraining on scientific reasoning with 75% fewer tokens.
Editor's noteAdvances the document extraction quality axis to "vision-native": during continued pretraining, directly use rendered PDF page images as input, train a shared autoregressive backbone with next visual latent prediction objective, without any text extraction or image-text paired supervision. Relative to Issue 2's "The Announcement Carries the Cue" (which treated structural cue encoding as a training variable), the increment is treating the "vision-native form of documents" itself as a training signal, demonstrating that with a 25% token budget on the same corpus, it surpasses text pretraining and improves cross-modal alignment; evidence comes from controlled comparisons at real scale on Qwen3.5 and Llama3.2 Vision backbones. Limitations: corpus recipe not disclosed, visual/text mixing ratios not systematically scanned, and criterion is post-SFT benchmarks without addressing RL trainability.
CAUSALMIX reformulates LLM data mixture optimization as causal inference, using double machine learning to estimate state-conditioned marginal returns that outperform static baselines across data scales, model sizes, and unseen data pools.
Editor's noteAdvances data mixture optimization from static regression to state-conditional causal marginal benefit estimation: uses DML/causal forests on 512 0.5B proxy runs to estimate CATE, and extrapolates to 7B and unseen data pools (LongCoT) without retraining proxies. Relative to RegMix-D's dynamic mixing in Issue 3, the increment is explicitly introducing data state covariates (complexity/difficulty/quality) and causal deconfounding, making the mixture strategy transferable to out-of-distribution data pools. Limitations: SFT setting (not pretraining) and code not released; should be referenced cautiously in pretraining contexts.
DecoupleMix decouples VLM data mixture optimization into inter-class ratio search and convex intra-class allocation, achieving competitive performance at 1/80th the multimodal training tokens of stronger baselines.
Editor's noteDecouples data mixture optimization into two orthogonal subproblems: inter-class proportion search and intra-class convex allocation, and proposes an attributable validation protocol for evaluating new dataset admission under fixed inter-class proportions and budget. Relative to RegMix-D's dynamic mixing in Issue 3, the increment is explicit two-level decoupling and attributable admission mechanism, and advances the evaluation unit from sample level to dataset level; evidence covers 2.5B–10B tokens and 32B models, with proxy-searched proportions transferring across scales without retuning. Limitations: only covers VLMs and data recipes not disclosed.
A 35B-parameter MoE VLM, fine-tuned on difficulty-filtered synthetic data, outperforms models 10× larger while cutting deployment costs by over 80% versus human annotation.
Editor's noteProposes a difficulty-aware data curation pipeline (DADC), filtering open-domain PDFs by layout diversity, factual extractability, and cross-model consistency, and explicitly coupling field-level accuracy with deployment economics; evidence comes from production-grade controlled ablations on a 35B MoE (3B active). Core observation: "naively generated samples are mostly solvable directly by the base model with near-zero signal," and unfiltered open data actually hurts the internal baseline—providing an actionable difficulty-axis criterion for mid-training data selection. Limitations: corpus and code not released.
Learnability, distinct from solvability, is estimated from endpoint reward deltas via TRAJVAL and triples training efficiency in LLM RL post-training.
Editor's noteProposes "learnability" as a task value signal orthogonal to snapshot solvability: uses lightweight probe estimators (TrajVal) on early/late endpoint rewards to construct static sampling priors, and in controlled pre-bin analysis isolates this effect from initial solvability. Relative to methods like BOTS/GRESO that use current pass rate as signal, the increment is explicitly targeting "positive response to continued training" rather than current performance; 1.7B–4B scale, controlled ablations. Limitations: limited to reasoning tasks with binary rewards, and gains are modest in some configurations.
KITE prevents model collapse in synthetic-data instruction tuning by targeting weak skills via DINA-based diagnosis and curating examples near the model's semantic knowledge boundary, achieving sustained monotonic improvement across benchmarks.
Editor's noteAdvances synthetic data collapse diagnosis from aggregate level to skill granularity, proposing the "capability polarization" mechanism (synthetic data strengthens already-strong skills and further weakens weak skills), and accordingly designs a two-stage framework KITE: cognitive diagnosis-driven failure-guided generation + kernel boundary uncertainty-based boundary-aware filtering. Relative to CDS and ToEdit, the increment is explicitly coupling "collapse diagnosis" and "data filtering" to skill boundaries, and validates monotonic improvement with 5 models over 9 generations of long-range evolution. Limitations: scale not reaching preference threshold ceiling, and criterion is post-SFT benchmarks.
Reasoning data quality is verifier-relative and base-relative, not intrinsic, so progress attribution requires documenting the full feedback interface, not just prompt-response pairs.
Editor's noteThe first systematic review of post-training reasoning data, organizing 150+ studies with a verifier-anchored taxonomy and attribution framework, providing counterintuitive empirical findings and attribution metadata fields around four questions (what, utility, construction, scaling). Relative to MidTool/OpenThoughts, the increment is explicitly treating data as the research object, using verification contracts rather than domain classification, and emphasizing recording attribution metadata for future releases. As a review rather than new empirical work, it serves as a methodological reference for the "post-training ceiling" criterion in this direction.
