Summary (Overview)

• WeVisDoc is a two-stage data-centric framework for robust end-to-end document parsing, addressing the gap between data coverage and model capability.
• Stage I builds broad coverage from ~40 million records spanning diverse layouts, languages, and acquisition conditions, combining heterogeneous supervision, executable page synthesis, and source-conditioned appearance degradation.
• Stage II uses a held-out diagnostic probe to measure residual errors within fixed visual–structural clusters, guiding targeted data construction and reallocation of a fixed target-token budget.
• Results: WeVisDoc-4B achieves an Overall score of 95.38 on OmniDocBench v1.6 and a mean Overall score of 75.54 across PureDocBench tracks, ranking first among compared end-to-end parsers in all four settings.
• Key finding: Stage II refinement yields larger gains on degraded pages (e.g., +4.03 points for the 4B model on the Real Degraded track), demonstrating the value of capability-aware data allocation.

---

Introduction and Theoretical Foundation

Background and Motivation

Document parsing converts document images into structured text while preserving content, element types, structural organization, and reading order. Despite architectural progress, reliable parsing remains challenging due to variations in language, layout, content density, element composition, and acquisition conditions.

Central Challenge

The paper identifies a critical distinction:

Expanding empirical support and increasing exposure to existing support are distinct interventions.

• Expanding empirical support: collecting, retrieving, or synthesizing new records
• Increasing exposure: replaying or reweighting existing records

A key insight is that residual error alone is insufficient evidence of a coverage deficit—it may arise from unreliable targets, low visual observability, decoding failure, or model limitations that additional nearby examples cannot resolve.

Theoretical Foundation

The training objective is defined as loss per supervised token. For record $i$ with input image $x_i$ and target sequence $y_i = (y_{i,1}, \ldots, y_{i,T_i})$, the parser models the target autoregressively:

$$p_\theta(y_i | x_i) = \prod_{t=1}^{T_i} p_\theta(y_{i,t} | x_i, y_{i,<t}) \tag{1}$$

where $T_i$ is the target length and $y_{i,<t}$ denotes preceding tokens.

With mask indicators $m_{i,t} \in \{0,1\}$ for whether target position $t$ contributes to training:

$$L_i = \sum_{t=1}^{T_i} m_{i,t}, \quad \ell_i(\theta) = -\sum_{t=1}^{T_i} m_{i,t} \log p_\theta(y_{i,t} | x_i, y_{i,<t}) \tag{2}$$

The training objective normalized by supervised target length is:

$$\mathcal{L}(\theta; q) = \frac{\mathbb{E}_{i \sim q}[\ell_i(\theta)]}{\mathbb{E}_{i \sim q}[L_i]} \tag{3}$$

---

Methodology

3.1 Task Formulation

The framework uses a unified document parsing approach where targets use Markdown for text, LaTeX for formulas, and HTML for tables. The same parser and output formats apply to pages, regions, and components.

3.2 Stage I: Broad-Coverage Data Construction

Data Sources and Coverage

• ~40 million records spanning:
  • Content domains: academic papers, textbooks, reports, forms, receipts, news, handwritten notes
  • Layouts: single/multi-column, free-form, sparse/dense pages, formula/table-rich pages
  • Languages: Simplified/Traditional Chinese, English, multilingual pages
  • Acquisition conditions: born-digital PDFs, scans, photographs, screenshots, screen recapture

Source Balancing and Token Accounting

For source $s$ with eligible records $I_s$ and supervised tokens $M_s = \sum_{i \in I_s} L_i$:

$$\omega_s = \frac{M_s^\rho}{\sum_{s'} M_{s'}^\rho}, \quad 0 \leq \rho \leq 1 \tag{4}$$

where $\rho = 1$ follows available token counts, $\rho = 0$ gives equal shares.

For stream-selection probabilities with desired token share $v_g$:

$$q(i) = \sum_{g=1}^{G} \alpha_g q^{(g)}(i), \quad \alpha_g = \frac{v_g / \bar{L}_g}{\sum_{h=1}^{G} v_h / \bar{L}_h} \tag{5}$$

Multi-Model Joint Annotation

Three expert parsers (MinerU2.5-Pro, PaddleOCR-VL-1.6, dots.mocr) generate candidate targets. Normalized edit distance (NED) between experts $m$ and $n$:

$$d_i^{(m,n)} = \frac{ED(\tilde{y}_i^{(m)}, \tilde{y}_i^{(n)})}{\max\{1, |\tilde{y}_i^{(m)}|, |\tilde{y}_i^{(n)}|\}} \tag{6}$$

Decision rules:

• All three experts agree (NED < threshold $\delta_{ann}$): high-confidence annotation
• Two agree, one differs: requires image-based review
• All three differ: withheld from Stage I, reviewed by stronger model + humans for Stage II

Broad-Coverage Synthesis

Executable page programs generate both page images and exact targets via dual compilation from semantic HTML, ensuring exact supervision with shared identifiers aligning image regions and structured targets.

Appearance Degradation

Source-specific acquisition paths (paper/copying/transmission effects, physical acquisition like perspective/curvature/illumination) applied while preserving target validity—views that lose content are rejected or assigned visible-content crop targets.

3.3 Stage II: Capability Diagnosis and Targeted Refinement

Hard-Example Mining

• Document-level: full-page NED between Stage I prediction $\hat{y}_i$ and reliable target $y_i$
• Component-level: identifies failures occupying little page area (e.g., formulas missing despite readable surrounding text)
• Mining from training pool, disjoint from held-out probe $\mathcal{D}_{probe}$

Verification

High-error records undergo target reliability verification before being treated as hard examples, distinguishing annotation errors from genuine model weaknesses.

Targeted Construction and Allocation

• Validated additions, Stage I replay, and audited hard examples form a ~5M-record refinement pool
• All exposure counted against a fixed Stage II target-token budget
• Stage II freezes the visual encoder and updates only the language model

---

Empirical Validation / Results

Benchmark Performance

Key Results

• WeVisDoc-4B ranks first among compared end-to-end parsers in all four settings
• WeVisDoc-2B delivers top-tier performance at half the parameter count
• Stage II improvements are larger on degraded PureDocBench tracks, including a +4.03-point gain for the 4B model on the Real Degraded track
• Stage-wise results are consistent with the benefit of the complete refinement protocol, particularly under degraded acquisition conditions

---

Theoretical and Practical Implications

Theoretical Implications

1. Coverage vs. capability distinction: The paper formalizes that expanding empirical support (new data) and increasing exposure (reweighting existing data) are distinct interventions requiring different diagnostic signals.
2. Token-budget framework: Using supervised target tokens (rather than record count) as the optimization budget provides a more principled accounting when records vary substantially in target length.
3. Residual error ≠ coverage deficit: The framework explicitly separates annotation errors, unreadable content, and genuine model weaknesses before allocating additional data.

Practical Implications

1. Two-stage data-centric pipeline: Provides a reproducible protocol for building robust document parsers without architectural changes.
2. Diagnostic probe design: Held-out probes disjoint from training/mining/evaluation enable reliable measurement of residual weaknesses.
3. Source-aware balancing: Prevents large corpora from dominating training while preserving diversity.
4. Degradation-aware supervision: Validated degraded image–target pairs improve robustness without compromising semantic supervision.

---

Conclusion

WeVisDoc demonstrates that robust end-to-end document parsing requires not just broad data coverage but capability-aware refinement guided by diagnosed residual weaknesses. The two-stage framework—broad coverage construction followed by targeted refinement under a fixed token budget—achieves state-of-the-art results at both 2B and 4B scales.

Key takeaways:

• Broad coverage alone does not specify how to address a parser's remaining weaknesses
• Systematic diagnosis (via held-out probes, multi-model verification, and group-level aggregation) enables effective allocation of training resources
• Degraded acquisition conditions benefit most from capability-aware refinement

Future directions implied by the work include:

• Isolating the individual contributions of Stage II interventions (validated additions vs. replay vs. hard-example reweighting)
• Extending the framework to other document parsing architectures and modalities
• Further scaling the diagnostic and refinement loop for even larger models

Resources:

• Project Page: https://tencent.github.io/WeVisDoc
• GitHub: https://github.com/Tencent/WeVisDoc
• WeVisDoc-4B: https://huggingface.co/Tencent/WeVisDoc-4B
• WeVisDoc-2B: https://huggingface.co/Tencent/WeVisDoc-2B