Summary of OvisOCR2 Technical Report

Summary (Overview)

• OvisOCR2 is a 0.8B parameter end-to-end document parsing model that converts a document page image directly into a Markdown representation in natural reading order, covering text, formulas, tables, and visual regions.
• A dual-pipeline data engine combines filtered real-document annotations (from OCR parsers with rule-based normalization and manual spot-checking) with synthetic pages generated from HTML templates, ensuring precise ground-truth alignment.
• The training recipe includes supervised fine-tuning (SFT), reinforcement learning (GRPO) on a 4B teacher branch, on-policy distillation (OPD) into the 0.8B student, and model fusion via weighted parameter averaging.
• On OmniDocBench v1.6, OvisOCR2 achieves a state-of-the-art overall score of 96.58, surpassing all pipeline methods (e.g., PaddleOCR-VL-1.6, MinerU2.5-Pro) and previous end-to-end models.
• On PureDocBench, it achieves the highest Avg3 score of 75.06, and on an in-house benchmark covering long-tail and challenging scenarios, it obtains the best overall performance, especially on handwriting and complex-table subsets.

Introduction and Theoretical Foundation

Document parsing extends plain OCR by preserving page organization, reading order, tables, formulas, figures, and other layout-dependent elements, typically outputting Markdown. Approaches fall into two families:

• Pipeline methods: Decompose parsing into layout analysis, region-level content recognition, and merging. They dominate public leaderboards but suffer from deployment complexity (multiple models with different runtime loads) and error accumulation across stages.
• End-to-end methods: Use a single model to generate the Markdown representation in one pass, simplifying deployment and allowing the model to condition on page-level context. However, they have historically lagged behind pipelines in performance.

The authors argue that the end-to-end approach is more elegant and develop OvisOCR2 by post-training Qwen3.5-0.8B, the smallest model in the Qwen3.5 family. The goal is to achieve state-of-the-art performance with a compact model, requiring both a carefully designed data engine and a training recipe suited to long, structured outputs.

Methodology

Data Engine

The data engine consists of two complementary pipelines (see Figure 2 in the paper):

Real-world data pipeline:

• Processes document images using specialized OCR parsers (PaddleOCR-VL-1.5 or MinerU2.5-Pro) to obtain structured JSON outputs.
• Converts these outputs into a unified Markdown format via rule-based normalization (category validation, text-block merging, LaTeX-style formula normalization, HTML table normalization, visual-region bounding box serialization).
• Applies manual spot-checking on subset samples to verify text correspondence, formula accuracy, table alignment, reading-order consistency, and visual-region placement. Low-quality subsets are removed while acceptable ones are retained.

Synthetic data pipeline:

• Generates both document images and their Markdown targets from the same HTML source, ensuring deterministic, noise-free labels.
• Starts with hard sample mining from failure cases (table-heavy layouts, irregular structures, handwritten regions, etc.) to create reusable HTML templates.
• Uses an agent-based diversification procedure to expand templates into diverse HTML pages with content-level and structure-level variations.
• Markdown ground truth is generated directly from the HTML source via a serializer with document-type-aware reading order (single-column: top-to-bottom, left-to-right; multi-column: column partitioning).
• Pages are rendered using Playwright for browser-based, realistic rendering, with element-level geometry recorded.
• Iterative quality control ensures renderable, category-compliant, and valid Markdown before large-scale generation.

Training Pipeline

The training pipeline is organized in stages (see Figure 3 in the paper):

Supervised Fine-Tuning (SFT):

• Establishes the base policy using the full data mixture (real + synthetic).
• Both Qwen3.5-0.8B and Qwen3.5-4B are trained with full-parameter SFT (0.8B for 2 epochs, 4B for 20% of an epoch). A 16K maximum sequence length and dynamic image-resolution budget are used.

Reinforcement Learning (RL):

• Uses Group Relative Policy Optimization (GRPO) to improve parsing on hard cases using multi-component rewards.
• RL data primarily from synthetic pipeline (where ground truth is accurate) plus a smaller set of high-quality real documents.
• On-policy filtering focuses training on pages where the model can occasionally produce clearly better responses.
• Multi-component reward design (Table 1):

The page-level reward is computed as:

$$R(y, y^*) = \frac{\sum_{c \in \mathcal{C}} a_c(y^*) \, s_c(y, y^*)}{\sum_{c \in \mathcal{C}} a_c(y^*)}$$

where $\mathcal{C} = \{\text{text}, \text{table}, \text{formula}\}$, $a_c(y^*)$ is the availability indicator (1 only if the reference contains evaluable units of that type), and $s_c(y, y^*) \in [0,1]$ is the component score.

• Scalable optimizations: hierarchical parallelism for reward computation (exact matches bypass expensive rendering), object-store-backed reference passing for large visual tensors, and common-prefix masks for long responses.

On-Policy Distillation (OPD):

• Direct RL on 0.8B shows higher KL divergence and unstable table quality (Figure 4). Therefore, the RL-aligned 4B model is used as a teacher.
• The student generates responses; the teacher evaluates on-policy trajectories with token-level distribution supervision.
• The loss uses student top-k reverse KL:

$$\bar{p}_{t,v} = \frac{\pi_\theta(v | c_t)}{\sum_{u \in S_t} \pi_\theta(u | c_t)}, \quad \bar{q}_{t,v} = \frac{\pi_\phi(v | c_t)}{\sum_{u \in S_t} \pi_\phi(u | c_t)}$$ $$\mathcal{L}_{\text{OPD}}(\theta) = \frac{1}{|I|} \sum_{t \in I} D_{\text{KL}}(\bar{p}_t \parallel \bar{q}_t)$$

where $S_t = \text{TopK}_k(\pi_\theta(\cdot | c_t))$ is the student's top-k support. This reduces tensor size from $O(TV)$ to $O(Tk)$ and is mode-seeking, discouraging student probability on tokens the teacher assigns low probability.

Model Fusion:

• Weighted parameter averaging of several candidate OvisOCR2 variants trained with different data mixtures and configurations.

Empirical Validation / Results

OmniDocBench v1.6

Evaluates 1,651 pages across 10 document types, 5 layout types, and 5 language types. Metrics: text edit distance, formula CDM, table TEDS/TEDS-S, reading-order edit distance. Overall score = average of text score (1 - edit distance), formula CDM, and table TEDS.

OvisOCR2 achieves the highest overall score, lowest text edit distance, highest formula CDM, tied highest TEDS, highest TEDS-S, and lowest reading-order edit distance, surpassing all pipeline and end-to-end methods.

PureDocBench

1,475 pages across Clean, Digital, and Real tracks (4,425 images total). Avg3 = mean of three track scores.

OvisOCR2 ranks first on Clean and Digital tracks, and first overall in Avg3. On the Real track, it remains below strong general VLMs (Gemini-3.1-Pro, Qwen3.5-122B-A10B), indicating a direction for future robustness improvement.

In-house Benchmark

1,000 pages covering long-tail scenarios (forms, scanned reports, handwritten annotations, complex tables). Evaluated using the OmniDocBench protocol.

Performance by difficulty level:

Handwriting subset:

Complex-table subset:

OvisOCR2 achieves the highest overall score and lowest missing rate on complex tables, while pipeline methods miss 13–17% of tables during layout parsing (an irrecoverable error).

Theoretical and Practical Implications

• End-to-end superiority: OvisOCR2 demonstrates that a compact end-to-end model can surpass pipeline methods on public benchmarks, contradicting the previous trend where pipelines dominated. The single-pass design simplifies deployment, eliminates error accumulation, and allows the model to leverage page-level context.
• Data engine design: The combination of filtered real-world data and source-aligned synthetic data is crucial. Synthetic data provide clean, controllable, and long-tail coverage, while real data introduce natural visual variation. The manual spot-checking stage ensures quality without relying on costly human annotation.
• Training recipe: RL with multi-component rewards (text, formula, table) addresses structural errors that next-token prediction alone cannot capture. On-policy distillation from a larger teacher (4B) to a smaller student (0.8B) avoids instability of direct RL on compact models. Model fusion further improves performance.
• Practical impact: The 0.8B model is deployable on edge devices with low latency, yet achieves state-of-the-art results. This opens possibilities for real-time document parsing in OCR-heavy workflows like digitization, information retrieval, and document understanding.
• Remaining challenges: Robustness to degraded real-world images (phone captures, photocopies, compressed screenshots) and performance on handwriting remain areas for improvement. The paper suggests future work on these fronts.

Conclusion

OvisOCR2 is a 0.8B end-to-end document parsing model that achieves state-of-the-art results on OmniDocBench v1.6 (96.58 overall) and PureDocBench (Avg3 75.06), outperforming both pipeline and end-to-end methods. Its success is enabled by a dual-pipeline data engine (filtered real annotations + source-aligned synthetic pages) and a multi-stage training recipe (SFT, RL with multi-component rewards, on-policy distillation, model fusion). On an in-house benchmark, it shows consistent leadership across difficulty levels, especially on handwriting and complex-table subsets. Future work will focus on improving robustness to degraded real-world images and further strengthening performance on handwriting-heavy and complex-table documents.