Summary (Overview)

• PureDocBench is a new document parsing benchmark with 1,475 pages × 3 tracks (clean, digitally degraded, real-degraded) = 4,425 images, covering 10 domains and 66 subcategories, with annotations programmatically derived from the same HTML/CSS source used to render images.
• The authors audited OmniDocBench (the de facto standard) and confirmed 2,580 annotation errors (12.08%) across a six-class taxonomy, raising concerns about annotation quality, score saturation, and contamination risk.
• Evaluating 40 models (pipeline specialists, end-to-end specialists, general VLMs), the best model scores only ~74/100, with a 44.6-point best-to-worst spread — document parsing is far from solved.
• Specialist parsers with ≤4B parameters rival or surpass general VLMs 5–100× larger, but formula recognition remains a shared bottleneck — no model exceeds 67% on track-averaged Avg-Formula.
• General VLMs are more robust to degradation (losing 0.99/8.52 Overall points under digital/real degradation vs. 4.90/14.21 for pipeline specialists), causing ranking reversals that make clean-only evaluation misleading.

---

Introduction and Theoretical Foundation

Document parsing recovers not only text but also structural information (table row/column structures, formula LaTeX, reading order) from document images. The past year has seen over 20 open-source specialist models, falling into two design philosophies:

1. Multi-stage specialists: explicit layout-then-content workflow (e.g., MinerU2.5, Dolphin-v2)
2. Fully end-to-end (E2E) specialists: single forward pass from page image to complete markdown (e.g., FireRed-OCR, HunyuanOCR, DeepSeek-OCR)

General-purpose VLMs (e.g., Qwen3-VL, Gemini-3.1-Pro) are also widely used for document parsing.

The authors identify four pressing issues with OmniDocBench, the current de facto standard:

• (P1) Annotation errors: 2,580 confirmed errors (12.08%) across 21,353 evaluator-scored blocks, falling into six categories: extra words, hallucination, missing words, table structure errors, category mismatch, and OCR residual errors.
• (P2) Score saturation: Top models score above 90% with inter-model gaps compressed to 1–2%, smaller than the annotation-error rate.
• (P3) Narrow coverage: 9 categories miss high-frequency enterprise types (financial invoices, medical records, legal contracts, logistics documents) and contain no photographically captured or physically degraded pages.
• (P4) Contamination risk: Data has been public since late 2024; 20+ models released since may have accessed it during training.

---

Methodology

Data Construction

PureDocBench's design rests on three axes:

(A) Taxonomy: 10 top-level domains and 66 fine-grained subcategories, including enterprise types absent from existing benchmarks.

(B) Triple-version design: Each page is rendered in three versions (clean, digitally degraded, real-degraded), forming a 1:1:1 evaluation set of 4,425 images. All three versions share the same ground-truth annotations since degradation only affects image quality.

• Digital degradation: 10 scene templates simulating artifacts (aged archive, book binding, multi-gen photocopy, fax/thermal print, ink bleed, JPEG compression, uneven lighting, geometric distortion, noise-blur combo)
• Real degradation: Four physical acquisition chains: (i) phone capture of printed pages, (ii) phone capture of photocopies, (iii) phone-of-monitor screen photography, (iv) screenshot with social-media compression

(C) Construction pipeline (three stages):

1. Meta-prompt design: For each of the 66 subcategories, author a meta-prompt encoding target document type, layout style, content domain, language, and element composition.
2. LLM HTML/CSS generation: LLMs emit complete self-contained HTML/CSS source files with realistic text, tables, LaTeX formulas, and varied layouts; each passes a rapid human screen.
3. Rendering & annotation extraction: Validated source is rendered to high-resolution PNG via a browser engine; an LLM-assisted source-extraction pass emits annotations following the OmniDocBench format (text, table HTML, LaTeX formulas, reading order).

Metrics

The evaluation adopts the OmniDocBench framework:

• TextEdit↓: page-level normalized edit distance
• FormulaCDM↑: render-based character detection matching for formulas
• TableTEDS↑: tree-edit-distance similarity on table HTML
• ReadOrderEdit↓: edit distance on reading order

Per-track Overall is computed as:

$$\text{Overall} = \frac{(1-\text{TextEdit}) \times 100 + \text{FormulaCDM} + \text{TableTEDS}}{3}$$

Avg₃ is the mean of the three track Overalls (Clean / Digital / Real).

Models

40 models evaluated across three architecture families:

• 10 Pipeline / multi-stage specialists
• 15 Fully end-to-end specialists
• 15 General-purpose VLMs

All models use officially recommended inference configurations (greedy decoding, T=0 where supported) on NVIDIA A100-80GB GPUs.

---

Empirical Validation / Results

Main Results

Document parsing is far from solved. The best Avg₃ score is ~74, with a 40-model mean of ~61 and a best-to-worst spread of ~44% — far wider than OmniDocBench's ~12% top-cluster compression.

Specialist parsers dominate cost-effectiveness. A 1.2B Pipeline specialist (MinerU2.5-Pro) ties Kimi K2.6 (1T total / 32B active) on Avg₃. Within Qwen3.5, the 397B-A17B MoE does not surpass the 122B-A10B MoE.

Key leaderboard results (top performers per architecture):

Degradation Robustness

Across all 40 models:

• Pipeline specialists: lose 4.90/14.21 Overall points under digital/real degradation
• E2E specialists: lose 4.62/13.48
• General VLMs: lose only 0.99/8.52

Rankings reverse accordingly: Clean champion FD-RL is overtaken by Qwen3.5-122B-A10B on Digital and Avg₃; Gemini-3.1-Pro climbs from Clean rank 18 to the top of Real.

Sub-Metric Analysis

Three track-averaged sub-metrics are defined:

• $\text{Avg-Text} \equiv \text{avg}_{\text{tracks}}(1-\text{TextEdit}) \times 100$
• $\text{Avg-Table} \equiv \text{avg}_{\text{tracks}}(\text{TableTEDS})$
• $\text{Avg-Formula} \equiv \text{avg}_{\text{tracks}}(\text{FormulaCDM})$

Avg-Formula is the largest single gap. Using FD-RL as reference, its distance to perfect decomposes as:

• 23.5% on Avg-Text
• 19.5% on Avg-Table
• 35.3% on Avg-Formula (45% of missing points)

No model exceeds 67 on track-averaged Avg-Formula, and Avg-Formula stays flat across a 30× Qwen parameter span.

Architecture complementarity (top-4 mean per group):

Case Studies

Case 1 (academic lab report, MinerU2.5-Pro):

• E1: Subscript/superscript recognition errors (e.g., m³ → mˆ3, $f_n$ garbled)
• E2: Sidebar note symbol errors (arrow → "errors")
• E3: Silent omission of entire document section

Case 2 (business datasheet, Logics-Parsing-v2):

• E1: Header metadata omission (brand and product model missing)
• E2: Reading order errors (tables mixed across sections)
• E3: Technical symbol mutation ($L_\sigma \leq 15$ nH → unit error)

Key observations: failure modes are domain-specific (STEM → notation fidelity; business → structural integrity), several failures are silent under Avg₃, and leaderboard rank does not guarantee correct reproduction.

---

Theoretical and Practical Implications

Benchmark Design Implications

• Source-traceability enables verifiability: annotations and images share the same HTML/CSS source, allowing third-party verification.
• Contamination resistance via regeneration: the pipeline can be re-rolled on demand, addressing the shortening model-release cycle.
• Scalability: both intra-category and inter-category expansion require only new meta-prompts.

Model Development Implications

1. Formula recognition is the field's shared bottleneck — closing it requires LaTeX-/STEM-rich pretraining and dedicated formula tokenizers, not larger backbones.
2. Architectures have complementary strengths: Pipelines can close most headroom by upgrading the formula module; VLMs benefit most from stronger text and reading-order pretraining.
3. Clean-only evaluation misleads deployment decisions — degradation robustness varies dramatically across architectures, and general VLMs' web-scale pretraining gives them an edge on real-world capture noise.

Practical Deployment Implications

For deployment, models should be selected based on per-metric profile matching the workload's dominant content type, not the highest Avg₃. The case studies demonstrate that even top-ranked models exhibit silent failures in complex documents.

---

Conclusion

PureDocBench provides a source-rendered, contamination-resistant benchmark covering 10 domains, 66 subcategories, and 1:1:1 triple-track images. Evaluating 40 models reveals:

1. Document parsing is far from saturated (Avg₃ spread 44.6%, mean ~61)
2. Specialist parsers with ≤4B parameters match general VLMs on clean data yet lag in robustness
3. Ranking reversals between clean and degraded settings confirm clean-only evaluation misleads real-world deployment

Limitations and future directions:

• (i) Current metrics ignore typographic emphasis, block-level completeness, and layout fidelity — richer metrics separating recognition noise, semantic omission, structural drift, and reading-order failure are needed
• (ii) Source-rendered generation cannot cover long-tail scenarios (handwritten notes, historical scans, heavily damaged documents) — authentic-document benchmarks remain necessary
• (iii) The benchmark is predominantly Chinese–English — extending to additional languages, scripts, and underrepresented domains will improve applicability

All data, code, generation pipeline, evaluation code, raw predictions from all 40 models, and corrected OmniDocBench annotations are publicly released.