Summary (Overview)

• SynthDocBench is a fully synthetic, controlled benchmark for long-context visual document understanding, designed to systematically vary document length, layout complexity, modality composition, and question difficulty as independent axes.
• The benchmark comprises 200 synthetic reports (average 51.1 pages, 16.7 charts, ~20,500 words) and 1,788 questions across three targeted subsets: chart-reading, cross-modal, and complex multi-hop.
• Evaluation of seven frontier VLMs reveals three previously unobservable failure modes: (1) sharp degradation with increasing evidence complexity and reasoning depth, (2) systematic positional sensitivity – the middle third of a document is hardest for five of six models, and (3) collapse of precise chart-reading accuracy in long-document contexts.
• Gemini-3.1-Pro leads overall (0.725 ACC), followed by Qwen3.5-VL-122B (0.655), while GPT-4o (0.386) and InternVL3-78B (0.383) are statistically indistinguishable.
• An OCR+text-only baseline confirms that chart questions genuinely require pixel-level visual decoding (vision wins by 46 pp over OCR on chart-reading), while complex multi-hop questions are largely text-recoverable.

Introduction and Theoretical Foundation

Real-world documents interleave text, tables, charts, and complex layouts across dozens of pages, demanding both long-range retrieval and cross-modal reasoning. Existing benchmarks such as DocVQA, ChartQA, and MMLongBench-Doc have driven progress but suffer from a diagnostic blind spot: when a model fails on a real document, it is difficult to attribute the failure to a specific cause (e.g., length, layout, modality, question difficulty). These factors co-vary in natural documents and cannot be disentangled.

Synthetic benchmarks have a long history of enabling controlled decomposition (e.g., bAbI, CLEVR, RAVEN, PuzzleVQA) by trading ecological validity for interpretability. SynthDocBench applies this principle to long-context visual document understanding, where confounds are most severe. The key theoretical motivation is that synthetic control enables attribution of failures to specific causes rather than an opaque bundle of confounds.

Methodology

Synthetic Document Generation

A three-stage pipeline (Figure 2 in the paper) maps a topic seed $\tau$ to a styled visual report $D$ and a structured QA manifest $M$:

1. Content generation: LLM-based construction of a semantic backbone with section boundaries, data-bearing spans, and content units.
2. Design and visual grammar: A layout archetype $a \in \mathcal{A}$ is sampled with probability 0.6 from a topic-conditioned distribution and 0.4 uniformly at random (to prevent spurious topic–layout correlations). Six archetypes are used: Magazine, Dashboard, Academic, Editorial, Infographic, Brutalist.
3. Grounded visualization synthesis: Each chart is generated in two aligned forms: a visible D3.js rendering and a structured metadata object $V_k$ recording axes, data points, and derived insights. This dual-layer formulation makes ground truth deterministic by construction.
4. Validation and assembly: Numeric values are recomputed and corrected; the report is assembled as HTML and rendered to PDF via Playwright.

QA Generation

The pipeline converts each report into structured QA items via evidence recovery, synthesis, and generation with validation. Three question families are produced:

• Chart-reading (597 questions): direct value lookup, comparison, trend/pattern from a single chart or table.
• Cross-modal (594 questions): joint reasoning over textual and visual evidence (e.g., verify a claim with a chart, integrate sources, compare representations).
• Complex multi-hop (597 questions): composition across 2–4 evidence units, with difficulty levels L1–L5 (see Table 13, paper).

Each question carries a difficulty label L1–L5 and a structured evidence trace. A validation stage filters weak items; manual review of 100 samples yields >96% acceptance rate.

Benchmark Statistics

• 200 reports, 1,788 questions.
• Average 51.1 pages, 16.7 charts, ~20,568 words.
• 24 chart types, 6 layout archetypes.
• Unimodal, tightly concentrated distributions (page count, word count, chart count) – enabling controlled ablations.

Evaluation Setup

• Vision-only protocol: models receive only rendered page images (144 DPI, concatenated into 5-page vertical strips, max 7,900 px, 4 MB). No HTML, metadata, or OCR.
• Judge: GPT-5 scores each candidate answer against the deterministic reference using a 0–10 rubric (Table 6). Threshold accuracy is defined as:

$$ACC(f_\theta) = \frac{1}{|Q_{valid}|} \sum_{q \in Q_{valid}} \mathbb{1}[J(\hat{a}_q, a^*_q) \geq 6]$$

where $\tau = 6$ ("core answer correct"). Parse failures receive score $-1$ and are excluded.

• Judge validation: GPT-5 and Gemini-as-judge agree within 3.5 ACC points (Pearson $r \geq 0.94$ across all question types), confirming robustness.

Empirical Validation / Results

Main Results (Table 2)

• Chart-reading is easiest; cross-modal is hardest.
• Ranking: Gemini > Qwen3.5 > Qwen3 > GPT-5.4 > GPT-4o ≈ InternVL3 > Claude > Qwen2.5-7B.
• Bootstrap 95% CIs ≤ ±0.023 overall; all pairwise gaps exceed combined CI half-widths.

Difficulty-Stratified Results (Table 3)

All models except Gemini-3.1-Pro degrade monotonically toward L5. Claude-Sonnet-4.5 drops 23 pp (L1→L5); Gemini stays flat (0.670–0.784). GPT-4o shows an anomalous L1 dip (0.271), suggesting precise value extraction is harder than compositional reasoning.

Positional Bias (Table 4)

Chart-reading ACC is bucketed by relative position ($p = k/K$) into equal thirds. The middle third is hardest for 5 of 8 models, dropping 5–18 pp below Early. Notable patterns:

• Claude-Sonnet-4.5: steepest monotonic Early→Late decline (−11.7 pp).
• Qwen3.5-VL-122B: steepest Early→Middle drop (−18.5 pp), partial recovery in Late.
• Gemini-3.1-Pro: U-shaped pattern (lost-in-the-middle effect).

OCR Baseline (Figure 5)

GPT-4o vision vs. OCR+text-only:

• Complex multi-hop: OCR 0.798 vs. vision 0.360 (text-recoverable).
• Chart-reading: OCR 0.297 vs. vision 0.457 (genuine visual decoding needed).
• 46 pp gap between OCR chart ACC and Gemini’s 0.759 isolates visual perception as the primary bottleneck.

Hard Failure Analysis

109 questions where all six models score ≤3. Cross-modal failures dominate (58 of 109). Visual hallucination is the most common error category: models return plausible-looking values absent from the ground-truth chart, concentrated on dense charts, dumbbell plots, and multi-series comparisons.

Ablation Studies (Table 5)

• Pages per strip: Gemini ACC increases monotonically from 0.369 (1 page) to 0.792 (10 pages); cross-modal benefits most (0.339→0.707).
• DPI: 144 DPI is optimal; higher resolution degrades due to JPEG compression under 4 MB API cap.
• Prompting: Chain-of-thought yields modest gains; no-prompt condition sometimes improves complex-question ACC (e.g., Claude +17 pp), suggesting the concise-answer framing constrains multi-step reasoning.

Theoretical and Practical Implications

• Three concurrent failure modes are identified that existing benchmarks cannot surface:

  1. Sharp degradation with evidence complexity and reasoning depth (L1→L5).
  2. Systematic positional sensitivity: the middle third of a document is hardest, and five of six models show a negative Early→Late trend (steepest decline: 8.3 pp). This extends the "lost-in-the-middle" effect to multimodal long-context understanding.
  3. Collapse of precise chart-reading accuracy in long-document contexts, even for models that perform well on isolated chart benchmarks (e.g., ChartQA, ChartQAPro).

• The benchmark provides clean, controlled signals for attribution: failures can be assigned to specific factors (length, modality, difficulty, position) rather than an opaque bundle of confounds.

• Practical implications for model development: the results suggest that current VLMs may be overfitting to benchmark artifacts. The wide gap between single-page and multi-page performance (best model only 57% on MMLongBench-Doc) indicates that long-range dependency tracking, retrieval over dispersed evidence, and cross-modal grounding remain open challenges.

• The OCR baseline demonstrates that chart questions genuinely require visual decoding, while complex multi-hop questions are largely text-recoverable – informing architectural decisions about modality fusion.

Conclusion

SynthDocBench is a fully synthetic, controlled benchmark for long-context visual document understanding that enables the first systematic decomposition of VLM failure modes. By independently varying document length, layout complexity, modality composition, and question type, the benchmark reveals three concurrent failure modes: sharp degradation with reasoning depth, positional sensitivity (middle-third hardest), and collapse of chart-reading in long contexts. These findings highlight a clear disparity between benchmark performance and genuine long-context visual reasoning ability.

Future work includes extending the benchmark to broader multimodal reasoning settings (tables, forms, mixed-layout reports), longer contexts, and more diverse reasoning tasks (multi-hop aggregation, cross-document grounding). The authors also note a potential confound: rendered D3.js charts may be stylistically consistent with Gemini’s training distribution, warranting validation with alternative rendering backends.

Reproducibility: Code and data are publicly released (GitHub, HuggingFace). All hyperparameters, prompts, and evaluation configurations are fully disclosed.