# Closing Cost-Quality Gap in Document VLMs: Difficulty-Aware Data Curation and Quality-Adjusted Deployment Economics

> A 35B-parameter MoE VLM, fine-tuned on difficulty-filtered synthetic data, outperforms models 10× larger while cutting deployment costs by over 80% versus human annotation.

- **Source:** [arXiv](https://arxiv.org/abs/2609.01575)
- **Published:** 2026-09-05
- **Permalink:** https://picx.dev/p/aiM0C1
- **Whiteboard:** https://picx.dev/p/aiM0C1/image

## Summary

## Summary (Overview)

- **Deployed document-understanding system**: The authors present a Mixture-of-Experts VLM (35B total, 3B active parameters) fine-tuned for production document understanding in regulated industries, deployed on a single H100 GPU.
- **Difficulty-Aware Data Curation (DADC) pipeline**: A novel synthetic data pipeline that filters open-domain Common Crawl PDFs for layout diversity, fact-extractability, and cross-model consistency, retaining only hard, information-rich samples.
- **State-of-the-art results**: The model achieves 0.814 average score across all evaluation groups, outperforming baselines up to 10× larger in total parameters (including reasoning-mode variants) while being deployable within strict latency constraints.
- **Quality-adjusted cost framework**: A closed-form economic model calibrated from production telemetry shows the system reduces expected costs by over 80% against human baseline and by more than 50% against the best competing open-source model.
- **Key insight**: Large baselines that clear quality thresholds are economically unviable—they cost more to serve than the human annotation they replace, while the proposed model closes the cost-quality gap.

## Introduction and Theoretical Foundation

The paper addresses a critical challenge in regulated industries (court judgments, invoices, tax certificates, receipts) where documents must be processed at scale (hundreds of millions of Cyrillic-language documents annually). Four barriers block automation:

1. **Privacy constraints**: Documents contain personal data, ruling out cloud-hosted models
2. **Quality gap**: Open-source VLMs under 10B fall short of production quality
3. **Cost barrier**: Larger models that clear quality thresholds are prohibitively expensive at document-flow scale
4. **Latency constraints**: Reasoning-mode inference is too slow for production SLAs

The theoretical foundation rests on several key observations:
- **MoE efficiency**: Mixture-of-Experts architectures offer dense-model quality at a fraction of active parameters (DeepSeek-VL2, MoE-LLaVA)
- **Data quality over quantity**: Quality-filtered, progressively complex data outperforms raw scaling (Dong et al., 2025)
- **Asymmetric confirmation vs. correction costs**: Human-assisted modes have different costs depending on whether the operator confirms or corrects a prediction

The work builds on Qwen family models (Bai et al., 2025b,a) and related document understanding systems including olmOCR (Poznanski et al., 2025a), VENUS (Zhao et al., 2025), and various cascaded OCR–LLM pipelines.

## Methodology

### Model Architecture and Training

- **Base model**: Qwen3.5-35B-A3B-Base (35B total, 3B active parameters)
- **Fine-tuning**: SFT on 4 nodes with 8 H100 GPUs
- **Four canonical tasks**: schema-based field extraction, closed-set classification, visual element validation (checkboxes, stamps, signatures), and freeform VQA/OCR

### Internal Data

Two production workflows provide naturally aligned supervision:
- **Judicial pipeline**: 60K court PDFs, 20 fields (8 per document)
- **Invoice pipeline**: 150K instances (PDFs, scans, screenshots, photos), 4–5 fields each
- Quality control: overlapping annotation (3 of 5 annotators per item)
- Prompt augmentation: semantically equivalent alternatives, random permutation of key-value pairs

### DADC Pipeline (Synthetic Data)

Built from 300K Common Crawl PDFs (Russian, Belarusian, Ukrainian, Kazakh), the pipeline has six stages:

1. **Text layer validation**: Word-wise consistency check against embedded text layers; retains 40% of pages
2. **Document sampling**: Two-stage VLM judge filter (Qwen2.5-VL-72B-Instruct) for taxonomic classification and fact-extractability scoring
3. **Text-to-image substitution**: Query–response pairs generated from text, then rendered via PyMuPDF
4. **Consistency verification**: Cross-family verifier pool (Qwen3-VL-235B, Qwen3.5-397B, Kimi K2.6) with GPT-OSS 120B judging; retains 35–40% of candidates
5. **Text refinement**: Rewriting annotations into target formats with explicit formatting rules
6. **Rendered document augmentation**: Photometric/geometric perturbations, elastic warps, and spatially varying noise to simulate camera captures and scans

### Cost Model

The framework defines per-document inference cost as:

$$y(m, w) = \frac{G_m \cdot C_{\mathrm{GPU}}}{u \cdot T_{\mathrm{month}} \cdot \mathrm{RPS}_m(w)} \tag{1}$$

where $G_m$ is the number of GPUs, $C_{\mathrm{GPU}}$ is monthly GPU cost, $u \approx 0.5$ is utilization, $T_{\mathrm{month}} = 30 \cdot 24 \cdot 3600$ seconds, and $\mathrm{RPS}_m(w)$ is profiled throughput.

Fields are routed into three modes via thresholds $\theta_{\mathrm{auto}} = 0.95$ and $\theta_{\mathrm{assist}} = 0.125$:
- **Fully automated**: $a_f \geq \theta_{\mathrm{auto}}$ — prediction accepted at inference cost only
- **Human-assisted**: $a_f < \theta_{\mathrm{auto}} \wedge p_f \geq \theta_{\mathrm{assist}}$ — prediction surfaced as pre-fill
- **Fully manual**: $a_f < \theta_{\mathrm{auto}} \wedge p_f < \theta_{\mathrm{assist}}$ — annotation from scratch

Expected assisted-mode cost:

$$c_{\mathrm{a}}(f) = 0.3x \cdot p_f + 1.1x \cdot (1 - p_f) \tag{2}$$

where $x$ is the fully loaded cost of manual annotation, confirmation costs $0.3x$, and correction costs $1.1x$.

Total per-document cost:

$$C_w(m) = y(m, w) + x \cdot |\mathcal{F}_w^{\mathrm{h}}| + \sum_{f \in \mathcal{F}_w^{\mathrm{a}}} c_{\mathrm{a}}(f) \tag{3}$$

Cost reduction ratio against manual baseline:

$$\rho_w(m) = 1 - \frac{C_w(m)}{C_w(\mathrm{baseline})} \tag{4}$$

## Empirical Validation / Results

### Main Benchmark Results

| Model | Single-page | MWS (val) | Multi-page | FT | Avg w/o FT | Avg |
|-------|------------|-----------|------------|-----|------------|-----|
| Qwen2.5-VL-72B-Instruct-AWQ | 0.785 | 0.635 | 0.698 | 0.645 | 0.706 | 0.691 |
| Qwen3.5-35B-A3B-Base | 0.827 | 0.677 | 0.697 | 0.722 | 0.733 | 0.730 |
| Kimi K2.6-Reasoning | 0.747 | 0.664 | 0.728 | 0.886 | 0.713 | 0.756 |
| Qwen3.5-122B-A10B-FP8 | 0.794 | 0.637 | 0.713 | 0.745 | 0.715 | 0.722 |
| Qwen3.5-397B-A17B-FP8 | 0.726 | 0.674 | 0.745 | 0.770 | 0.715 | 0.729 |
| Qwen3.5-397B-A17B-FP8-Reasoning | 0.757 | 0.695 | 0.777 | 0.815 | 0.743 | 0.761 |
| **Our model** | **0.842** | **0.700** | **0.764** | **0.956** | **0.767** | **0.814** |

The model leads all deployable (non-reasoning) baselines on every benchmark group, achieving 0.767 average on non-fine-tuned benchmarks—2.4 points above the strongest reasoning-mode baseline (Qwen3.5-397B-A17B-FP8-Reasoning) despite being 10× smaller.

### SFT Effect Across Base Scales

| Model | SP | MWS | MP | FT | Avg w/o FT |
|-------|-----|------|-----|-----|------------|
| 9B-Base | 0.780 | 0.588 | 0.626 | 0.640 | 0.667 |
| 9B-Base + SFT | 0.808 | 0.620 | 0.718 | 0.955 | 0.714 |
| 35B-A3B-Base | 0.827 | 0.677 | 0.697 | 0.722 | 0.732 |
| 35B-A3B-Base + SFT | 0.842 | 0.700 | 0.764 | 0.956 | 0.767 |

SFT gains are disproportionately large relative to corpus size and generalize beyond SFT coverage, with pre-training capacity setting the upper bound.

### Data Mixing Ablations

| Mixture | SP | MWS | MP | FT | Avg w/o FT |
|---------|-----|------|-----|-----|------------|
| Base (no SFT) | 0.827 | 0.672 | 0.697 | 0.722 | 0.729 |
| Internal only | 0.840 | 0.685 | 0.738 | 0.951 | 0.754 |
| Open only | 0.835 | 0.700 | 0.677 | 0.793 | 0.737 |
| 1:1 mix | 0.837 | 0.689 | 0.768 | 0.955 | 0.764 |
| **1:4 (open-heavy)** | **0.842** | **0.700** | **0.764** | **0.956** | **0.767** |

Internal data saturates quickly; open-domain data scales further, with open-heavy ratio yielding the best trade-off.

### Cross-lingual Transfer

| Benchmark | Base | Ours | Qwen3.5 397B |
|-----------|------|-------|--------------|
| OCRBench | 848 | 863 | 889 |
| OCRBench v2 (en) | 55.1 | 56.2 | 61.1 |
| CC-OCR | 81.74 | 80.40 | 84.86 |
| DocVQA (test) | 82.14 | **90.10** | 96.74 |
| InfoVQA (test) | 51.10 | **66.30** | 89.58 |

DADC transfers to English despite predominantly Russian training, with largest gains on DocVQA (+8 points) and InfoVQA (+15.2 points).

### Cost Analysis Results

| Model | $G_m$ | Throughput | $\rho_{w_1}(m)$ | $\rho_{w_2}(m)$ |
|-------|-------|------------|-----------------|-----------------|
| Qwen3.5-397B-A17B-FP8 | 8 | 0.025× | -0.115 | -1.582 |
| Qwen2.5-VL-72B-Instruct-AWQ | 1 | 0.200× | 0.563 | 0.534 |
| Qwen3.5-35B-A3b-Base | 1 | 1.000× | 0.691 | 0.791 |
| **Our model** | **1** | **1.000×** | **0.819** | **0.861** |

Large models (Qwen3.5-397B) have negative cost reduction ratios—serving them costs more than the human annotation they replace. The proposed model sustains 5× higher throughput than Qwen2.5-VL-72B and achieves the highest cost reduction in both workflow groups.

## Theoretical and Practical Implications

### Theoretical Contributions

1. **Difficulty-Aware Data Curation (DADC)**: Demonstrates that naive synthetic data generation from open-domain PDFs degrades performance; aggressive difficulty-based filtering is essential for extracting high-signal supervision from commodity data sources.

2. **Quality-adjusted cost framework**: Formalizes the trade-off between model quality and serving economics with asymmetric confirmation ($0.3x$) vs. correction ($1.1x$) costs, deriving the analytical assistance threshold $\theta_{\mathrm{assist}} = 0.125$ from the break-even condition.

3. **SFT as capacity unlocking**: Shows that SFT unlocks latent capacity from pre-training rather than merely refining existing behavior, with gains generalizing beyond training coverage.

### Practical Implications

- **Single-model deployment**: Replaces cascaded OCR–LLM pipelines with a single prompt-driven model on one H100, reducing infrastructure complexity
- **Economic viability**: The model reduces expected costs by over 80% vs. human baseline and 53% vs. best competing open-source model
- **Privacy compliance**: On-premise fine-tuning and deployment satisfies regulated PII constraints
- **Workflow generalization**: The methodology is workflow-independent; adaptation requires only recalibration of coefficients (workload distribution, GPU utilization, routing policy, per-action costs)

## Conclusion

The paper demonstrates that deploying a VLM at scale in regulated settings is gated by per-document economics, not benchmark scores. The 35B-A3B MoE VLM, domain-adapted via DADC on open-domain PDFs mixed with in-house production data, outperforms baselines an order of magnitude larger while remaining economically viable.

**Key limitations and future directions**:
- **Reasoning/RL**: The corpus lacks multi-step reasoning examples; RL with verifiable rewards is a natural next step once SFT data scaling returns diminish
- **Cross-lingual coverage**: No per-language ablation performed for Cyrillic languages
- **Post-training ceiling**: SFT has not reached the model's performance ceiling; plans to expand data diversity, reasoning coverage, and supervision quality

The paradigm offers a broadly useful template for grounded model selection in deployment economics, making the cost-quality trade-off explicit and actionable.

---

_Markdown view of https://picx.dev/p/aiM0C1, served by PicX — AI-generated visual whiteboard summaries of research papers._