# Data Quality in RVL-CDIP: A Systematic Audit

> RVL-CDIP is compromised by label errors, test-train leakage, and spurious Bates codes, making published accuracy gains smaller than OCR-engine choice effects.

- **Source:** [arXiv](https://arxiv.org/abs/2610.32126)
- **Published:** 2026-11-07
- **Permalink:** https://picx.dev/p/aCeleX
- **Whiteboard:** https://picx.dev/p/aCeleX/image

## Summary

## Summary (Overview)

- **Systematic audit of RVL-CDIP**: The paper presents a comprehensive audit of the RVL-CDIP document classification benchmark, identifying multiple independent defects: approximately 10.5% erroneous labels, 35.2% of the test set having duplicates or near-duplicates in training, and identification codes on pages serving as spurious cues sufficient for 69% classification accuracy.
- **OCR engine as uncontrolled confound**: The choice of OCR engine changes model accuracy by 4.1–8.3 points, which is 5–40 times larger than the 0.19–0.82 point differences between successive LayoutLM generations, making cross-paper comparisons unsound.
- **Sensitive data exposure and remediation**: Nearly 16,000 RVL-CDIP documents (4.0%) contain sensitive personal data including 2,342 US Social Security numbers; the authors developed a synthetic-replacement de-identification pipeline that leaves classifier predictions unchanged.
- **Joint remediation re-benchmarking**: With all defects remediated, the 13 retrained models are 0.9–2.9 points more accurate than in the original condition; the most accurate model, LayoutLMv3, achieves 96.5% accuracy. The effects of defects are not additive.
- **Public release of corrected resources**: The authors release de-identified corpus variants, corrected labels, duplicate annotations, OCR text layers, and out-of-distribution evaluation sets (RVL-CDIP-N and RVL-CDIP-O).

## Introduction and Theoretical Foundation

The paper addresses a fundamental assumption in benchmark-based research: that reported accuracy on a benchmark measures genuine task capability rather than exploitation of dataset defects. RVL-CDIP (Harley et al., 2015) has become the *de facto* standard for document classification, with reported accuracy now reaching 98.8% (LayoutLLM, Fujitake, 2024). However, the authors identify that:

- RVL-CDIP is sampled from IIT-CDIP (Lewis et al., 2006), a collection of ~7 million documents released through tobacco industry litigation, assembled for legal discovery and never curated for machine learning.
- 23 of 34 surveyed papers evaluate classification on RVL-CDIP alone; 29 evaluate only on datasets derived from the same litigation archive.
- The error rate of leading models (1.2%) is now approximately one-ninth of the measured label error rate in the benchmark itself.

The theoretical foundation draws on prior work showing that benchmarks frequently contain label noise (Northcutt et al., 2021a), test–train overlap (Allamanis, 2019; Elangovan et al., 2021), and collection artifacts enabling shortcut learning (Gururangan et al., 2018; McCoy et al., 2019).

The audit is organized around five axes:

1. **Label validity**: The correctness of the 400,000 category labels
2. **Split integrity**: Whether test documents recur in training
3. **Spurious cues**: Identification (Bates) codes stamped on pages
4. **Distributional narrowness**: The limited scope of the corpus (pre-2006, scanned, tobacco industry)
5. **OCR engine choice**: Uncontrolled variation in text extraction across papers

## Methodology

### Sensitive Data Annotation and De-identification

- **Manual annotation** of 400,000 documents by a team of seven volunteer undergraduate researchers over ~2 years, identifying 11 sensitive entity types (Social Security numbers, dates of birth, home addresses, etc.)
- **Synthetic replacement pipeline**: Each sensitive entity is erased and replaced with a generated, plausible value matched to the page's typography (font, size, stroke weight, baseline, skew) through an 8-step process including chamfer distance font matching and Augraphy-based noise degradation
- **Validation**: De-identification changed none of the predictions of a DiT classifier on 445 resumes; synthetic replacement is more similar to originals than black-box redaction for 61.6% of documents

### Label Error Detection

- **Exhaustive human review** of all 400,000 documents against published label criteria (absent from the original release)
- Three error types distinguished: *unknown* (no valid label exists), *wrong* (different label is correct), *mixed* (multiple valid labels)
- Inter-rater agreement: Fleiss's kappa of 0.74 (substantial agreement)

### Duplicate Detection (Filter-and-Refine)

- **Filter stage**: CLIP embeddings (512-dimensional, ℓ-normalized) with exhaustive cosine similarity computation $S = X_{train}X_{test}^\top$, retaining top-10 candidates per test document
- **Refine stage**: Two complementary methods:
  - SuperPoint keypoint detection + LightGlue matching for image comparison
  - Ratcliff/Obershelp sequence similarity on OCR text
- **Calibration**: Precision of 100%, recall of 82.9%, F₁ of 90.6%; all reported rates are lower bounds

### Identification Code Analysis

- **Annotation**: 11,200 documents sampled from RVL-CDIP; codes present on 92.0% of documents
- **Predictive signal**: XGBoost classifier on code features (position, dimensions, text format, nearest-number categories) achieves 69.2% accuracy (chance: 6.25%)
- **Detection and removal**: YOLOv8 object detector (precision 97.5%, recall 96.8%, F₁ 97.1%) applied to all 400,000 documents

### Out-of-Distribution Evaluation

- **RVL-CDIP-N**: 1,002 documents belonging to the 16 categories but from different sources (DocumentCloud, web search), largely born-digital and post-2006
- **RVL-CDIP-O**: 2,915 documents belonging to none of the 16 categories
- **Abstention metrics**: Maximum softmax probability and energy score, reported as AUC

### OCR Engine Comparison

- Two text layers produced: Tesseract 4.1.1 and AWS Textract over all 400,000 pages
- Identical models (BERT, RoBERTa, LayoutLM, TF-IDF SVM) trained on each layer

### Re-benchmarking Protocol

- 25 models in 7 groups: image models (AlexNet, GoogLeNet, SqueezeNet, VGG-16, ResNet-50, ResNeXt-50, DiT-base), text models (BERT, RoBERTa), multi-modal (LayoutLM, LayoutLMv3), linear baselines (TF-IDF SVM, TF-IDF+SigLIP SVM), public checkpoints (DocXClassifier, Donut), zero-shot image models (CLIP, ALIGN, SigLIP, SigLIP 2, Jev), and zero-shot text models (Jev, Llama 4 Maverick, gpt-oss-20b, Qwen 3 32B, Mistral 7B)
- 11 evaluation conditions varying labels (original/corrected), duplicates (present/removed), and codes (present/removed)

## Empirical Validation / Results

### Label Errors

| Category | Any Error | Category | Any Error |
|----------|-----------|----------|-----------|
| letter | 31.26% | memo | 5.96% |
| budget | 17.35% | presentation | 5.47% |
| invoice | 14.94% | scientific_publication | 4.54% |
| questionnaire | 14.44% | file_folder | 4.01% |
| form | 14.25% | specification | 3.42% |
| handwritten | 12.66% | resume | 1.48% |
| scientific_report | 12.16% | **RVL-CDIP** | **10.51%** |
| email | 9.51% | | |

### Test–Train Duplication

- **35.2%** of the test set has a duplicate or near-duplicate in training
- Highest rates: specification (82.20%), invoice (68.75%), questionnaire (61.56%)
- Lowest rates: file_folder (1.98%), handwritten (4.03%), email (6.96%)
- Validation split rate (35.38%) nearly matches test rate, indicating duplication was introduced during corpus assembly, not partitioning

### Identification Codes

- Codes present on 92.0% of documents
- Code-features classifier: **69.2%** accuracy (chance: 6.25%)
- File-name-only classifier: 68.8% accuracy
- Removing codes from test pages lowers image model accuracy by 3.1–4.8 points; Donut loses 11.0 points (95.2 → 84.2%)

### OCR Engine Effects

| Model | Tesseract | Textract | Δ |
|-------|-----------|----------|---|
| TF-IDF SVM | 81.96 | 90.23 | +8.27 |
| BERT-base | 86.66 | 92.54 | +5.88 |
| RoBERTa-base | 86.50 | 92.47 | +5.97 |
| LayoutLM-base | 89.39 | 93.50 | +4.11 |

### Out-of-Distribution Performance

- Image-only models trained on RVL-CDIP lose **16–28 accuracy points** on RVL-CDIP-N
- Text models lose only 3–4 points
- Image models' abstention ability (N-O AUC) drops to 0.594–0.673, near chance (0.5)
- With corrected labels, LayoutLMv3 reaches 97.2% on RVL-CDIP-N (from 83.7%)

### Joint Remediation Results

- Fully remediated accuracy: retrained models are 0.9–2.9 points more accurate than original condition
- LayoutLMv3: 96.5% (best retrained model)
- TF-IDF + SigLIP SVM: 96.0% (outperforms LayoutLM at 95.2%)
- Defect effects are **not additive**: fully remediated accuracy is 1.2–1.8 points higher than the sum of single-remediation effects predicts

### Abstention (N-O Setting, MSP AUC)

| Model | Original | Corrected Labels |
|-------|----------|-----------------|
| AlexNet | 0.645 | 0.731 |
| VGG-16 | 0.671 | 0.763 |
| BERT | 0.841 | 0.872 |
| LayoutLMv3 | 0.832 | 0.924 |
| TF-IDF+SigLIP SVM | 0.883 | 0.947 |

## Theoretical and Practical Implications

### Implications for Benchmark Interpretation

1. **Published accuracies are composite quantities**: An RVL-CDIP score partly measures document classification ability and partly measures a model's ability to learn the benchmark's defects. The 98.8% reported accuracy reflects reproduction of wrong labels (10.5% error rate) rather than task mastery.

2. **Leaderboard differences are not meaningful**: Differences of 0.19–0.82 points between model generations are dwarfed by the 4.1–8.3 point effect of OCR engine choice and the 2.3–4.1 point effect of label errors.

3. **Catastrophic shortcut learning**: The Bates codes alone enable 69% accuracy, and a state-of-the-art model loses 11 points without them. Models trained on RVL-CDIP are essentially learning how documents were numbered during litigation.

4. **Privacy concerns**: Models trained on this corpus may memorize and leak sensitive personal data (Carlini et al., 2021). The de-identified corpus should be the default distribution.

### Guidance for Researchers

- **Use remediated variants**: Corrected labels, code-free pages, de-duplicated test split
- **Report out-of-distribution results**: RVL-CDIP-N and RVL-CDIP-O alongside in-distribution accuracy
- **Use published OCR text layers**: To enable cross-paper comparability
- **Release fine-tuned classifiers**: Only 5 of 34 papers release RVL-CDIP classifiers; 18 release no code

### Requirements for Successor Benchmarks

1. Published annotation guidelines
2. Measured and reported label error rate
3. Multi-label annotation for mixed documents
4. Verified split integrity at construction time
5. No sensitive personal data, verified before release
6. Documented provenance artifacts (stamps, watermarks)
7. Canonical text layer
8. Paired out-of-distribution evaluation data from the outset
9. Contemporary and born-digital documents
10. Breadth and multilinguality (more than 16 categories, more than one language)

## Conclusion

The audit demonstrates that RVL-CDIP is compromised along multiple independent dimensions simultaneously: label errors (10.5%), test–train duplication (35.2%), spurious identification codes (69% accuracy from codes alone), distributional narrowness (16–28 point drops out-of-distribution for image models), and uncontrolled OCR engine effects (4.1–8.3 points). The defects are not additive—their joint remediation yields accuracies 1.2–1.8 points higher than the sum of individual effects predicts.

The authors recommend that researchers continue to use RVL-CDIP (rather than abandoning it, since no comparable alternative exists) but use the remediated variants they release, report out-of-distribution results, and use published OCR text layers. The field should simultaneously work toward a successor benchmark meeting the ten requirements specified.

**Key released artifacts** (Appendix A):
- De-identified corpus (default distribution) with and without identification codes
- Corrected labels for all 400,000 documents
- Duplicate pair annotations
- Textract and Tesseract OCR text layers with bounding boxes
- RVL-CDIP-N (1,002 documents) and RVL-CDIP-O (2,915 documents) evaluation sets
- De-identification pipeline tool with fonts and word lists
- YOLOv8 code detector weights

The broader impact statement emphasizes that removing thousands of Social Security numbers from actively used datasets outweighs the residual risk of documenting the removal, and that the audit serves to measure the gap between benchmark performance and real-world deployment readiness for document classifiers used in records processing, claims handling, and compliance review.

---

_Markdown view of https://picx.dev/p/aCeleX, served by PicX — AI-generated visual whiteboard summaries of research papers._