# Counting Documents Is Not Counting Text: Unit Bias in Web-PDF Corpus Statistics

> Counting documents instead of tokens misrepresents web-PDF corpora by up to 11x, with 63% of text lost to truncation that recovers only 1-11% via standard parsers.

- **Source:** [arXiv](https://arxiv.org/abs/2608.16390)
- **Published:** 2026-10-17
- **Permalink:** https://picx.dev/p/gRMyYm
- **Whiteboard:** https://picx.dev/p/gRMyYm/image

## Summary

# Summary of "Counting Documents Is Not Counting Text: Unit Bias in Web-PDF Corpus Statistics"

## Summary (Overview)

- **Core finding**: Web-PDF corpus statistics computed per-document versus per-token describe fundamentally different corpora, with text mass being extremely concentrated in a small fraction of long documents (Gini coefficient 0.807).
- **Key quantitative result**: Common Crawl's 1 MiB payload cap truncated only 23.06% of documents but 63.08% of the corpus's text tokens—a 2.74× over-representation of text in truncated files.
- **Recovery failure**: Two widely used text-layer parsers (PyMuPDF and PDFium) recover only 11.4% and 1.4% respectively of exposed text from truncated PDFs, leaving 55–62% of corpus text unrecoverable from the crawl.
- **Policy implication**: The March 2025 cap increase to 5 MiB halves exposure (to 30.19% of tokens) but what remains exposed is still almost entirely unrecoverable; the paper recommends reporting corpus statistics in both document and token units.
- **Methodological contribution**: First token-weighted characterization of a web-PDF corpus, using the unique CC-MAIN-2021-31-PDF-UNTRUNCATED dataset that pairs truncated crawl fragments with re-fetched originals.

## Introduction and Theoretical Foundation

The paper addresses a fundamental unit mismatch in how PDF corpora are described. While these corpora headline their size in tokens (e.g., FinePDFs claims "about 3 trillion tokens across 475 million documents"), every rate they publish—coverage, OCR routing, re-fetch recovery, language mix—is computed per document. None decomposes the token total.

The theoretical foundation rests on the observation that **documents are not interchangeable units of text**. PDF length is extremely skewed, far more so than HTML. The paper cites FinePDFs' own observation of a "~5.3k character median" with a "95th percentile ~68k," yet even that corpus reports all rates per document.

The paper's central question: *How do a web-PDF corpus's headline statistics change when each document is weighted by the text it contributes rather than counted once?*

The truncation problem is identified as particularly severe: Common Crawl's payload cap stores only a prefix of each PDF, and while the cap's effect on documents was known (25.7% of PDFs truncated in CC-MAIN-2025-13, falling to 6.8% after the 5 MiB change), nobody had quantified how much *text* it removes.

## Methodology

### Corpus
- **CC-MAIN-2021-31-PDF-UNTRUNCATED**: Contains every PDF found in Common Crawl CC-MAIN-2021-31, with truncated payloads refetched whole from origin
- Deduplicated to **7,932,654 unique documents** (from 8.3M URLs)
- Text mass measured via `tika_eval_num_tokens` (Apache Tika 2.8.0, whitespace/ICU count), totaling **32,570,135,761 tokens**

### Measures
For any subset $S$ of the $N$ documents, the paper reports:
- **Document share**: $|S|/N$
- **Token share**: $\sum_{d \in S} t_d / \sum_d t_d$

The **Gini coefficient** is computed over the $n = 7{,}292{,}093$ documents with at least one token:

$$
G = \frac{2 \sum_{i=1}^{n} i t_{(i)}}{n \sum_{i=1}^{n} t_{(i)}} - \frac{n+1}{n} \tag{1}
$$

### Key definitions
- **Exposure $E$**: Token share of documents Common Crawl truncated
- **Recovery $r_{e,d}$**: Tokens extractor $e$ extracts from truncated document divided by tokens from whole document
- **Loss**: $E(1 - r_e)$ — share of corpus text a pipeline cannot obtain from the crawl

### Truncation reconstruction
The paper re-creates what Common Crawl held by cutting each of the 1,829,061 truncated documents at 1 MiB, then extracts text from both fragment and whole file using two independent engines: **PyMuPDF (MuPDF)** and **PDFium (Chromium's)**.

## Empirical Validation / Results

### Text mass concentration (Section 4.1)

**Table 1: The same corpus counted two ways**

| Bin | doc % | tok % | ratio |
|-----|-------|-------|-------|
| >50 pp | 4.42 | 49.71 | 11.24× |
| 4–30 pp | 31.64 | 26.87 | 0.85× |
| 31–50 pp | 3.21 | 9.08 | 2.83× |
| landscape, any length | 14.08 | 7.76 | 0.55× |
| 2–3 pp | 21.20 | 3.76 | 0.18× |
| 1 pp | 24.96 | 2.70 | 0.11× |
| >50 pp, any orientation | 5.00 | 53.53 | 10.71× |

Key concentration results:
- **3.02% of text-bearing documents hold half the tokens**
- **15.54% hold 80% of tokens**
- Gini coefficient of tokens = **0.807**

### Validation against independent extractor (Section 4.2)
Page counts (extracted with Poppler, an independent tool) reproduce the same concentration:
- Gini of pages = 0.767 (vs. 0.807 for tokens)
- 3.98% of documents hold half the pages (vs. 3.02% for tokens)
- Documents over 50 pages: 54.27% of pages vs. 53.53% of tokens

### Toolchain shares (Section 4.3)
- **TeX**: 1.66% of documents but 4.05% of tokens (2.43×)
- **PowerPoint**: 2.24% of documents but 0.82% of tokens (0.37×)
- **Microsoft Word**: 24.95% of documents vs. 19.37% of tokens

### Truncation exposure (Section 4.4)

**Table 2: Common Crawl's truncation cap, expressed in both units**

| | docs % | tokens % |
|---|---|---|
| Truncated by CC (1 MiB) | 23.06 | 63.08 |
| still truncated at 5 MiB | 5.94 | 30.19 |

- Truncated documents carry **2.74× the mean token mass** (median true size 2.62 MB vs. 190 KB)
- In bytes: 1 MiB cap kept 1.738 TiB of 9.449 TiB total (discarding 81.6%); 5 MiB cap would keep 5.106 TiB (46.0% discarded)
- Byte loss overstates text loss: 81.6% of bytes vs. 63.08% of tokens at 1 MiB

### Recovery rates (Section 4.5)

On the 1,225,130 truncated documents both tools processed:
- **PyMuPDF**: recovers 11.4% of tokens, 20.7% of pages
- **PDFium**: recovers 1.4% of tokens, 1.9% of pages (8.4× gap)
- Both tools agree within 1.7% on intact versions—divergence is specific to damaged input

Failure modes differ dramatically:
- PyMuPDF opens 98.9% of truncated files (rebuilds missing cross-reference table) but 72.4% yield nothing
- PDFium refuses 96.7% outright (fails to open)

**Combined loss**: $E(1-r)$ = 63.08% × (1 − 0.114) ≈ **56%** (PyMuPDF) to 63.08% × (1 − 0.014) ≈ **62%** (PDFium), reported as **55–62% range**

### 5 MiB cap analysis (Section 4.6)
On documents truncated at both caps:
- PyMuPDF recovery rises from 3.3% to 13.2% of tokens
- PDFium moves from 0.1% to 2.4%
- 74% of rescued documents lie between 5 and 10 MiB
- 77.6% of these documents still yield no text at 5 MiB (97.0% under PDFium)

## Theoretical and Practical Implications

1. **Unit bias in corpus statistics**: Every rate in the PDF corpus literature (coverage, filter drop, OCR routing) is per-document or unreported. The paper demonstrates these can misrepresent the corpus by up to 11× per category.

2. **Filter design**: Any filter that selects on size (PDFA's 100 MB/500 ms cuts, FinePDFs' OCR router, the crawler's cap) removes far more text than documents, because "size is where the text is." Effects should be reported in tokens.

3. **Truncation policy**: The 1 MiB → 5 MiB change halves exposure but leaves most of what remains unrecoverable. The paper recommends a **size-aware fetch policy for large PDFs** rather than a higher cap.

4. **Parser behavior**: Recovery rates are not defined until the extractor is named—the 8.4× gap between engines on identical input shows that "recovery" is a property of the parser, not the file.

5. **OCR is not an alternative**: A page can only be rasterized if its objects lie inside the fragment, in which case its text layer is extractable anyway. Refetching from origin is the only true remedy.

## Conclusion

The paper demonstrates that CC-MAIN-2021-31-PDF-UNTRUNCATED is "two different corpora" depending on whether one counts documents or text. The central recommendation is that **corpus statistics should be reported in both units**—documents and tokens—since the token is the unit models consume, data mixes are set, and compute is scaled.

Future directions include:
- Re-measuring exposure on post-2025 crawls (needs only WARC metadata)
- Decomposing token totals by length and quality in other corpora
- Implementing size-aware fetch policies for large PDFs in Common Crawl

**Limitations acknowledged**: Recovery figures cover 67% of truncated documents (technical extractor errors on some shards); producer strings are self-reported with 8.08% unclassified; the 5 MiB figure is a counterfactual on 2021 file sizes, not a measurement of current crawls; no OCR arm was run.

---

_Markdown view of https://picx.dev/p/gRMyYm, served by PicX — AI-generated visual whiteboard summaries of research papers._