Summary of "Counting Documents Is Not Counting Text: Unit Bias in Web-PDF Corpus Statistics"

Summary (Overview)

• Core finding: Web-PDF corpus statistics computed per-document versus per-token describe fundamentally different corpora, with text mass being extremely concentrated in a small fraction of long documents (Gini coefficient 0.807).
• Key quantitative result: Common Crawl's 1 MiB payload cap truncated only 23.06% of documents but 63.08% of the corpus's text tokens—a 2.74× over-representation of text in truncated files.
• Recovery failure: Two widely used text-layer parsers (PyMuPDF and PDFium) recover only 11.4% and 1.4% respectively of exposed text from truncated PDFs, leaving 55–62% of corpus text unrecoverable from the crawl.
• Policy implication: The March 2025 cap increase to 5 MiB halves exposure (to 30.19% of tokens) but what remains exposed is still almost entirely unrecoverable; the paper recommends reporting corpus statistics in both document and token units.
• Methodological contribution: First token-weighted characterization of a web-PDF corpus, using the unique CC-MAIN-2021-31-PDF-UNTRUNCATED dataset that pairs truncated crawl fragments with re-fetched originals.

Introduction and Theoretical Foundation

The paper addresses a fundamental unit mismatch in how PDF corpora are described. While these corpora headline their size in tokens (e.g., FinePDFs claims "about 3 trillion tokens across 475 million documents"), every rate they publish—coverage, OCR routing, re-fetch recovery, language mix—is computed per document. None decomposes the token total.

The theoretical foundation rests on the observation that documents are not interchangeable units of text. PDF length is extremely skewed, far more so than HTML. The paper cites FinePDFs' own observation of a "~5.3k character median" with a "95th percentile ~68k," yet even that corpus reports all rates per document.

The paper's central question: How do a web-PDF corpus's headline statistics change when each document is weighted by the text it contributes rather than counted once?

The truncation problem is identified as particularly severe: Common Crawl's payload cap stores only a prefix of each PDF, and while the cap's effect on documents was known (25.7% of PDFs truncated in CC-MAIN-2025-13, falling to 6.8% after the 5 MiB change), nobody had quantified how much text it removes.

Methodology

Corpus

• CC-MAIN-2021-31-PDF-UNTRUNCATED: Contains every PDF found in Common Crawl CC-MAIN-2021-31, with truncated payloads refetched whole from origin
• Deduplicated to 7,932,654 unique documents (from 8.3M URLs)
• Text mass measured via tika_eval_num_tokens (Apache Tika 2.8.0, whitespace/ICU count), totaling 32,570,135,761 tokens

Measures

For any subset $S$ of the $N$ documents, the paper reports:

• Document share: $|S|/N$
• Token share: $\sum_{d \in S} t_d / \sum_d t_d$

The Gini coefficient is computed over the $n = 7{,}292{,}093$ documents with at least one token:

$$G = \frac{2 \sum_{i=1}^{n} i t_{(i)}}{n \sum_{i=1}^{n} t_{(i)}} - \frac{n+1}{n} \tag{1}$$

Key definitions

• Exposure $E$: Token share of documents Common Crawl truncated
• Recovery $r_{e,d}$: Tokens extractor $e$ extracts from truncated document divided by tokens from whole document
• Loss: $E(1 - r_e)$ — share of corpus text a pipeline cannot obtain from the crawl

Truncation reconstruction

The paper re-creates what Common Crawl held by cutting each of the 1,829,061 truncated documents at 1 MiB, then extracts text from both fragment and whole file using two independent engines: PyMuPDF (MuPDF) and PDFium (Chromium's).

Empirical Validation / Results

Text mass concentration (Section 4.1)

Table 1: The same corpus counted two ways

Key concentration results:

• 3.02% of text-bearing documents hold half the tokens
• 15.54% hold 80% of tokens
• Gini coefficient of tokens = 0.807

Validation against independent extractor (Section 4.2)

Page counts (extracted with Poppler, an independent tool) reproduce the same concentration:

• Gini of pages = 0.767 (vs. 0.807 for tokens)
• 3.98% of documents hold half the pages (vs. 3.02% for tokens)
• Documents over 50 pages: 54.27% of pages vs. 53.53% of tokens

Toolchain shares (Section 4.3)

• TeX: 1.66% of documents but 4.05% of tokens (2.43×)
• PowerPoint: 2.24% of documents but 0.82% of tokens (0.37×)
• Microsoft Word: 24.95% of documents vs. 19.37% of tokens

Truncation exposure (Section 4.4)

Table 2: Common Crawl's truncation cap, expressed in both units

• Truncated documents carry 2.74× the mean token mass (median true size 2.62 MB vs. 190 KB)
• In bytes: 1 MiB cap kept 1.738 TiB of 9.449 TiB total (discarding 81.6%); 5 MiB cap would keep 5.106 TiB (46.0% discarded)
• Byte loss overstates text loss: 81.6% of bytes vs. 63.08% of tokens at 1 MiB

Recovery rates (Section 4.5)

On the 1,225,130 truncated documents both tools processed:

• PyMuPDF: recovers 11.4% of tokens, 20.7% of pages
• PDFium: recovers 1.4% of tokens, 1.9% of pages (8.4× gap)
• Both tools agree within 1.7% on intact versions—divergence is specific to damaged input

Failure modes differ dramatically:

• PyMuPDF opens 98.9% of truncated files (rebuilds missing cross-reference table) but 72.4% yield nothing
• PDFium refuses 96.7% outright (fails to open)

Combined loss: $E(1-r)$ = 63.08% × (1 − 0.114) ≈ 56% (PyMuPDF) to 63.08% × (1 − 0.014) ≈ 62% (PDFium), reported as 55–62% range

5 MiB cap analysis (Section 4.6)

On documents truncated at both caps:

• PyMuPDF recovery rises from 3.3% to 13.2% of tokens
• PDFium moves from 0.1% to 2.4%
• 74% of rescued documents lie between 5 and 10 MiB
• 77.6% of these documents still yield no text at 5 MiB (97.0% under PDFium)

Theoretical and Practical Implications

1. Unit bias in corpus statistics: Every rate in the PDF corpus literature (coverage, filter drop, OCR routing) is per-document or unreported. The paper demonstrates these can misrepresent the corpus by up to 11× per category.

2. Filter design: Any filter that selects on size (PDFA's 100 MB/500 ms cuts, FinePDFs' OCR router, the crawler's cap) removes far more text than documents, because "size is where the text is." Effects should be reported in tokens.

3. Truncation policy: The 1 MiB → 5 MiB change halves exposure but leaves most of what remains unrecoverable. The paper recommends a size-aware fetch policy for large PDFs rather than a higher cap.

4. Parser behavior: Recovery rates are not defined until the extractor is named—the 8.4× gap between engines on identical input shows that "recovery" is a property of the parser, not the file.

5. OCR is not an alternative: A page can only be rasterized if its objects lie inside the fragment, in which case its text layer is extractable anyway. Refetching from origin is the only true remedy.

Conclusion

The paper demonstrates that CC-MAIN-2021-31-PDF-UNTRUNCATED is "two different corpora" depending on whether one counts documents or text. The central recommendation is that corpus statistics should be reported in both units—documents and tokens—since the token is the unit models consume, data mixes are set, and compute is scaled.

Future directions include:

• Re-measuring exposure on post-2025 crawls (needs only WARC metadata)
• Decomposing token totals by length and quality in other corpora
• Implementing size-aware fetch policies for large PDFs in Common Crawl

Limitations acknowledged: Recovery figures cover 67% of truncated documents (technical extractor errors on some shards); producer strings are self-reported with 8.08% unclassified; the 5 MiB figure is a counterfactual on 2021 file sizes, not a measurement of current crawls; no OCR arm was run.