# The Stanford EDGAR Filings Dataset: Reconstructing U.S. Corporate and Financial Disclosures into Layout-Faithful and Token-Efficient Pretraining Data

> SEFD reconstructs the SEC EDGAR archive into layout-faithful MultiMarkdown, achieving over 99 percent structural accuracy and enabling token-efficient financial pretraining with less than 0.1 percent Common Crawl overlap.

- **Source:** [arXiv](https://arxiv.org/abs/2606.18192)
- **Published:** 2026-08-18
- **Permalink:** https://picx.dev/p/6pdG96
- **Whiteboard:** https://picx.dev/p/6pdG96/image

## Summary

# The Stanford EDGAR Filings Dataset (SEFD): A Comprehensive Summary

## Summary (Overview)

- **SEFD** is an open reconstruction of the complete SEC EDGAR archive (1994–present) into layout-faithful MultiMarkdown (MMD), designed for financial language modeling and evaluation. The full corpus is estimated at **550B tokens** across 18.5M filings; **SEFD-v1** releases a 152B-token snapshot covering January 2022–June 2025.

- The dataset employs a **"visual-first" parsing methodology** that reconstructs spatial semantics from rendered documents rather than relying on naive DOM-based text extraction, preserving tables, indentation, and visual hierarchy that carry financial meaning.

- SEFD achieves **>99% structural and semantic accuracy** (per human evaluation) with **<0.1% overlap** with Common Crawl-derived corpora, making it a scarce source of clean, long-context financial data.

- Two novel benchmarks are introduced: **EDGAR-Forecast** (filing-grounded numerical forecasting after knowledge cutoffs; best model GPT-5.5 achieves 51.8%) and **EDGAR-OCR** (transcription of complex financial tables; Qwen3.6-35B-A3B achieves 75.78%).

- The corpus is **token-efficient**—clean reconstruction yields >99% token reduction relative to raw source, as much filing HTML consists of presentation scaffolding rather than filer-authored content.

---

## Introduction and Theoretical Foundation

### The "Quality over Quantity" Paradigm

The paper situates SEFD within the broader shift in LLM pretraining from brute-force scaling toward **data-centric curation**. Key motivating evidence:

- **Diminishing returns** from scaling alone: GPT-4.5 and Llama 4 Behemoth show reduced gains even at 30T+ training tokens.
- **Microsoft's Phi line** demonstrates that "textbook-quality" curation can outperform larger models on common benchmarks.
- As architectures and training recipes converge, **data itself becomes the primary differentiator** of performance.

### Why EDGAR?

EDGAR is one of the largest public repositories of factual, business-oriented long-context data, processing ~4,700 filings daily and ~40,000 new filers annually. However, it has remained largely untapped for pretraining because its 18.5M filings present broad parsing challenges:

- **Layout engineering**: Filings use table-based layout (`<table>`, `<tr>`, `<td>`), presentational markup, and explicit whitespace (`&nbsp;`, `<br>`) constrained by EDGAR's HTML 3.2/4.0-era restrictions.
- **Fragmentary structure**: Visual elements are exploded across multiple cells/rows to maintain print-like appearance in browsers, disconnecting semantically related content.
- **Format heterogeneity**: Plaintext, HTML, XML, SGML, and PDF filings coexist across different eras.

### Key Insight

> "Flattening EDGAR tables can create ambiguity by detaching values from labels, duplicating headers, or flipping accounting signs."

Indentation disambiguates statement hierarchies, merged headers connect periods and segments, and numeric-cell reconstruction reattaches signs, currency symbols, and percentage symbols to their values.

---

## Methodology

### 1. Format-Specific Parsing Pipeline

SEFD routes filings across **33 specialized XML schemas** and **over 350 filing types** based on the SGML header's `CONFORMED SUBMISSION TYPE`, then applies format-specific reconstruction:

#### Plaintext (Legacy ASCII)
- **Preservation strategy**: Fixed-width whitespace is semantically meaningful, so no normalization is applied beyond collapsing 3+ consecutive line breaks into 2.
- Identified by regex-based check for absence of structural HTML tags.

#### HTML (~62% of corpus tokens)
The most complex case, addressing three key reconstruction challenges:

**a) The "Three-Column Hack"** (Figure 1): Numerical values are exploded across three cells for decimal alignment:

| Index | 0 | 1 | 2 |
|-------|---|---|---|
| 0 | $ | 1000.00 | NaN |
| 1 | ( | 500.50 | ) |
| 2 | NaN | 250.00 | NaN |

- **Prefix/Gutter Column** (width="1%"): holds currency symbols or opening parentheses
- **Value Column**: integer and decimal
- **Suffix Column**: closing parentheses or percentage signs

**Symbol Reconstruction** uses a bidirectional rule-based merging algorithm that scans for "modifier columns" (exclusively currency symbols, percentages, or parenthetical fragments) and merges them into adjacent value columns with context-aware classification.

**b) Fragmented Headers** (Figure 2): Filing agents encode single semantic headers as multiple table rows with cell-level attributes. SEFD reverse-engineers these using border-* and margin-* styling cues, filters candidates by row cardinality and content density, and coalesces rows into unified text blocks.

**c) MultiMarkdown Representation**: MMD encodes horizontal spans with consecutive pipe delimiters (`||`) and vertical spans with caret markers (`^^`), preserving merged-cell structure without HTML tags.

#### XML (~25.7% of SEFD-v1 tokens)
- 33 supported schemas across four archetypes:
  - Transactional/ownership (Forms 3, 4, 5, 13F, 13D/G, 144)
  - Fund/portfolio reporting (N-PORT, N-CEN, N-MFP, N-PX)
  - Primary offerings (Form D, 1-A/K/Z, C, ABS-EE)
  - Entity registration/compliance (Form MA, TA-1/2, ATS-N, X-17A-5)
- Reconstructs document structure from schema elements, optional branches, and repeated records.

#### SGML
- `<SEC-HEADER>` provides indexing metadata (CIK, filing type)
- Investment company filings (Form 497, 24F-2NT) use `<SERIES-AND-CLASSES-CONTRACTS-DATA>` for Series/Class hierarchies

#### PDFs (<2% of corpus)
- Processed with **Mistral OCR 3** in 10-page batches with pixel-variance blank-page filtering
- HTML table output converted to MultiMarkdown for consistency

### 2. Deterministic Normalization

- Removing commas from numerical values
- Dropping non-semantic artifacts (isolated page numbers)
- Prepending metadata (CIK, SIC code) from `<SEC-HEADER>`

---

## Empirical Validation / Results

### Dataset Composition Analysis

**Token share by filing type** (sample of 3.0B tokens): The five largest filing types account for 38.0% of tokens (ABS-EE, 485BPOS, NPORT-P, 8-K, 10-Q). Standard 10-K/10-Q reports account for only 9.6% of tokens, while Form 4 filings are ~25% of filings but just 1.7% of tokens.

**Source format evolution**:

| Year | Plaintext | HTML | XML |
|------|-----------|------|-----|
| 1994 | 98.7% | — | — |
| 2002 | — | 24.1% | — |
| 2010 | — | 83.5% | — |
| 2022 | — | — | 37.8% |
| 2025 | — | — | 44.0% |

**Length distribution**: Filings with 1k–10k tokens are 64.6% of filings but only 6.2% of tokens; filings above 100k tokens are 5.5% of filings but 68.7% of tokens.

### Reconstruction Quality (Table 1)

On 100 complex EDGAR HTML tables, GPT-5.4 (xhigh) asked to reconstruct original HTML from parsed representations:

| Representation | Adjusted Recall |
|----------------|-----------------|
| SEFD (MMD) | **94.5%** |
| EdgarTools (Markdown) | 75.7% |
| Generic to_markdown | 70.4% |

### EDGAR-OCR Results

- **300 hand-selected tables** synthetically transformed (entities, dates, labels, values replaced while preserving arithmetic relationships)
- **241/300** tables have at least one alternate valid transcription (782 alternates total)
- Scoring: 1.0 for exact match (content + formatting + placement), 0.5 for correct text with wrong formatting, 0.25 for correct content in wrong cell

| Model | Adjusted Recall | Median Latency |
|-------|----------------|----------------|
| Qwen3.6-35B-A3B | **75.78%** | 24.58s |
| Mistral OCR 3 | 75.33% | **2.29s** |

### EDGAR-Forecast Results

**250 targets across 50 companies**; models receive 5 years of prior filings, must predict 5 numeric values from hidden 2026 10-Q filings:

| Model | Score |
|-------|-------|
| GPT-5.5 | **51.8%** |
| GPT-5.4 | 48.2% |
| Gemini 3.1 Pro | 45.2% |
| Claude Opus 4.7 | 42.6% |

**Correlation with visible filing history**: GPT-5.5 ($r = +0.38$), Claude Opus 4.7 ($r = +0.22$), GPT-5.4 ($r = +0.20$), Gemini 3.1 Pro ($r = +0.15$).

**Token usage**: 131.6M (GPT-5.4) to 188.8M (Claude Opus 4.7) tokens, mostly cached inputs.

---

## Theoretical and Practical Implications

### For Pretraining Data Curation

1. **Layout-faithful representation matters**: The 94.5% vs. 75.7% reconstruction recall demonstrates that preserving visual structure (tables, indentation, merged cells) enables downstream models to infer original layouts, directly supporting financial reasoning tasks.

2. **Token efficiency without information loss**: MMD's compact syntax achieves >99% token reduction from raw HTML while preserving more semantic structure than standard Markdown or plain text extraction, validating the "quality over quantity" paradigm.

3. **Contamination resistance**: <0.1% overlap with Common Crawl corpora makes SEFD a valuable source for post-cutoff evaluation and RLVR-style dataset construction.

### For Financial AI Applications

- **Long-context pretraining**: 68.7% of tokens come from filings >100k tokens, providing natural long-context training examples.
- **Agentic financial reasoning**: EDGAR-Forecast demonstrates that frontier models can synthesize multi-year filing histories for forecasting, though the 51.8% top score shows substantial room for improvement.
- **Document understanding**: EDGAR-OCR establishes a benchmark for table transcription that distinguishes text accuracy from formatting preservation.

### Benchmark Design Innovations

- **Synthetic transformation** of source tables reduces memorization and retrieval contamination
- **Multiple-answer truth sets** avoid penalizing valid alternative table structures
- **Sandboxed evaluation harness** (Codex) prevents web access, isolating filing-grounded reasoning

---

## Conclusion

SEFD addresses the growing scarcity of clean, long-context training data by converting the SEC EDGAR archive's 18.5M filings (19.8TB raw, 1.9TB parsed) into token-efficient, layout-faithful MultiMarkdown. The corpus spans heterogeneous formats—plaintext, HTML, XML, SGML, and PDF—each requiring specialized reconstruction strategies that preserve financial meaning encoded in visual structure.

Key takeaways:

1. **SEFD is a practical resource** for financial language modeling, long-context pretraining, financial reasoning, document understanding, and evaluation, comparable to ~1.3B pages of reconstructed filing content.

2. **Training signal is highly uneven** across filing type, source format, and length—high-frequency forms are not necessarily high-token forms, and a small tail of long filings contributes most corpus mass.

3. **The benchmarks reveal current model limitations**: EDGAR-Forecast's 51.8% top score and EDGAR-OCR's 75.78% top score suggest significant headroom for improvement in financial document understanding and forecasting.

### Future Directions

- Expanding XML schema coverage to rare conditional branches
- Extending SEFD-v1 backward from 2022 to include pre-2022 filings
- Using SEFD for RLVR (Reinforcement Learning from Verifiable Rewards) dataset construction
- Improving OCR formatting preservation (bold, italic, superscript/subscript remain challenging)

The paper positions SEFD as a foundational resource that "expands EDGAR beyond retrieval-augmented search, supporting model evaluation for financial, business, and accounting tasks as well as pretraining and RLVR-style dataset construction."

---

_Markdown view of https://picx.dev/p/6pdG96, served by PicX — AI-generated visual whiteboard summaries of research papers._