The Stanford EDGAR Filings Dataset (SEFD): A Comprehensive Summary

Summary (Overview)

• SEFD is an open reconstruction of the complete SEC EDGAR archive (1994–present) into layout-faithful MultiMarkdown (MMD), designed for financial language modeling and evaluation. The full corpus is estimated at 550B tokens across 18.5M filings; SEFD-v1 releases a 152B-token snapshot covering January 2022–June 2025.

• The dataset employs a "visual-first" parsing methodology that reconstructs spatial semantics from rendered documents rather than relying on naive DOM-based text extraction, preserving tables, indentation, and visual hierarchy that carry financial meaning.

• SEFD achieves >99% structural and semantic accuracy (per human evaluation) with <0.1% overlap with Common Crawl-derived corpora, making it a scarce source of clean, long-context financial data.

• Two novel benchmarks are introduced: EDGAR-Forecast (filing-grounded numerical forecasting after knowledge cutoffs; best model GPT-5.5 achieves 51.8%) and EDGAR-OCR (transcription of complex financial tables; Qwen3.6-35B-A3B achieves 75.78%).

• The corpus is token-efficient—clean reconstruction yields >99% token reduction relative to raw source, as much filing HTML consists of presentation scaffolding rather than filer-authored content.

---

Introduction and Theoretical Foundation

The "Quality over Quantity" Paradigm

The paper situates SEFD within the broader shift in LLM pretraining from brute-force scaling toward data-centric curation. Key motivating evidence:

• Diminishing returns from scaling alone: GPT-4.5 and Llama 4 Behemoth show reduced gains even at 30T+ training tokens.
• Microsoft's Phi line demonstrates that "textbook-quality" curation can outperform larger models on common benchmarks.
• As architectures and training recipes converge, data itself becomes the primary differentiator of performance.

Why EDGAR?

EDGAR is one of the largest public repositories of factual, business-oriented long-context data, processing ~4,700 filings daily and ~40,000 new filers annually. However, it has remained largely untapped for pretraining because its 18.5M filings present broad parsing challenges:

• Layout engineering: Filings use table-based layout (<table>, <tr>, <td>), presentational markup, and explicit whitespace ( , <br>) constrained by EDGAR's HTML 3.2/4.0-era restrictions.
• Fragmentary structure: Visual elements are exploded across multiple cells/rows to maintain print-like appearance in browsers, disconnecting semantically related content.
• Format heterogeneity: Plaintext, HTML, XML, SGML, and PDF filings coexist across different eras.

Key Insight

"Flattening EDGAR tables can create ambiguity by detaching values from labels, duplicating headers, or flipping accounting signs."

Indentation disambiguates statement hierarchies, merged headers connect periods and segments, and numeric-cell reconstruction reattaches signs, currency symbols, and percentage symbols to their values.

---

Methodology

1. Format-Specific Parsing Pipeline

SEFD routes filings across 33 specialized XML schemas and over 350 filing types based on the SGML header's CONFORMED SUBMISSION TYPE, then applies format-specific reconstruction:

Plaintext (Legacy ASCII)

• Preservation strategy: Fixed-width whitespace is semantically meaningful, so no normalization is applied beyond collapsing 3+ consecutive line breaks into 2.
• Identified by regex-based check for absence of structural HTML tags.

HTML (~62% of corpus tokens)

The most complex case, addressing three key reconstruction challenges:

a) The "Three-Column Hack" (Figure 1): Numerical values are exploded across three cells for decimal alignment:

• Prefix/Gutter Column (width="1%"): holds currency symbols or opening parentheses
• Value Column: integer and decimal
• Suffix Column: closing parentheses or percentage signs

Symbol Reconstruction uses a bidirectional rule-based merging algorithm that scans for "modifier columns" (exclusively currency symbols, percentages, or parenthetical fragments) and merges them into adjacent value columns with context-aware classification.

b) Fragmented Headers (Figure 2): Filing agents encode single semantic headers as multiple table rows with cell-level attributes. SEFD reverse-engineers these using border-* and margin-* styling cues, filters candidates by row cardinality and content density, and coalesces rows into unified text blocks.

c) MultiMarkdown Representation: MMD encodes horizontal spans with consecutive pipe delimiters (||) and vertical spans with caret markers (^^), preserving merged-cell structure without HTML tags.

XML (~25.7% of SEFD-v1 tokens)

• 33 supported schemas across four archetypes:
  • Transactional/ownership (Forms 3, 4, 5, 13F, 13D/G, 144)
  • Fund/portfolio reporting (N-PORT, N-CEN, N-MFP, N-PX)
  • Primary offerings (Form D, 1-A/K/Z, C, ABS-EE)
  • Entity registration/compliance (Form MA, TA-1/2, ATS-N, X-17A-5)
• Reconstructs document structure from schema elements, optional branches, and repeated records.

SGML

• <SEC-HEADER> provides indexing metadata (CIK, filing type)
• Investment company filings (Form 497, 24F-2NT) use <SERIES-AND-CLASSES-CONTRACTS-DATA> for Series/Class hierarchies

PDFs (<2% of corpus)

• Processed with Mistral OCR 3 in 10-page batches with pixel-variance blank-page filtering
• HTML table output converted to MultiMarkdown for consistency

2. Deterministic Normalization

• Removing commas from numerical values
• Dropping non-semantic artifacts (isolated page numbers)
• Prepending metadata (CIK, SIC code) from <SEC-HEADER>

---

Empirical Validation / Results

Dataset Composition Analysis

Token share by filing type (sample of 3.0B tokens): The five largest filing types account for 38.0% of tokens (ABS-EE, 485BPOS, NPORT-P, 8-K, 10-Q). Standard 10-K/10-Q reports account for only 9.6% of tokens, while Form 4 filings are ~25% of filings but just 1.7% of tokens.

Source format evolution:

Length distribution: Filings with 1k–10k tokens are 64.6% of filings but only 6.2% of tokens; filings above 100k tokens are 5.5% of filings but 68.7% of tokens.

Reconstruction Quality (Table 1)

On 100 complex EDGAR HTML tables, GPT-5.4 (xhigh) asked to reconstruct original HTML from parsed representations:

EDGAR-OCR Results

• 300 hand-selected tables synthetically transformed (entities, dates, labels, values replaced while preserving arithmetic relationships)
• 241/300 tables have at least one alternate valid transcription (782 alternates total)
• Scoring: 1.0 for exact match (content + formatting + placement), 0.5 for correct text with wrong formatting, 0.25 for correct content in wrong cell

EDGAR-Forecast Results

250 targets across 50 companies; models receive 5 years of prior filings, must predict 5 numeric values from hidden 2026 10-Q filings:

Correlation with visible filing history: GPT-5.5 ($r = +0.38$), Claude Opus 4.7 ($r = +0.22$), GPT-5.4 ($r = +0.20$), Gemini 3.1 Pro ($r = +0.15$).

Token usage: 131.6M (GPT-5.4) to 188.8M (Claude Opus 4.7) tokens, mostly cached inputs.

---

Theoretical and Practical Implications

For Pretraining Data Curation

1. Layout-faithful representation matters: The 94.5% vs. 75.7% reconstruction recall demonstrates that preserving visual structure (tables, indentation, merged cells) enables downstream models to infer original layouts, directly supporting financial reasoning tasks.

2. Token efficiency without information loss: MMD's compact syntax achieves >99% token reduction from raw HTML while preserving more semantic structure than standard Markdown or plain text extraction, validating the "quality over quantity" paradigm.

3. Contamination resistance: <0.1% overlap with Common Crawl corpora makes SEFD a valuable source for post-cutoff evaluation and RLVR-style dataset construction.

For Financial AI Applications

• Long-context pretraining: 68.7% of tokens come from filings >100k tokens, providing natural long-context training examples.
• Agentic financial reasoning: EDGAR-Forecast demonstrates that frontier models can synthesize multi-year filing histories for forecasting, though the 51.8% top score shows substantial room for improvement.
• Document understanding: EDGAR-OCR establishes a benchmark for table transcription that distinguishes text accuracy from formatting preservation.

Benchmark Design Innovations

• Synthetic transformation of source tables reduces memorization and retrieval contamination
• Multiple-answer truth sets avoid penalizing valid alternative table structures
• Sandboxed evaluation harness (Codex) prevents web access, isolating filing-grounded reasoning

---

Conclusion

SEFD addresses the growing scarcity of clean, long-context training data by converting the SEC EDGAR archive's 18.5M filings (19.8TB raw, 1.9TB parsed) into token-efficient, layout-faithful MultiMarkdown. The corpus spans heterogeneous formats—plaintext, HTML, XML, SGML, and PDF—each requiring specialized reconstruction strategies that preserve financial meaning encoded in visual structure.

Key takeaways:

1. SEFD is a practical resource for financial language modeling, long-context pretraining, financial reasoning, document understanding, and evaluation, comparable to ~1.3B pages of reconstructed filing content.

2. Training signal is highly uneven across filing type, source format, and length—high-frequency forms are not necessarily high-token forms, and a small tail of long filings contributes most corpus mass.

3. The benchmarks reveal current model limitations: EDGAR-Forecast's 51.8% top score and EDGAR-OCR's 75.78% top score suggest significant headroom for improvement in financial document understanding and forecasting.

Future Directions

• Expanding XML schema coverage to rare conditional branches
• Extending SEFD-v1 backward from 2022 to include pre-2022 filings
• Using SEFD for RLVR (Reinforcement Learning from Verifiable Rewards) dataset construction
• Improving OCR formatting preservation (bold, italic, superscript/subscript remain challenging)

The paper positions SEFD as a foundational resource that "expands EDGAR beyond retrieval-augmented search, supporting model evaluation for financial, business, and accounting tasks as well as pretraining and RLVR-style dataset construction."