Summary of "The Announcement Carries the Cue: Markup, Boundaries, and the Notation of Pre-Training Corpora"

Summary (Overview)

• Core thesis: The notation (markup/formatting) of pre-training corpora is an unmeasured and unrecorded variable that affects model behavior. The paper defines clean-window survival $S(W)$ — the fraction of non-overlapping $W$-token windows containing no structural markup — as a deterministic, auditable measure of how much of a corpus "demands boundary inference."
• Key empirical findings: (1) A census of 13 public corpora shows clean-window survival ranges from 0.153 (olmOCR slice of Dolma 3) to 0.889 (C4); (2) deleting structural announcements (chapter headings, etc.) makes following prose measurably harder to predict across 5-6 base models (0.6B–8.2B parameters), while swapping notation (Markdown vs. plain text) has zero effect; (3) base models do not re-impose the marked register on unmarked prose.
• Central conceptual claim: The operative cue is the announcement (a short line saying "a boundary is here"), not the sigil (the markup character itself). Flattening notation while keeping announcement lines changes nothing; deleting announcements entirely is the only manipulation that changes reading behavior.
• Proposed intervention: The pure frame format — paragraphs in authored order with every structural announcement deleted into a reversible sidecar, mixed with the marked copy to destroy cue reliability at the long-context training stage.
• Recommendation: Choose format operators by the capability they train, not by the fidelity they preserve; record extractor identity and survival statistics on data cards.

Introduction and Theoretical Foundation

The paper opens with a striking demonstration: deleting all structural announcements from Moby-Dick (143 lines, 1,442 tokens, 0.51% of the file) produces a continuous 283,267-token stretch with no surface markers of chapter boundaries. The field has established that text-extraction choices change model behavior but has "never once measured the notation of what those choices put into the corpus."

Key theoretical constructs:

• Notation: "the distribution, across a corpus, of how documents' arrangement is written down" — a decision currently made "by accident, at no apparent cost, by people optimizing something else."
• Serialization register vs. authorial composition register: The former is the notation a text is stored in (set by pipelines); the latter is whether the writer thought in structure at all (set by history). Pipelines "cannot create authorial composition register; they can only preserve or destroy the evidence of it."
• The core mechanism: A Markdown heading is "a two-token cue that is present at every discourse boundary and... close to absent everywhere else." Shortcut-learning literature shows what governs feature adoption is not frequency but reliability and availability; the induction-head scaling work puts the sensitivity to reliability at "more than twice the sensitivity to frequency."

The historical narrative shows the current state emerged accidentally: C4's line filter (keeping only lines ending in terminal punctuation) discarded headings; The Pile used jusText "based on visual inspection"; FineWeb left trafilatura in default plain-text mode; DCLM chose resiliparse for speed; Books3 used html2text at library defaults (writing .md files). The Llama 3 pipeline deliberately removed Markdown as "harmful," while the frontier moved to vision-model PDF conversion that emits structure "by design."

Methodology

Core instrument — clean-window survival:

$$S(W) = \frac{\text{number of non-overlapping } W\text{-token windows with no structural markup}}{\text{total number of such windows}}$$

The distribution of clean runs (maximal markup-free spans) is measured alongside. Code fences and table pipes are excluded from the core tag set. All numbers use the o200k_base tokenizer.

Three fronts of measurement:

1. What corpora carry: Census of 13 public corpora (Table 1), with 6 rows re-verified (Table 2). Pre-registered supply test with three candidates: UK Hansard (proceedings), Creative-Commons YouTube transcripts, and pre-1929 library-scan books. A corpus "supplies the missing format" at horizon W iff median document ≥ W tokens AND S(W) ≥ 0.50.

2. What readers use: Three-arm probe over 21 works — marked (Markdown), flat (heading words kept, notation stripped), and silent (announcement line deleted). Targets are ordinary prose, scored with 768 and 7,936 tokens of context; information gain is the difference in millinats per byte. Runs on 5 base models (0.6B–8.2B) from two pipelines, plus a whole-book curve on Moby-Dick.

3. What writers impose: Same three serializations, four base models continue contexts under greedy decoding, scored for markup emitted per 1,000 tokens, boundary-insertion rate, and announcement rate. Reference level is the author's own continuation (not zero), because authored text carries markup too.

Pre-registration: All decision rules, strata, horizons, and thresholds were fixed before the runs. A "falsifier ledger" (Table 9) pre-commits responses to each possible outcome.

Empirical Validation / Results

Front I: What corpora carry

Key findings:

• Heading presence swings from 0.12% of documents (FineWeb) to 33.3% (Dolma 3 olmOCR), over a hundredfold change.
• "The scarce resource is not unmarked text; it is long unmarked text." All web corpora have p99 clean runs between 2,400 and 6,500 tokens because their median document is only ~244 tokens.
• No source supplies all three properties (long, clean, coherent) at once.

Pre-registered supply test: UK Hansard confirms at both horizons (S(8,192) = 0.930, S(32,768) = 0.802); YouTube fails on length (median 1,779 tokens); pre-1929 library scans fail on density (92.5% carry markup, S(8,192) = 0.413). "The long, unmarked speech supply is institutional, not consumer."

Converter null (E2): Pre-registered prediction that converters would fabricate structure on prose — failed. Imposition loss is significantly negative (-0.029 [-0.047, -0.010] at W = 8,192), meaning converters slightly under-mark. "Modern converters slightly under-mark. They are not inventing boundaries; they are faithfully marking real ones." The null killed the fabrication mechanism; the replacement is the reliability mechanism: "A faithful cue is a trustworthy one, so accuracy in conversion is not a mitigation. It is the problem."

Notation invariance: Flattening notation while keeping heading words changes where boundaries are detectable by at most 0.010 at any horizon. "The sigil is not the cue. The announcement is."

Cue reliability: Pre-registered test of "the cue never lies" — verdict UNDERMINED as stated. Sustained argument returns precision 0.765 (vs. required 0.99), dragged down by one document's endnote apparatus. Narrative (the material the proposal addresses) holds at 0.994, or 0.967 adjusted, in the SUPPORTED, WEAKENED band.

Front II: What readers use

R1 (deficit persists): Deleting the announcement raises prediction difficulty at all five reader scales:

• 0.6B: +0.466 [+0.159, +0.770] millinats/byte
• 1.7B: +0.223 [+0.017, +0.436]
• 4B: +0.571 [+0.195, +1.011]
• 7.3B: +0.578 [+0.288, +0.849]
• 8B: +0.459 [+0.123, +0.883]

R3 (notation swap is zero): All five readers' notation contrast contains zero, and tight against the pre-registered 2% bound (widest endpoint 1.68%). "A zero does not say that notation is unimportant. It says the sigil is not the operative cue."

R2 (reconstruction): Fired at 1.7B (+0.270 [+0.010, +0.521]) but fails to replicate at 4B, 7.3B, or 8B. Reported as a headline finding despite non-replication.

Whole-book curve: The pure frame costs +0.0335 nats/byte over the first 16 tokens after a boundary, +0.0085 over the next 48, +0.0030 from 65–256, +0.0009 from 257–1,024, and nothing beyond. "The announcement is worth about a thousand tokens."

Replication: A sixth reader (SmolLM3-3B) from a third pipeline replicates the deficit (+0.370 [+0.040, +0.758]) and the swap null.

Front III: What writers impose

• I1 (writers impose the register): Not satisfied — bounded null. None of four base models' markup emission is distinguishable from the author's (intervals all touch zero).
• I3 (restore register at silent boundaries): Not satisfied — the front's robust result. All four readers return 0.000 [0.000, 0.000] announcement rate at deleted boundaries. "A base model tolerates the pure frame; it does not scaffold it."
• I2 (imposition is input-dependent): Satisfied, but carried entirely by single-anchor cells with zero-width intervals — treated as unevaluable.
• I4 (imposition scales): Not satisfied, indeterminate rather than flat.
• Greedy decoding collapse: 84–100% of continuations flagged for repetition, leaving only 77/1,328 generations. Sampling at temperature 1.0 eliminates collapse, confirming "the collapse is a property of the decoding regime rather than of the models."

Theoretical and Practical Implications

The pure frame format: Paragraphs in authored order, every structural announcement deleted, every deletion recorded in a reversible sidecar. The validator passes a file only if it contains zero announcements and every record's sidecar round-trips byte-exactly.

The mixture: Not a mixture over notation (Markdown vs. plain), but over announcement presence: the same work present both marked and pure, so "at the same underlying event the announcement is sometimes there and sometimes not." This destroys the cue's reliability at the level where reliability does its work.

The central bet: "We are betting that the factorization exists and that reliability is what forces it: that a model trained on the same arrangement with and without its announcements forms a discourse channel separable from the surface channel that renders it." If the paired arm of the three-arm test is indistinguishable from the marked arm, "the factorization story collapses to a notation preference."

The decisive test: Three arms — marked (current practice), pure (the format), and paired (half marked/half pure, reliability ~0.5) — at 100B tokens per arm, costing ~3,100 H100-hours (1.4% of an OLMo-3-7B run).

Conclusion

The paper concludes that nobody deliberately decided the pre-training stream should stop demanding structural inference; it "fell out of a line filter here and a default output mode there, a books subset quietly dropped, a long-context slot filled with whatever was longest." Each step improved the number being watched while the cost landed on an uninstrumented dimension.

The recommendation is twofold: record it (a data-card field for extractor identity and S(W), generalizing a field FinePDFs already carries) and build it (a public-domain corpus, a deletion, a sidecar, and a validator, aimed at the long-context stage that is under 1% of the budget).

The paper's central sentence: "Choose format operators by the capability they train, not by the fidelity they preserve."