Summary (Overview)

• AFCS (Atomic-Fact-Conditioned Synthesis) is a novel pipeline that extracts source-supported atomic facts from documents and uses each retained fact as an explicit generation target for synthetic pretraining data, rather than conditioning generation on whole passages or segments.
• Key finding: Source-conditioned synthesis (WRAP, Nemotron-CC) often omits fine-grained facts—secondary entities, list/table entries, and precise temporal attributes—while AFCS achieves 85.1% source-fact coverage vs. 73.0% (WRAP) and 77.0% (Nemotron-CC).
• Performance gains: AFCS outperforms baselines on SimpleQA, SimpleQA Verified, PopQA, and HotpotQA at both 7B and 32B scales, with the largest gains on middle- and long-tail factual recall (PopQA).
• Granularity matters: Fact-level conditioning outperforms chunk-level, paragraph-level, and entity-level synthesis under matched token budgets.
• Mid-training integration: Replacing 5B tokens in a 100B-token OLMo-3 mid-training mixture with AFCS data improves the 12-task general benchmark average by +1.09.

Introduction and Theoretical Foundation

The paper addresses a fact-coverage issue in knowledge-corpus synthesis for language model pretraining. Existing methods (WRAP, Nemotron-CC) take a source passage or document segment as input and generate rewritten or QA-style outputs. While these methods improve surface form and prompt diversity, they leave fact selection implicit—the generator decides which source facts to mention, often preserving salient entities and headline events while omitting:

• Secondary entities (people, organizations, locations mentioned infrequently)
• List- and table-embedded facts (e.g., individual year–recipient rows)
• Fine-grained temporal attributes (coarsened from month/day to year-level)

Theoretical framing: The paper formalizes this as a source-fact selection problem rather than a ground-truth coverage metric. For a source unit $d$, let $\mathcal{F}(d) = \{f_1, \ldots, f_{n_d}\}$ denote the atomic facts expressed in that unit. An atomic fact is defined as:

A minimal source-supported assertion that can be verified against a local evidence span.

Crucially, atomic facts need not be centered on canonical entities or correspond to canonical entity–relation edges. Records are stored in subject–predicate–value-like form for convenience, but subjects can be local anchors (table rows, list items, dates, numerical values). This distinguishes AFCS from entity- or graph-conditioned methods (EntiGraph, Synthesize-on-Graph) which typically provide entities, attributes, or relation edges as synthesis context, leaving the generator to determine which facts are ultimately realized.

Methodology

AFCS is a four-stage pipeline:

1. Structure-Aware Atomic Fact Extraction

Wikipedia pages are processed using their natural document tree (title, sections, paragraphs, lists, tables, infoboxes). Different prompt templates handle tables, lists, numeric-heavy content, and prose separately. Facts are represented as assertion-centric units containing:

• The statement to be realized
• Its target value
• The local evidence span for verification and realization

This representation covers non-canonical assertions such as evidential absence, conditional statements, causal explanations, reported speech, and full-clause values.

2. Source-Grounded Verification

Extracted records are checked against their source spans before synthesis. This filters unsupported values and preserves source offsets for later context expansion. The verifier reduces unsupported facts before they are amplified into multiple synthetic samples.

3. Multi-Format Generation

For each retained fact $f$ and its local context $\text{ctx}(f)$, AFCS generates complementary natural-language samples:

• Summary: Contextual summaries preserving nearby factual relations
• Rewrite: Paraphrases in alternative registers
• Continuation: Continuations from a local prefix that must include the target fact
• QA: Answer-guided question generation (the fact value is supplied as the target answer)

4. Corpus-Scale Construction

Applied to ~6.7M FineWiki articles, the pipeline produces:

• 312.7M atomic facts (34.9% prose, 29.2% tables/infoboxes, 19.6% numeric attributes, 16.3% lists)
• 142M natural-language synthetic samples across four formats

Implementation details: The extractor, verifier, and generator all use Qwen3-30B-A3B served with vLLM. Extraction/verification use deterministic decoding (temperature 0); generation uses temperature 0.7. QA samples are deduplicated by exact question match, fact ID (≤2 questions per fact), and character-trigram overlap.

Empirical Validation / Results

Extraction Quality (500-article author-checked subset)

Main Factual QA Results (Table 2)

At 7B scale (OLMo-3-7B, 20B tokens):

At 32B scale:

Source-Fact Coverage (Table 3)

AFCS achieves 85.1% overall coverage vs. 73.0% (WRAP) and 77.0% (Nemotron-CC). For documents with 100+ facts, AFCS reaches 82.5% vs. 61.2% (WRAP) and 66.5% (Nemotron-CC).

Granularity and Mechanism Ablations (Table 4, 7B scale)

Popularity-Stratified PopQA (Table 6)

AFCS gains are largest on rare entities (0–100 monthly views): +5.7 points over WRAP at 7B, +7.7 at 32B. In the 10K+ bucket, all methods are close, suggesting prompt diversity is competitive for popular entities.

Mid-Training Integration (Table 7)

Replacing 5B general tokens with AFCS in a 100B-token OLMo-3 mid-training mixture:

Pass@k Results (Table 5, SimpleQA Verified)

AFCS is consistently strongest at all $k$ values. At 7B, pass@64 reaches 37.50 vs. 34.50 (WRAP) and 34.40 (Nemotron-CC).

Theoretical and Practical Implications

Theoretical implications:

• The paper reframes synthetic-data quality as a fact-selection problem rather than a surface-form problem. Prompt diversity (WRAP) and multi-prompt mixtures (Nemotron-CC) do not fully address which facts are covered.
• Atomic facts as conditioning units are more effective than entities/triples because they can represent non-canonical assertions (negation, evidential absence, conditionality, full-clause values) not naturally expressed as single entity–relation edges.
• The results suggest a coverage–acquisition distinction: explicit source-fact coverage (corpus property) and model-side acquisition (measured via QA benchmarks) are complementary but not causally linked.

Practical implications:

• AFCS provides a scalable, one-time data-construction pipeline (offline inference cost separate from training compute) that can be integrated into existing mid-training recipes.
• The 32B results demonstrate that benefits persist at larger scales.
• The mid-training integration shows AFCS can replace a small fraction of general tokens (5%) and improve both factual benchmarks and general capabilities.

Conclusion

AFCS addresses a coverage failure mode in knowledge-corpus synthesis by making fact selection explicit before generation. Key takeaways:

1. Fact-level conditioning outperforms coarser units (chunk, paragraph, entity) under matched budgets.
2. Multi-format generation, repeated realizations per fact, and broader fact coverage each contribute to gains.
3. AFCS is strongest on middle- and long-tail knowledge, where source-conditioned methods struggle most.
4. Integration with existing pipelines works: replacing 5B tokens in mid-training improves factual QA and the 12-task average to 69.33.

Future directions identified by the authors:

• Stronger extraction verification
• Multilingual extension
• Combining atomic-fact-conditioned synthesis with corpus filtering and retrieval

Limitations: Extractor dependence (residual extraction errors propagate), coverage does not guarantee learning, scale limited to 7B/32B, English Wikipedia only, and no comparison against retrieval-augmented systems (which are complementary rather than competing approaches).