# Beyond Rephrasing: Book-Level Organization Improves Synthetic Textbook Data for Mid-Training

> Book-level organization of synthetic training data, not content alone, improves LLM mid-training by over one point across 28 benchmarks.

- **Source:** [arXiv](https://arxiv.org/abs/2607.28109)
- **Published:** 2026-08-22
- **Permalink:** https://picx.dev/p/5Pt6Rt
- **Whiteboard:** https://picx.dev/p/5Pt6Rt/image

## Summary

## Summary (Overview)

- **Core finding**: Book-level organization of synthetic textbook data—not just content quality or local rewriting—significantly improves mid-training performance in language models. The authors demonstrate this through controlled experiments isolating document packaging, length, and structured synthesis.
- **Scalable pipeline**: A five-stage retrieval-grounded synthesis pipeline produces 686K textbooks (32B tokens) across 15,000+ disciplines, guided by a four-level taxonomy covering natural sciences, engineering, humanities, medicine, and agriculture.
- **Key result**: Replacing natural books in a mid-training mix with the synthetic corpus improves downstream performance by **+1.09** on average across 28 benchmarks, with gains in all four categories (STEM, Knowledge, Reasoning, Code).
- **Controlled isolation**: A content-matched Split condition (sections as independent documents) trails Full by +1.02; a length-matched RandomConcat control trails by +1.01; and a retrieval-pool-matched Rephrase baseline trails by +1.17—demonstrating that book-level structure, not just content or length, drives gains.
- **Cross-architecture validation**: Results transfer to Llama3-8B, where Full outperforms both RandomConcat (+0.86) and Natural Books (+1.51), supporting book-level organization as a general design principle for synthetic pre-training data.

## Introduction and Theoretical Foundation

Large language models derive their capabilities primarily from the scale, quality, and structure of pre-training data. Prior work on synthetic textbooks (e.g., "Textbooks Are All You Need," Cosmopedia, WRAP) has emphasized content—the knowledge and explanations written into the text—or local rewriting style. However, the question of **how that content is organized**—assembled into coherent books rather than left as a flat collection of passages—remains underexplored.

The authors argue that organization is not incidental:

- **Generation side**: A book-level plan provides the scaffold needed to turn scattered retrieved materials into long, structured, coherent textbook data rather than a collection of locally plausible passages.
- **Training side**: Preserving that structure determines whether planned adjacent sections remain in a shared document with continuous positions and shared intra-document attention, or are split into independent examples with resets between sections.

The central research question: **Does book-level organization add value both when constructing synthetic data and when preserving that data as training documents, beyond the content itself or the local cleanup obtained by rephrasing individual documents?**

Related work spans three areas:
1. **Synthetic textbooks and rewriting** (TinyStories, Cosmopedia, WRAP, LiteLong, ACER)—improve coverage, grounding, and pedagogy but typically produce seed-level documents, long-context concatenations, or domain-specific curricula.
2. **Data curation, grounding, and mid-training**—retrieval-augmented methods use retrieved documents for factuality and domain adaptation; this work uses retrieval to transform corpus material into long-form, book-organized training documents.
3. **Document boundaries, packing, and sequence composition** (In-Context Pretraining, SPLiCe, best-fit packing)—studies how training examples are assembled into sequences; the Full vs. Split comparison asks the analogous question for synthetic textbooks.

## Methodology

### Pipeline Overview

The five-stage pipeline transforms raw pre-training corpora into structured, textbook-form training data:

**Phase I (Knowledge Extraction, Steps 0–2)**:
- **Discipline taxonomy**: Four-level taxonomy (62 L1 / 754 L2 / 2,730 L3 nodes from the Chinese National Standard GB/T 13745, plus 12,657 LLM-expanded L4 subfields), totaling 15,387 disciplines.
- **Query generation**: For each leaf discipline, an LLM generates 10–15 diverse statement-form queries covering different knowledge angles. Example: "The backpropagation algorithm efficiently computes gradients of a loss function with respect to all network parameters."
- **Retrieval and clustering**: Each query retrieves top-100 passages via keyword matching; retrieved documents are chunked (~4K characters), embedded, and clustered via KMeans (K=40). Each cluster becomes a coherent subtopic mapped to chapters or sections.

**Phase II (Structured Generation, Steps 3–4)**:
- **TOC planning**: The planner converts the cluster map into a hierarchical TOC (chapters → sections → leaf sections), with each leaf specifying a title, learning objective, source cluster IDs, and token-budget estimate. TOCs are generated for 4 audiences × 4 styles = 16 variants per discipline.
- **Quality gate**: An LLM judge evaluates each TOC for coherence, audience/style appropriateness, and grounding; failed TOCs are discarded (91.8% pass rate overall).
- **Source-grounded generation**: For each TOC leaf, several source chunks (k=10) are sampled from mapped clusters as context. The model writes section content conditioned on retrieved evidence.
- **Assembly**: Sections are assembled into complete books following the TOC hierarchy; multiple books are generated per TOC variant by resampling source chunks.

### Experimental Design

**Training conditions** (all with fixed token budgets, sequence packing, and intra-document attention masking):
- **Full (ours)**: TOC-planned textbooks as complete training documents.
- **Split**: Identical text and tokens, but each section is an independent document (positions and intra-document attention reset).
- **RandomConcat**: Sections from different books grouped to match Full's document-length distribution (separates length from planned adjacency).
- **Rephrase**: Independently rewrites documents from the same retrieval pool under the same audience×style scheme, without clustering, TOC planning, or book assembly.
- **Natural Books**: Original book slice from the pre-training mix (unmatched baseline).

**Models**: Five matched variants of a 3B-active-parameter MoE (30B total) with 200B-token mid-training mix; transfer check on Llama3-8B with 100B-token mixture. The book slice is 16B tokens (8%) in the MoE study and 8B tokens (8%) in the Llama study.

**Evaluation**: 28 benchmarks spanning STEM, knowledge/multilingual, reasoning/math word problems, and code; multiple-choice tasks use few-shot likelihood scoring, generative tasks use greedy decoding (temperature 0).

## Empirical Validation / Results

### Main Results (3B-Active MoE)

**Full vs. Natural Books**: +1.09 overall (55.90 vs. 54.81), with gains in all categories:
- STEM: +1.86 (47.86 vs. 45.99)
- Reasoning: +1.49 (78.78 vs. 77.29)
- Code: +1.07 (46.18 vs. 45.11)
- Knowledge: +0.43 (55.04 vs. 54.61)

**Full vs. Split** (content-identical, packaging differs): +1.02 overall (55.90 vs. 54.88), winning 22/28 benchmarks. Largest gains in code (+2.23) and reasoning (+1.35).

**Full vs. RandomConcat** (length-matched): +1.01 (55.90 vs. 54.89), winning 21/28 benchmarks—planned adjacency matters beyond length.

**Full vs. Rephrase** (same retrieval pool, local rewriting only): +1.17 (55.90 vs. 54.73), winning 25/28 benchmarks. Rephrase ties Natural Books (−0.08), showing local rewriting alone is insufficient.

### Llama3-8B Transfer Results

| Category | Full | Random-Concat | Natural Books | Δ_F−R | Δ_F−N |
|---|---|---|---|---|---|
| STEM | 27.35 | 25.86 | 23.70 | +1.49 | +3.65 |
| Knowledge | 42.42 | 42.17 | 41.90 | +0.24 | +0.52 |
| Reasoning | 68.83 | 67.67 | 67.63 | +1.17 | +1.20 |
| Code | 28.94 | 27.92 | 27.89 | +1.02 | +1.05 |
| **Overall (28)** | **40.96** | **40.10** | **39.45** | **+0.86** | **+1.51** |

Statistical significance: One-sided sign tests give $p = 0.044, 0.0019$, and $0.000014$ for Full against Natural Books, Split, and Rephrase, respectively.

### Ablations

**Source chunk count (k)**: From k=1 to k=10, overall/structure/educational scores improve by +0.22/+0.16/+0.13; from 10 to 30 chunks, overall improves only +0.05. **k=10 selected** (~47K input characters per section).

**Pipeline components** (100 matched book instances, Gemini-3.1-Pro judge, 1–10 scale):
- **Full vs. Baseline C** (no TOC, no clustering, no source chunks): 9.38 vs. 6.12 (−3.26)—structured generation has the largest effect.
- **Full vs. Baseline A** (no source chunks): grounding improves specificity (+0.23), examples (+0.35), factual accuracy (+0.32).
- **Baseline A vs. B** (no cluster-informed TOC): cluster-informed planning improves audience fit (+0.59), structure (+0.44), pedagogy (+0.23).
- **Quality gate**: Failed TOCs score −0.25 overall, −0.20 structure, −0.19 pedagogy—a safety net rather than primary quality driver.

## Theoretical and Practical Implications

- **Book-level organization is a first-class data-design axis**: The results establish that how synthetic content is organized—both at generation time (clustering, TOC planning) and at training time (preserving book boundaries)—is as important as content quality itself. This extends prior work that focused primarily on content generation or local rewriting.

- **Two complementary effects**: 
  1. **Generation side**: Clustering and hierarchical TOC planning determine which knowledge is combined and how it progresses; book assembly realizes this plan as a coherent document.
  2. **Training side**: Preserving book structure exposes planned adjacency and continuous intra-document attention to the model, which Split and RandomConcat destroy.

- **Practical guidance for synthetic data pipelines**: The component ablations provide actionable insights—retrieval grounding chiefly adds domain-specific details, cluster-informed planning improves structure and audience fit, and hierarchical section-wise generation supplies book-scale depth. The quality gate acts as a cheap safety net (8.2% rejection rate) that prevents structural defects from propagating.

- **Efficiency considerations**: k=10 source chunks captures most of the quality benefit at ~47K input characters per section; increasing to k=30 nearly triples input for marginal gains. The pipeline requires a searchable corpus index, adding preprocessing overhead, but final text generation costs are comparable to simpler rewriting approaches (20K vs. 19.7K H20 GPU-hours for Full vs. Rephrase).

## Conclusion

This work demonstrates that **book-level organization is central to the effectiveness of synthetic textbooks for mid-training**, beyond the content itself or local rewriting. The retrieval-grounded pipeline scales to 686K books (32B tokens) across 15,000+ disciplines, and controlled comparisons isolate two complementary effects:

1. **Structured synthesis outperforms local rewriting**: Clustering, TOC planning, and book assembly add value beyond independently rewriting individual documents (Full beats Rephrase by +1.17).
2. **Preserving book structure benefits training**: Content-matched Split and length-matched RandomConcat both trail Full, showing that planned adjacency and document continuity matter.

The authors will release a research-licensed subset of the synthetic corpus and provide synthesis/control-construction code as supplementary material.

**Limitations**: Each condition is trained once (training-run variance unmeasured); the pipeline requires a searchable corpus index; component ablations rely on a fixed LLM judge and should be interpreted as diagnostic rather than downstream evidence.

**Future directions** implied by this work include: optimizing the synthetic-data proportion (the fixed 8% allocation was not tuned), exploring book-level organization in other training regimes (continued pre-training, instruction tuning), and investigating whether even longer coherent documents (beyond the ~50K-token book scale) yield further gains.

---

_Markdown view of https://picx.dev/p/5Pt6Rt, served by PicX — AI-generated visual whiteboard summaries of research papers._