Scaling Laws and the Data Substrate: An Attribution Framework for Post-Training Reasoning Data

Abstract

The rapid advancement of large language models (LLMs) has been substantially driven by post-training on reasoning data. However, the field's understanding of what constitutes a "good" reasoning dataset remains fragmented and often conflates distinct concepts. This primer provides a comprehensive synthesis of the current landscape of post-training reasoning data, moving beyond the dominant "prompt–response" paradigm. We introduce a verifier-anchored taxonomy that deconstructs reasoning data across three cross-cutting axes: (1) verification contracts (formal, environmental, judgment-based), (2) supervision granularity (outcome, step-level, state–action), and (3) cross-generational lineage (synthetic, filtering, contamination). We analyze the quality of reasoning data not as an intrinsic property but as a base-relative and verifier-relative attribute, and explore the core question of attribution : when a model improves, which component of the data interface changed? We conclude by outlining a release lineage framework, synthesizing best practices for documenting the source, construction, and limitations of reasoning data.

1. Introduction

The post-training phase of LLMs has increasingly focused on enhancing reasoning capabilities, with a growing body of work highlighting the critical role of reasoning data—the data used to fine-tune models to reason through complex problems. Despite this, the field lacks a unified framework for systematically understanding, comparing, and attributing the gains and failures of these data. This primer addresses this gap by proposing a structured taxonomy for reasoning data, moving beyond the simplistic "prompt–response" view.

We organize the field around four fundamental questions:

What is reasoning data? (Data objects, components, and verifiability)
What makes reasoning data useful? (Quality metrics, difficulty, and base-relative value)
How is reasoning data built? (Construction pipelines, prompt sourcing, trace generation, and search)
How do we attribute gains? (Scaling laws, efficiency, and attribution)

Our central thesis is that a reasoning data object is not a static artifact but a verifier-bearing feedback interface. A data point is useful not merely because it contains a correct answer, but because it provides a verifiable, auditable, and attributable signal. We argue that the field's focus should shift from simply constructing larger datasets to designing better feedback interfaces.

2. What Is Reasoning Data?

We define a reasoning data object as a verifier-bearing feedback interface, not just a prompt–response pair. This includes:

Correctness (formal, environmental, judgment-based)
Trace quality (trajectory-relative, chain-relative, process-relative)
Difficulty (base-relative, verifier-relative, base-relative)
Coverage (lineage-relative, leakage-relative, contamination-relative)

A reasoning data object is a verifier-bearing feedback interface because it carries within it a verifier (or a verifiable signal) that can be used for training, evaluation, or attribution.

3. What Makes Reasoning Data Good?

Reasoning data quality is a relational attribute. We identify three axes:

3.1 Correctness Is Verifier-Relative

Correctness is a versioned verifier contract, not an answer string. A correct answer is only as good as the verifier that validates it. This is particularly critical in domains with formal verifiers (e.g., math, code) versus environmental verifiers (e.g., tool use, web, code execution).

3.2 Good Traces Are Not Just Long Traces

A visible trace is suggestive evidence, not proof of mechanism. A rationale may be locally valid, but it may be task-relevant or grounded in a verifier. Trace quality is therefore trajectory-relative and process-relative.

3.3 Difficulty Is Base-Relative

Difficulty is an item–model–sampling relation. A hard item is not necessarily a useful item, because it may be unreachable for one base, gradient-producing for another, and saturated for a third.

3.4 Coverage, Leakage, and Lineage

Coverage is a recipe, not a count. A useful release should therefore report the generator, filter, split, teacher, verifier of record, decontamination status, and known inheritance risks.

4. How Is Post-Training Reasoning Data Built?

Public reports have made the RL scaffold increasingly visible: long-CoT SFT, distillation, RLVR, reward modelling, and GRPO-style optimization recur across DeepSeek-R1, Kimi K1.5, Qwen3, Magistral, Phi-4-reasoning, and Llama-Nemotron. But optimizer visibility is not causal isolation. DAPO changes filtering and loss geometry, MiniMax-M1 changes importance-weight control, and Qwen3-Coder-style systems move signal into long-horizon tool interaction.

4.1 Where Do Reasoning Prompts Come From?

Prompt sourcing is where a recipe first becomes data-specific. OpenThoughts makes source, mixture, filtering, answer generation, and teacher choice explicit ablation variables; math and code corpora turn olympiad pools, web-mined problems, executable tests, and model-conditioned solve rates into prompt-side fields. Small-set results such as LIMO and s1 provide the counterpoint: few demonstrations can elicit strong behaviour only when the skill already lies within the base policy’s support. Prompt releases should therefore report source, filter, verifier, per-base pass rate, and estimate date, not only domain and count.

4.2 Who Writes the Trace?

Trace construction fixes the author, process field, and grounding of the trajectory. Teacher traces can transfer decomposition style, uncertainty expression, formatting, tool use, and stopping behaviour, not merely answers. Process-supervised variants make hidden structure more explicit: PRM800K labels steps, Math-Shepherd estimates rollout value, OmegaPRM searches for first errors, PRIME derives implicit process reward, and PROF filters incoherent chains. The dispute is therefore not whether traces matter, but whether outcome-only RL, explicit PRMs, or trainable verifiers expose the relevant process field.

4.3 Where Do Search-Time Gains Live?

Agentic reasoning moves search from decoding into an executable substrate. Repository, web, app, OS, and tool benchmarks define samples through states, actions, observations, files, tests, and terminal predicates. The lesson is not that every task needs an environment, but that environment data should be replayable. Successful transcripts alone erase failed actions, recoveries, retries, state diffs, and hidden predicates—the branches where credit assignment is visible.

4.4 Can Self-Play Eliminate Curation?

Self-play does not eliminate curation; it moves curation into the anchor that turns generated behaviour into trainable feedback. STaR anchors rollouts to external answers, R-Zero separates Challenger and Solver roles, Absolute Zero uses the Python interpreter, TTRL turns test-time majority into reward, and multi-agent variants distribute these functions across roles. These systems differ less in agent count than in what makes a trajectory admissible: answer availability, executable feedback, majority selection, role-mediated challenge, or archive-based evaluation.

4.5 What Does the Policy Train Against?

The reward channel is also a data object: a “reward” may be a formal checker, process verifier, learned reward model, rubric judge, or closed-loop selection rule. Master-key attacks, spurious rewards, GSM-Symbolic perturbations, and verifier-gaming tests all show that reward signals can be broad yet brittle. Rubric systems extend the same problem to domains without symbolic checking: HealthBench, RaR, OnlineRubrics, Omni-RM, AutoRubric, RubricArm, and PoP make criteria, elicitation, modality, and judge-policy interaction part of the reward-data pipeline.

4.6 What Does a Frontier Pipeline Look Like?

A frontier pipeline is an orchestration of the fields above. Distill-then-RL systems buy cold-start speed through teacher traces; small-warmup multistage RL systems let RL reshape the policy; pure-RL lines reduce teacher inheritance but increase dependence on verifier and prompt support. The optimizer is therefore the wrong final unit of comparison: a reported gain is interpretable only after the prompt support, trace teacher, substrate, anchor, verifier, scaffold, and inference budget are declared.

5. Scaling: Asymptotes on the Data Substrate, Efficiency on the Optimiser

Recent scaling sweeps make reasoning post-training look less like a single law than a ledger of what changed. We use their equations as a reading device rather than a universal law:

R(C) = R_0 + \frac{A - R_0}{1 + (C_{\mathrm{mid}} / C)^B}, \quad \log L(N,C) = E(N) - k(N) \log C

Here A is the reachable ceiling, while B or $k(N)$ captures approach efficiency. A benchmark gain is therefore not self-explanatory: it may move the ceiling, improve approach speed, or change the measurement surface.

5.1 Asymptotes and Efficiency

The useful commonality between the Khatri and Tan laws is the separation: some choices change the reachable frontier, while others change the path toward it. Data substrate, verifier quality, support coverage, context, architecture, and search topology can move the ceiling; loss design, sampling, rollout budget, curriculum, precision, and warm-start distillation more directly alter efficiency.

This distinction also organizes the RLVR debate. Some read current RLVR as sharpening trajectories already accessible to the base policy. The counter-literature relaxes different closures: ProRL extends horizon length; RL-PLUS injects external rollouts; CoT-Pass@K changes the success metric; and PASS@(k, T) adds interaction depth. Thus “RL expands capability” is not a scalar claim; it specifies which closure was broken.

5.2 Data Uniqueness and Stage Composition

Tan et al.’s decomposition $D_{\mathrm{total}} = D_{\mathrm{unique}} \times \tau$ makes reuse part of the scaling surface rather than a preprocessing detail. Small-pool releases such as LIMO and s1 show hundreds or thousands of examples can elicit strong behaviour when the base already supports the skill. Cross-base studies sharpen the point: cognitive behaviours and even spurious rewards are base-conditional rather than generic. Distillation and stage composition decide what enters RL before scaling is measured, since teacher traces, grounded thoughts, mid-training, SFT, and RL prefer different quality–diversity trade-offs. Scaling is therefore parameter count under a base prior, uniqueness budget, teacher lineage, and stage order.

5.3 Inference-Time Compute and Trace Topology

Inference-time scaling changes the observable, not merely the decoding setting. Repeated sampling can scale coverage when candidates are verifiable; s1 makes budget forcing an explicit intervention on thinking length; and parallel-distill-refine, TTRL, and MiGrATe make the deployment-time selector, verifier, or mixed policy part of the reported data object. Long-context reasoning adds another topology: more context can raise the ceiling while reducing efficiency, whereas Markovian Thinking changes the environment so reasoning proceeds through fixed-size chunks and learned state. A pass@N score under serial extension, repeated sampling, parallel refinement, test-time training, or Markovian chunking is therefore not the same observable.

5.4 Verifier Scaling and Release Lineage

The verifier scales with the policy, but not necessarily in the right direction. Master-key attacks, GSM-Symbolic perturbations, and verifier-robustness studies show that reward signals can be broad yet brittle. Process-verifier corpora follow the same logic: PRM800K, Math-Shepherd, OmegaPRM, and Qwen2.5-Math-PRM differ in whether supervision comes from human labels, Monte-Carlo scores, first-error localization, or judge consensus. CoVerRL and DeepSeekMath-V2 suggest co-evolving generators and verifiers, but co-evolution increases the need for versioning rather than removing it.

6. Conclusion

Post-training reasoning data have become the hidden substrate of reasoning-model progress, and the stream of releases is accelerating. The evidence reviewed here suggests that the reusable unit is not a prompt–response pair, but a verifier-bearing feedback interface whose value depends on the verifier, base model, lineage, optimizer, scaffold, and inference budget. The central unresolved question is therefore attribution : when a model improves, which part of that interface changed? This primer helps the community move from reporting gains to making them inspectable, comparable, and testable.

Limitations

This primer is limited by the public evidence available for post-training reasoning data. Closed pipelines, proprietary data mixtures, and undocumented release practices are necessarily absent, and many public reports omit lineage cards, verifier versions, compute and inference budgets, and contamination audits. Our synthesis is also question-driven rather than a formal meta-analysis: we include work that exposes reasoning-data objects, feedback interfaces, construction recipes, or scaling surfaces, which may underrepresent work that reports only aggregate benchmark gains or uses non-public artifacts. In addition, the evidence base is heterogeneous, spanning peer-reviewed papers, arXiv preprints, technical reports, benchmarks, and model cards; we use these sources to document public recipes and release practices, not to establish causal rankings. Finally, we do not independently re-run training recipes, audit contamination, or validate every verifier. The taxonomy should therefore be read as an attribution and reporting framework whose fields will need updating as reasoning post-training expands to new multimodal, multilingual, agentic, and co-evolving verifier-generator settings.

Acknowledgments

We thank the authors of the works discussed in this primer for their contributions to the field, and we refer readers to the original papers for full details. We also thank the authors of the DeepSeek-R1, Kimi K1.5, Qwen3, Magistral, Phi-4-reasoning, and Llama-Nemotron teams for their public releases and technical reports, which have made this synthesis possible. We also thank the broader community for the open-source releases of data, models, and benchmarks that have shaped the field.