Analysis: 2019-2025 Pretraining Compute Efficiency Gains Largely from Data, Not Models
#pretraining#compute-efficiency#data-quality#ai-research
An analysis by Dwarkesh Podcast decomposing six years of pretraining progress found that compute efficiency gains from 2019 to 2025 were driven mostly by data improvements rather than model improvements. The breakdown quantifies how much of the rapid progress in AI stemmed from data-side changes versus architectural or algorithmic changes.
Coverage timeline
Techmeme
Dwarkesh Podcast : Analysis: from 2019 to 2025, gains in pretraining compute efficiency came mostly from data improvements rather than model improvements — Breaking down 6 years of pretraining progress into data vs model improvements — How much of the rapid progress in AI that we've seen over the last few years 1 …
