Analysis: 2019-2025 Pretraining Compute Efficiency Gains Largely from Data, Not Models
An analysis by Dwarkesh Podcast decomposing six years of pretraining progress found that compute efficiency gains from 2019 to 2025 were driven mostly by data improvements rather than model improvements. The breakdown quantifies how much of the rapid progress in AI stemmed from data-side changes versus architectural or algorithmic changes.
Coverage timeline
Techmeme
Dwarkesh Podcast : Analysis: from 2019 to 2025, gains in pretraining compute efficiency came mostly from data improvements rather than model improvements — Breaking down 6 years of pretraining progress into data vs model improvements — How much of the rapid progress in AI that we've seen over the last few years 1 …
Related stories
Hacker News Discusses Claim of Over 10x Pretraining Efficiency
A Hacker News thread on September 8, 2026, discusses a claim of more than 10x efficiency improvement in pretraining, likely referencing a specific technical method or result. The discussion focuses on the feasibility and implications of such a gain for AI research and development.
1 reports · 1 sources117 points · 62 comments
Karpathy: No Continual Learning in LLMs, a Decade Away; Multiple Research Paths Emerge
Andrej Karpathy stated in October 2025 that current large language models lack continual learning, estimating it will take about a decade to address this and other cognitive gaps. The field has since seen a surge of research and startups pursuing continual learning, with multiple technical routes including external memory, weight modification, and re-pretraining, all aiming to overcome catastrophic forgetting.
1 reports · 1 sources
Podcast: Redwood Research's Greenblatt debates AI R&D, RSI, data bottlenecks
Dwarkesh Patel's podcast features an interview with Redwood Research Chief Scientist Ryan Greenblatt, covering topics including AI research and development, recursive self-improvement, whether human expert data is a bottleneck, token prices, and alignment. The conversation is framed as a debate about recursive self-improvement.
1 reports · 1 sources
Anthropic reports automated AI researchers mitigate alignment failures across all 10 benchmarks
Anthropic released a report on its Automated Alignment Researcher (AAR) project, where Claude agents autonomously search papers, propose methods, generate data, and train models to fix alignment failures. The system improved performance on all 10 misalignment benchmarks without degrading overall performance, and outperformed human researchers on some tasks at a fraction of the cost.
4 reports · 4 sources
News updates every hour. Subscribe in your RSS reader to get new stories automatically.
Subscribe