Karpathy: No Continual Learning in LLMs, a Decade Away; Multiple Research Paths Emerge
Andrej Karpathy stated in October 2025 that current large language models lack continual learning, estimating it will take about a decade to address this and other cognitive gaps. The field has since seen a surge of research and startups pursuing continual learning, with multiple technical routes including external memory, weight modification, and re-pretraining, all aiming to overcome catastrophic forgetting.
Coverage timeline
机器之心机器之心
编辑｜Panda 最近我们报道了不少剑指「持续学习」方向的创业公司和研究成果，比如《 Mind Lab 连续发布 LoRA 最新进展，大模型「持续学习」新范式浮现 》、《 告别 KV Cache 枷锁，将长上下文压入权重，持续学习大模型有希望了？ 》、《 ICML 2026 | 突破极限！港大提出首个适配 300+任务的持续学习架构，破解遗忘难题 》、《 能让 DeepSeek 自进化的 Harness！LlamaFactory 作者开源新工具：0.2 元自动造 Agent 》、《 当「变大」不再是唯一的路，又一国产模型开源了 》…… 是的，相当密集，「 持续学习 」也正在成为我们最常遭遇的关键词之一。而这背后也昭示着一个被反复念叨的判断。 2025 年 10 月，Andrej Karpathy 在 Dwarkesh Patel 的播客里说：当下的大模型「没有持续学习。你没法告诉它一件事，然后指望它记住」，他认为把这些认知缺陷补齐， 大概还要花上十年 。两个月后，他在年度回顾里把这句话放进了更大的图景：2025 年模型能力的跃迁主要来自可验证奖励强化学习（RLVR），但整个 LLM 技术栈里，记忆、多模态感知、持续学习、操作电脑的能力仍是明显的短板，「我们有原型，但还没有能当同事用的智能体」。 持续学习 （Continual Learning，也叫 终身学习 /Lifelong Learning）由此成了过去一年里最热的概念之一。 它指的是模型在部署之后，还能像人一样从新任务、新知识、新经验里持续吸收，并且不把之前学会的东西忘掉。 这件事听上去理所当然，做起来却极难，难到足以成为通往「AI 同事」路上最硬的一块骨头。而上面那一串报道也说明，这条路已经不是一个方向，而是好几条岔开的路线在同时往前拱。 围绕怎么让模型「边用边学」，学术界和产业界这一年里岔出了好几条互不相同的技术路线。有的往模型外面挂记忆，有的持续改写权重，有的干脆重新预训练，还有一批更新的思路试图重新定义「学习」这件事本身。这篇文章尝试把这些方向逐一摊开，说清楚每条路在赌什么、卡在哪里。 先说清楚：难的不是「学」，是「不忘」 持续学习的核心障碍有一个专门的名字： 灾难性遗忘（catastrophic forgetting） 。神经网络的知识存储在数十亿个权重里，当你用新数据去微调模型、更新这些权重时，模
Related stories
Analysis: 2019-2025 Pretraining Compute Efficiency Gains Largely from Data, Not Models
An analysis by Dwarkesh Podcast decomposing six years of pretraining progress found that compute efficiency gains from 2019 to 2025 were driven mostly by data improvements rather than model improvements. The breakdown quantifies how much of the rapid progress in AI stemmed from data-side changes versus architectural or algorithmic changes.
1 reports · 1 sources
vLLM Inference System Anatomy Detailed in New Technical Series
A new blog post begins a series breaking down the architecture of vLLM, a high-throughput LLM inference system. The first installment covers core components like scheduling, paged attention, and continuous batching, plus advanced features such as chunked prefill and prefix caching. Future posts will explore subsystems, scaling, serving, and benchmarking.
1 reports · 1 sources149 points · 10 comments
Former OpenAI and Google AI leads launch Core Automation, target next-gen architecture beyond Transformer
Jerry Tworek, former OpenAI reasoning lead, and Rohan Anil, former Google Gemini pretraining lead, co-founded Core Automation, arguing that large-scale reinforcement learning has not led to AGI and that current models lack the ability to continue learning after deployment. Separately, the Shanghai AI Laboratory's Mobius team proposed a knowledge-reasoning decoupled architecture as a potential path beyond Transformer's scaling limits.
2 reports · 1 sources
Former OpenAI researcher questions frontier lab business models; Dwarkesh counters
Former OpenAI researcher Andrew Ho published a critique arguing that frontier AI labs may not earn enough to sustain development until AGI, citing OpenAI's 2025 losses of over $20 billion on $13 billion revenue. Podcaster Dwarkesh Patel countered that rapid capability growth itself creates commercial value and that AGI would self-diffuse, with Elon Musk endorsing the optimistic view. The debate highlights the tension between escalating R&D costs and uncertain monetization timelines for leading model developers.
News updates every hour. Subscribe in your RSS reader to get new stories automatically.
Subscribe