Schmidhuber-led team publishes 84-page survey on next-gen LLM architectures
A team led by Jürgen Schmidhuber, with researchers from KAUST, HKU, Huawei, and other institutions, published a comprehensive 84-page technical survey on large language model architectures, covering nearly 600 references from 1913 to 2026. The survey traces the evolution from Markov chains to Transformer and examines emerging directions such as linear attention, mixture-of-experts, multi-token prediction, and test-time training.
Coverage timeline
机器之心机器之心
Scaling up 是大模型的标志，也引领了近几年语言模型技术的飞跃。然而实际上，大语言模型架构，作为其最本质的计算基础，也在近年来持续发展。GQA、MLA、各式稀疏注意力模块的提出，或是 Mamba、Gated DeltaNet 之类的线性注意力网络，DeepSeek 的细粒度 MoE，还有多 token 预测和 Looped 架构等等，几乎每一版旗舰模型，都在先前架构的基础上提出了新的架构改进。持续学习、测试时训练等新需求也在推动架构继续演变，像 Ilya 创立的 SSI 正在研发能在推理时更新权重的模型。当前的语言模型架构不是终点，这几乎已是领域共识。 最近，「LSTM 之父」 Jürgen Schmidhuber 领衔，来自 香港大学、KAUST、华为、香港中文大学、北京大学 等机构的研究者合作发布了面向大语言模型架构研究领域的深度技术综述。正文共 84 页，引用文献近 600 篇，时间跨越 1913 到 2026，追问下一代语言模型架构的发展方向。 论文标题：Towards Next-Generation Architectures of Large Language Models: A Survey 论文链接：https://www.preprints.org/manuscript/202610.1532 项目主页：https://surveyllmarch.github.io/ GitHub：https://github.com/SurveyLLMArch/LLMArchitectureSurvey 从 1913 年讲起 语言模型的演变经过了近百年的历史。1913 年，马尔可夫统计了普希金《叶甫盖尼・奥涅金》里元音和辅音的排列，第一次用数据说明语言里存在条件依赖。1948 年，香农把语言看作随机信源。后来，语音识别里的 n-gram 模型、循环网络（1990 年）、LSTM 模型（1991 年）被依次提出。到如今，Transformer 架构支撑起了当前整个大模型领域。 更长的时间跨度让我们可以看到架构技术的交叉演变。比如 1991 年 Schmidhuber 提出的 Fast Weight Programmers 里，有一种加性外积更新规则，其今天可以被称为 “无归一化线性 Transformer”。而 2020 年的核化线性注意力，数学
Related stories
LoopVL open-source VLM shows data efficiency with 260x less training data
Researchers from Renmin University, Baidu, and others released LoopVL, an open-source vision-language model based on the Loop Transformer. Despite using only 0.14T training tokens versus about 36T for Qwen3.5-2B and InternVL3.5-2B, LoopVL matches or exceeds them on benchmarks like LogicVista, MMMU-Pro, and MathVision, demonstrating high data efficiency. The team also observed cross-loop visual attention reorganization, termed 'Visual Aha Moment'.
1 reports · 1 sources
PKU Researchers Propose MapFlow to Reuse Attention Maps Across LLMs, Cutting Prefill Latency 15-20%
Researchers at Peking University's PKU-DAIR lab propose MapFlow, a method that reuses attention maps as intermediate states across different LLMs to avoid recomputing context when switching models. In evaluations, MapFlow reduces prefill latency by 15-20% and boosts throughput by up to 25% under a P90 first-token latency constraint in multi-turn routing scenarios, while maintaining generation quality. The paper has been accepted at SIGMOD 2027 and the code is open-sourced.
LeCun paper argues next-token prediction inefficient, proposes JEPA world models
A new paper by Yann LeCun and colleagues mathematically argues that next-token prediction is inefficient and proposes an alternative based on world models and JEPA (Joint Embedding Predictive Architecture). The paper claims a 109M-parameter model outperforms a 72B LLM, and formalizes the Platonic Representation Hypothesis into a theorem with actionable sufficient conditions.
Realtime-Bench: Frontier LLMs achieve only 44% success on real-time interactive tasks
Researchers from NVIDIA, UT-Austin, and other institutions released Realtime-Bench, a benchmark of 50 interactive game and web tasks that require real-time decision-making. Testing 12 frontier models, the best performer (GPT-6-Astra) achieved only 44% success, highlighting LLMs' weakness in low-latency tasks.
News updates every hour. Subscribe in your RSS reader to get new stories automatically.
Subscribe