Fair Comparison Shows Looped Transformers Beat Standard Ones Under Matched Compute, Parameters, and KV Cache
A study by Tsinghua University and ByteDance Seed, detailed in the paper SMELT, provides the first fair comparison of looped transformers against standard ones by simultaneously matching FLOPs, parameter count, and KV cache across multiple model scales. The results show looped transformers can outperform standard models under these controlled budgets, with optimal looping of the middle 50% of layers. This addresses a long-standing question of whether looped architectures' benefits stem from the architecture itself or simply from increased compute.
Coverage timeline
机器之心机器之心
在大模型的发展中，提升能力的主要手段一直是 "做大"—— 更多参数、更多数据、更多算力。如今，另一条思路开始受到关注：与其不断堆叠新的层，不如让已有的层再跑一遍。这就是 Looped Transformer。普通 Transformer 的每一层只执行一次，而 Looped Transformer 会把其中一部分层重复执行，让模型在不增加参数的情况下获得更深的计算 。 这个想法在推理和数学等任务上已经展现出不小的潜力，但一个关键问题始终没有被正面回答：Looped Transformer 的收益，究竟是循环架构本身的优势，还是因为多算了几遍、实际上用了更多的算力？清华大学、字节跳动 Seed 等机构的研究团队在论文 SMELT 中正面回答了这个问题。他们 同时对齐算力、参数量和 KV cache，首次在多个模型尺度上进行了公平比较 。 论文链接：https://arxiv.org/abs/2609.01343 本文的第一作者为清华大学博士生王少文，师从李建教授，研究方向为大语言模型预训练，曾在 ICML、NeurIPS、ACL、EMNLP 等国际会议上发表多篇论文，并获 LIT@ICLR 2026 最佳论文奖。 预算匹配框架 公平比较需要同时控制三个变量：决定训练和推理成本的每 token FLOPs、约束知识容量的总参数量、以及限制可服务上下文长度的 KV cache。 论文利用 MoE 将参数量和计算量解耦，具体的匹配方式如下：1）缩窄或变浅模型来抵消 Loop 架构循环执行的计算，2）增加专家数来补全总参数量，3）再调整注意力配置（GQA Ratio / Head Dim）来匹配 KV cache。 图 1 (a)｜SMELT 的核心做法：将中间 50% 的层重复执行两次，同时匹配计算量、参数量和 KV cache。 最优循环配方 按上述控制预算的规则，论文在小规模上依次搜索循环范围、执行深宽比和循环次数，得到了以下结论： 1）循环中间 50% 的层收益最大。 论文把循环层比例从 0%（不循环）一路调到 100%（全部循环）。结果显示，比例在 50% 左右时，验证损失最低。作者推测，模型首尾的层分别承担输入编码和输出预测等特殊功能。因此，这些层保留独立参数，可能比参与共享更有利。 表 3｜循环中间约 50% 的层效果最好。 2）Looped 模型的最优执行深宽
Related stories
OpenAI's Jalapeño chip beats Nvidia on inference benchmarks
OpenAI announced first benchmark results for its custom inference ASIC, Jalapeño, developed with Broadcom in about 16 months. Independent testing by SemiAnalysis on the InferenceX benchmark showed the chip outperforming Nvidia, AMD, and Google chips on multiple open-weight models, with 1.5x-1.9x more work per watt and 1.7x-3.6x lower latency than Nvidia GB200/GB300 systems. OpenAI stated it will continue using Nvidia for training.
8 reports · 7 sources583 points · 377 comments
Inclusion AI's 6B LLaDA-Image tops Qwen-Image-Bench, open-sources weights
Inclusion AI released LLaDA-Image, a 6B-parameter diffusion transformer trained from scratch, achieving top scores on Qwen-Image-Bench for both English (53.53) and Chinese (53.38) among open-source models. Over 90% of its 220 million cumulative training samples used image-only supervision, with image-text pairs reserved for later language alignment. The model weights, training code, and recipes are open-sourced, along with a distilled Turbo variant requiring only 2-4 sampling steps.
1 reports · 1 sources
Alibaba open-sources Qwen3.8-27B, a 27B multimodal dense model
Alibaba's Qwen team released the open-weights Qwen3.8-27B, a 27B-parameter native multimodal dense model under the Apache 2.0 license. It supports 262K native context (extendable to 1M tokens) and outperforms Qwen3.7-Plus in coding and office tasks, with independent benchmarks showing it matching GPT-5.6 Luna on the Artificial Analysis Intelligence Index. The model surpassed 1 million downloads within days of release.
11 reports · 4 sources1421 points · 788 comments
Zhipu AI open-sources GLM-5.3-Flash, a 320B multimodal model priced at 1/40 of Claude Opus 4.8
Zhipu AI released and open-sourced GLM-5.3-Flash, the first natively multimodal model in the GLM-5 series, with 320B total parameters (18B active). It scores 57 on the Artificial Analysis Intelligence Index, matching Claude Opus 4.8, and is priced at one-tenth of GLM-5.3, or one-fortieth of Opus 4.8. The model uses a hybrid sparse and linear attention architecture, runs on domestic Chinese chips, and was tested anonymously as Ox-Alpha on OpenRouter and OpenCode.
8 reports · 5 sources1126 points · 573 comments
News updates every hour. Subscribe in your RSS reader to get new stories automatically.
Subscribe