Fudan team compares hybrid LLM architectures mixing full attention with sliding-window and gated linear attention
Fudan University's OpenMOSS team released a paper, 'Mechanics of Long-Context Hybrid Models', systematically comparing hybrid architectures that mix full attention with sliding-window attention (SWA) and gated linear attention (GLA/GDN) for long-context LLMs. Experiments at 376M, 776M, 1B, and 3B scales show that mixing NoPE attention with position-biased attention (RoPE, SWA, GLA/GDN) enhances length extrapolation and long-context training. The team found that hybrid placement improves long-context performance.
Coverage timeline
机器之心机器之心
本文第一作者柳潇然，复旦大学、上海创智学院 2022 级硕博连读生，导师邱锡鹏、何子薇。研究方向为长上下文大语言模型，曾在 ICLR、ICML、AAAI、EMNLP 等国际顶级人工智能会议发表论文十余篇。 背景介绍 长上下文已经成为大语言模型重要的基础能力，但长上下文也给模型架构带来了一个现实问题：全注意力能够直接建模任意位置之间的关系，却带来随序列长度显著增长的计算和显存开销；滑窗注意力和线性注意力更加高效，但效果上都存在明显的不足。因此混合架构逐渐成为当前大语言模型设计的主流。 面对滑窗注意力和线性注意力两类固定计算与存储复杂度的高效方案，混合架构应该采取全注意力与哪种的混合呢？复旦创智邱锡鹏团队的最新工作 混合架构的长文力学 Mechanics of Long-Context Hybrid Models 给出了一个有趣的答案： 我全都要 。 论文从混合位置的角度，系统比较了基于 RoPE/NoPE 全注意力、滑窗注意力（SWA），以及 GLA/GDN 等门控线性注意力的混合架构在长上下文任务中的表现，并进一步分析了不同注意力的协同机制。 Paper：https://arxiv.org/abs/2610.10114 GitHub：https://github.com/OpenMOSS/Hybrid-Mechanics 知乎：https://zhuanlan.zhihu.com/p/2092654065377457038 观察 研究团队在 376M、776M、1B 和 3B 四种规模上开展实验。模型先进行 4k 上下文的短文预训练，再进行 32k 上下文的长文续训；评测覆盖 PG19 PPL、LongPPL、RULER、BABILong 和 LongBench，包括 短文训练下的长度外推表现和长文续训后的扩长训练表现 ，同时比较层混合（LH）与头混合（HH）两种方式。 论文首先给出的观察是： 混合位置让混合架构具有更好的长文表现 。RoPE、滑窗和门控线性注意力都带有显式或隐式的位置偏置，而 NoPE 注意力则不引入位置偏置。NoPE 与上述带位置偏置的注意力的混合，不仅可以增强长文训练的扩长表现，同时也可以实现即插即用的长度外推。 SWA/GLA/GDN-NoPE-LH 的长度外推表现 SWA/GLA/GDN-NoPE-LH 的长文续训表现 不过，不同混合架构仍
Related stories
STILL: Content-Aware Intra-Layer Hybrid Attention Cuts Memory 80%, Speeds Prefill 83%
Researchers from Harbin Institute of Technology (Shenzhen) and collaborators present STILL, a NeurIPS 2026-accepted architecture that routes tokens to Softmax or linear attention within the same layer based on content importance via a self-saliency score. It achieves 80% memory reduction and 83% prefilling speedup at 16K context, with only 0.04B training tokens for linearization, and up to 86.2% improvement over prior linearization methods on long-context benchmarks.
1 reports · 1 sources
Fudan, Tencent Hunyuan, Tsinghua Release ExplorationBench to Test AI Rule Discovery
Fudan University, Tencent Hunyuan, and Tsinghua University released ExplorationBench, a benchmark measuring how AI systems discover rules in unfamiliar environments. It includes two altered 'alien worlds'—the programming language AlienCode and the logic proof system AlienLogic—where models must learn rules through trial and error. Results showed that without environment feedback, scores remained low (max 11.0%), while autonomous exploration led to high scores (up to 89.0%).
Meta releases Muse Spark Open, an 816B-parameter open-weight model optimized for Apple silicon
Meta released Muse Spark Open, an open-weight multimodal mixture-of-experts model with 816B total and 42B active parameters and a 1M-token context window, claiming it is the most capable US open-weight model. Optimized for Apple silicon via MLX, it can run locally on two M5 Ultra Mac Studios, with day-0 support from vLLM and other inference engines.
SimpleVLN: Native MLLM Method Achieves SOTA in Vision-Language Navigation
Researchers from Fudan University and Nanyang Technological University propose SimpleVLN, a method that achieves state-of-the-art results in vision-language navigation without adding specialized modules, by leveraging the native capabilities of multimodal large language models (MLLMs). The approach treats each model call as an independent decision, using a global-plus-local strategy for historical observations, and demonstrates through ablation studies that uniform sampling of global history outperforms feeding all historical images.
News updates every hour. Subscribe in your RSS reader to get new stories automatically.
Subscribe