Baidu open-sources ERNIE 5.5 Flash Base, a 280B-parameter MoE model
On November 5, Baidu open-sourced ERNIE 5.5 Flash Base, a Mixture-of-Experts base model with 280 billion total parameters and 14 billion active parameters per token, pretrained on 33.5 trillion tokens with a maximum pretraining context length of 128K and up to 1 million tokens during fine-tuning and inference. The model reportedly achieves state-of-the-art results among same-size models on benchmarks covering knowledge, reasoning, math, code, multilingual, and long-context tasks, surpassing DeepSeek-V4-Flash-Base, Hy3-preview-Base, and MiMo-V2.5-Base. It introduces Latent-Gated Sparse Attention, Multi-Router Latent MoE, and the ECHO training objective to improve efficiency.
Coverage timeline
机器之心新闻资讯
11月5日，百度正式开源最新基座模型文心5.5 Flash Base（ERNIE 5.5 Flash Base）。模型采用MoE架构，总参数量2800亿，每个 Token 激活140亿参数，使用33.5万亿个 Token 完成预训练，预训练上下文长度最高达128K，微调与推理阶段支持最长100万 Token上下文。在知识、推理、数学、代码、多语言和长上下文等多个基准测试中取得同尺寸模型 SOTA 表现，超过 DeepSeek-V4-Flash-Base、Hy3-preview-Base、MiMo-V2.5-Base 等模型。 开源地址： https://huggingface.co/baidu/ERNIE-5.5-Flash-Base 随着大模型应用从文本问答向长文档分析、多轮交互和智能体任务延伸，模型需要处理的信息量持续增加，计算和显存开销也随之上升。在保证模型能力的同时提高资源利用效率，已成为大模型研发的重要课题。 文心 5.5 Flash Base围绕“效率”这一设计主线，在注意力机制、专家路由及训练机制等关键环节进行系统性优化，并实现了多项算法创新。 在注意力机制上，文心提出 Latent-Gated Sparse Attention ，将混合稀疏注意力、低秩门控注意力与虚拟头注意力结合起来。其中，模型通过组合HCA与MLA，让不同注意力层分别承担压缩历史信息和细粒度访问历史位置的任务；低秩门控注意力（Low-Rank Gated Attention）则利用已有的低维表示计算信息门控，使门控投影权重参数减少75%；此外，虚拟头注意力（Virtual Head Attention，VHA）可提升模型表达力与训练质量，推理时融合进注意力算子，开销极小。这些设计降低了长上下文处理的资源开销，同时保留模型对关键信息的处理能力。 在专家路由上，模型采用 Multi-Router Latent MoE ，由多个独立路由器共同参与专家选择，改善路由评分与专家实际效用之间的相关性。相比依赖单一路由器的传统方案，该设计旨在降低单一评分偏差对专家选择的影响，同时保持激活专家数量、专家计算量和通信量不变。在训练目标上，模型引入 ECHO ，无需额外增加 MTP 模块（Multi-Token Prediction），而是复用主干最后一层，让主干自身学习预测后续 token，从而提前
Related stories
Meta releases Muse Spark Open, an 816B-parameter open-weight model optimized for Apple silicon
Meta released Muse Spark Open, an open-weight multimodal mixture-of-experts model with 816B total and 42B active parameters and a 1M-token context window, claiming it is the most capable US open-weight model. Optimized for Apple silicon via MLX, it can run locally on two M5 Ultra Mac Studios, with day-0 support from vLLM and other inference engines.
6 reports · 6 sources
Ant Group open-sources Ling-3.1-flash, a 560B-parameter MoE model with 1M context
Ant Group's Ling-3.1-flash, a 560B-parameter mixture-of-experts model with 25B active parameters and a 1M-token context window, was released open-source on October 15. The model targets long-horizon complex task execution, using a hybrid linear attention design with KDA layers and Gated MLA to manage computational costs on long sequences.
DeepSeek releases V4.1-Flash-1017 open-weight model with faster, cheaper agent tasks
DeepSeek released V4.1-Flash-1017, an open-weight model checkpoint with 763B parameters, now live on its website, app, and API. The update claims stronger benchmark performance, faster task completion, and shorter outputs than its predecessor, achieved via reinforcement learning with self-feedback for end-to-end project development. DeepSeek also opened its Search & Fetch APIs to everyone, priced at $0.75 per 1000 successful requests, and requires DeepSeek Harness v0.2.1-rc.1 or later for optimal use.
9 reports · 6 sources955 points · 717 comments
Microsoft unveils MAI-Base-2 MoE model with 2.8T total parameters
Microsoft AI announced MAI-Base-2, a mixture-of-experts model with 2.8 trillion total parameters and 80 billion active parameters. The company claims it achieves 4x-12x greater pretraining FLOPs-efficiency compared to DeepSeek-V4-Pro-Base. This follows the June announcement of MAI-Thinking-1, a reasoning model post-trained on MAI-Base-1.
News updates every hour. Subscribe in your RSS reader to get new stories automatically.
Subscribe