Shanghai AI Lab and SJTU open-source 8.9B latent-space model NCP-ArchPreview, matching OLMo-3-7B loss with 51.3% tokens
Shanghai AI Laboratory and Shanghai Jiao Tong University's LUMIA Lab released a technical report and open-sourced NCP-ArchPreview, an 8.9B-parameter discrete latent-space model introducing next-concept prediction (NCP) as a pretraining task. On the 5.73T-token Dolmo-3 corpus, it matched OLMo-3-7B's final pretraining loss using only 51.3% of the token budget, a 1.95x effective convergence speedup, with GSM8K reasoning improving by nearly 6 points. The release includes all training checkpoints and assets.
Coverage timeline
机器之心机器之心
自现代大模型诞生以来，“下一个Token预测（Next-Token Prediction, NTP）”就被奉为不可撼动的黄金律条。然而，逼着模型逐字死记硬背，真能学会宏观语义规划吗？ 近日，arXiv 上悄然挂出了一篇 来自上海人工智能实验室与上海交通大学人工智能学院 LUMIA Lab 团队联 合撰写的技术报告——《NCP-ArchPreview Tech Report》。没有什么铺天盖地的宣发，这篇论文却在短短几天内自发引爆了开源与极客社区： 一举登顶 HuggingFace Daily Papers 榜首 ，HuggingPapers 官方及海外多位知名科技博主纷纷转发剖析；海外科技评论人 VISION IA 更是在 X 上给出了极高评价：“这篇论文很可能构成了未来 AI 的第一块基石（might constitute the first brick of the future of AI...）”。 让全球研究者如此兴奋的原因极其纯粹：他们打破了纯 Next-Token Prediction（NTP）的单一教条，在大规模预训练中直接引入“ 下一个概念预测（Next Concept Prediction, NCP） ”这一全新的预训练任务，跑通了全球首个 8.9B 规模的离散隐空间基座模型，在Dolmo-3的 5.73T 大规模开源语料预训练中，仅消耗 51.3% 的 Token 预算就追平了 OLMo-3-7B 的最终预训练 Loss，等效收敛速度狂飙 1.95 倍！ 与此同时，GSM8K 推理暴涨近 6 个点，更附带了“17M 参数零遗忘微调”、“推测解码免费加速”等一系列意想不到的外溢红利。目前，团队已将包含数十个训练阶段 Checkpoint 在内的全部资产彻底开源。 在视觉生成领域，从像素空间走向隐空间（Latent Space）的 Latent Diffusion 彻底改写了文生图的效率格局；在视频与具身智能领域，LeCun 力推的 JEPA 架构也在证明预测高维表征远优于重建底层细节。但在大语言模型（LLM）的世界里，大家似乎习惯了“逐词预测”的自回归范式。即使是当下大火的多Token预测（Multi-Token Prediction, MTP），其损失函数依然被牢牢绑定在浅层表面的 Token 上。语言模型内部明明已经自发形成了丰富的语义抽象与
Related stories
Tencent Hunyuan analyzes batch size impact on scaling RL training at 10K-100K GPU clusters
Tencent Hunyuan's team published a technical analysis on scaling reinforcement learning (RL) training for clusters of 10,000 to 100,000 GPUs, focusing on batch size tuning and scaling. They found that moderately increasing batch size with adjusted learning rates improves throughput, but at 16B batch size the additional sample cost outweighs the gains, slowing time-to-target.
2 reports · 2 sources
LAMA Framework Auctions Ads Token-by-Token During LLM Generation
Researchers from Renmin University and Stanford propose LAMA, a large-model-native advertising framework that auctions ads per generated token, integrating ad allocation into the generation process. This shifts from traditional pre-defined ad slots to dynamic allocation where each token influences both content continuation and ad exposure.
1 reports · 1 sources
Z.ai confirms Ox Alpha is GLM-5.3 Flash, releases open weights
Z.ai confirmed that the anonymous model 'Ox Alpha', which went viral on OpenRouter, is a new iteration of its GLM series, specifically GLM-5.3 Flash, and released its weights on Hugging Face under a new license. The model, a 320B-parameter natively multimodal model running on domestic chips, topped OpenRouter's leaderboard and ended DeepSeek's 56-day streak on OpenCode. The license requires companies with over $10B in revenue to pass a security review.
9 reports · 4 sources801 points · 281 comments
Stanford's Marin Open Labs Live-Streams Training of 535B-Parameter Model
Percy Liang's Marin Open Labs has begun publicly live-streaming the training of the Marin 535B-A23B model, a 535B-total-parameter (23B active) model trained on 18.75T tokens over approximately three months, totaling about 2.7e24 FLOPs. The team previously ran a four-stage scaling ladder from 1.6B to 27.7B parameters to debug and forecast the hero run, and plans to open data, logs, and engineering details.
News updates every hour. Subscribe in your RSS reader to get new stories automatically.
Subscribe