NICT's OptiMer merges distribution vectors to optimize continual pretraining data ratios, wins EMNLP 2026 Outstanding Paper
Researchers at Japan's NICT proposed OptiMer, a method that replaces data mixing with merging distribution vectors for continual pre-training, allowing different data ratios to be tried without retraining. The paper won an Outstanding Paper Award at EMNLP 2026, one of 22 among 17,661 submissions, and achieved 15-35x speedup in ratio optimization.
Coverage timeline
机器之心机器之心
不同数据，到底各训多少？每个模型训练师都绕不开这个问题。 训练集中不同数据配比出错，往往只能从头训。 每个数据集各放多少入训练集，训练前就得定好；但是配比好不好，却要等训练结束评测后才知道 。效果若不理想，可能要调整比例重新训练，试错成本极高。 日本信息通信研究机构（NICT）的宋海越及团队 针对这一问题在论文「OptiMer: Optimal Distribution Vector Merging Is Better than Data Mixing for Continual Pre-Training」中提出了合并分布向量的解决方案，让持续预训练中尝试不同配比不必从头训模型。 此论文荣获 EMNLP 2026 杰出论文奖（Outstanding Paper Award） ，是本届 17,661 篇投稿中获此殊荣的 22 篇论文之一。 论文：OptiMer: Optimal Distribution Vector Merging Is Better than Data Mixing for Continual Pre-Training 作者：Haiyue Song, Huy Hien Vu, Masao Utiyama 网站：https://shyyhs.github.io/optimer 方法 | 从合并数据集到合并分布向量 我们提出最优合并，OptiMer。 第一步，分别训练模型，学习各类数据。 从基座模型出发，在各个数据集上分别进行持续预训练。例如，在日语，中文和数学数据上，分别各训练一个模型，得到三个模型。 第二步，提取分布向量，记录训练带来的能力变化 。将每个训练后模型的参数减去基础模型的参数，可得到对应的分布向量。它代表了此数据上训练后模型发生的变化。 第三步，自动搜索，寻找最优合并比例。 基础模型权重，加上不同比例的多个分布向量，外加指令遵循向量，可以得到一个新模型。根据其在验证集上的表现，我们可以不断优化分布向量的比例，达到最优合并。每次尝试只包含模型合并和评估，无需再训练。 效率 | 提升 15～35 倍 完成各个数据集的初始训练后尝试不同配比，传统数据混合方法仅训练一个配比就需要 129.3 小时，而 OptiMer 尝试 100 次配比仅需 8.5 小时。实验中我们发现在配比最优化阶段，OptiMer 实现了 15～35 倍加速 。 效果 |
Related stories
NV-CoT: Multimodal LLMs Predict Continuous Coordinates for Visual Chain-of-Thought, NeurIPS 2026 Oral
Researchers from NTU, USTC, NUS, and Microsoft propose NV-CoT (Numerical Visual Chain-of-Thought), a method that lets multimodal LLMs directly predict continuous numerical coordinates for region selection instead of generating bounding boxes as text tokens. The work, accepted as a NeurIPS 2026 Oral (top 0.4%), supports supervised fine-tuning and reinforcement learning, addressing mismatches between token-level training and spatial error.
1 reports · 1 sources
Realtime-Bench: Frontier LLMs achieve only 44% success on real-time interactive tasks
Researchers from NVIDIA, UT-Austin, and other institutions released Realtime-Bench, a benchmark of 50 interactive game and web tasks that require real-time decision-making. Testing 12 frontier models, the best performer (GPT-6-Astra) achieved only 44% success, highlighting LLMs' weakness in low-latency tasks.
Recursive releases Flint, 40B coding model built largely by its Eureka Machine AI system
Startup Recursive has released Flint, an open-weight, 40B-parameter mixture-of-experts coding model with 2.5B active parameters per token, built in under four months with heavy automation from its Eureka Machine AI research system. The model outperforms sub-50B MoE models on coding benchmarks like SWE-Bench Pro v2 and runs at over 125 tokens per second on a MacBook Pro M4 Pro, with weights available on Hugging Face under Apache 2.0.
Fudan, Tencent Hunyuan, Tsinghua Release ExplorationBench to Test AI Rule Discovery
Fudan University, Tencent Hunyuan, and Tsinghua University released ExplorationBench, a benchmark measuring how AI systems discover rules in unfamiliar environments. It includes two altered 'alien worlds'—the programming language AlienCode and the logic proof system AlienLogic—where models must learn rules through trial and error. Results showed that without environment feedback, scores remained low (max 11.0%), while autonomous exploration led to high scores (up to 89.0%).
News updates every hour. Subscribe in your RSS reader to get new stories automatically.
Subscribe