Multimodal Flow: Fully Continuous Architecture Unifies Language and Vision
Researchers from HUST and Horizon Robotics propose Multimodal Flow, a fully continuous multimodal architecture that models both language and vision in embedding spaces using a unified Flow Matching objective, avoiding tokenization and hybrid systems. In controlled comparisons with equal data and parameters, it outperforms hybrid and fully discrete approaches, and the 1.6B-parameter MF-1 model shows strong multimodal generalization and efficient transfer to downstream tasks.
Coverage timeline
机器之心机器之心
走向物理 AI 时代，构建一个能真正统一语言与视觉的多模态基座模型是关键一步。然而，语言本质上是有序的 Token 序列，视觉则是连续的高维像素分布，两者在信息结构上存在天然的鸿沟。如何在一个模型内统一这两种截然不同的分布，一直是一个备受争论的话题。 目前，主流的解法主要有两条。一条是 全离散路线 （如 Chameleon、Emu3），它把图像也量化成离散 Token，与语言拼在一起做自回归建模。但这种方法受限于视觉 Tokenizer，图像的高频细节在量化过程中极易丢失。另一条是 混合离散 - 连续路线 （如 Transfusion、JanusFlow），文本走离散预测，图像走连续扩散。虽然避开了量化瓶颈，但一个模型里得同时维护两套截然不同的目标函数和采样流程，不仅工程实现复杂，联合优化的难度也很大。 既然两条路都有妥协，有没有第三种解法？ 近日，华中科技大学王兴刚团队与地平线机器人等机构合作，提出了 Multimodal Flow ，给出了第三条路线： 全连续 。在这个架构下，语言和视觉各自保留在原有的连续嵌入空间中，但共享同一个 Flow Matching 目标。这意味着，模型无需强行将视觉翻译成语言，也不用维护割裂的两套系统，仅用一个 Flow 模型就能统一建模两者的联合分布。实验表明，Multimodal Flow 不仅展现出了强大的多模态泛化能力，混合预训练阶段学到的知识也能高效迁移至下游任务；在同等数据与参数预算的受控对比中，它的表现更是全面超越了混合式和全离散架构。 论文标题：MULTIMODAL FLOW: UNIFIED FLOW MODELING OF LANGUAGE AND VISION IN EMBEDDING SPACES 论文地址：https://arxiv.org/pdf/2609.40362 开源代码：https://github.com/hustvl/Multimodal-Flow 项目主页：https://hustvl.github.io/Multimodal-Flow 1.6B 模型，展现了什么信息 在谈理念之前，我们先看干货数据。团队实例化了主模型 MF-1 ，仅有 1.6B 参数，并且是从零开始预训练，仅消耗了 150B Tokens。相比当前动辄上万亿 Token 的大模型，这个数据量相当克制，但成绩却十分亮眼。 在文
Related stories
RAVEL: Training-Free Method Improves Vision Encoder Evaluation for Multimodal LLMs
Researchers proposed RAVEL, a training-free evaluation method with two improvements—stabilized PCA whitening and patch-level similarity scoring—to predict how well vision encoders perform in multimodal LLMs. Experiments on 210 multimodal LLMs showed existing cross-modal methods degrade as candidate pools grow, while RAVEL achieved an average Spearman correlation of 0.85, outperforming prior methods by up to 101.1%.
1 reports · 1 sources
Fudan team compares hybrid LLM architectures mixing full attention with sliding-window and gated linear attention
Fudan University's OpenMOSS team released a paper, 'Mechanics of Long-Context Hybrid Models', systematically comparing hybrid architectures that mix full attention with sliding-window attention (SWA) and gated linear attention (GLA/GDN) for long-context LLMs. Experiments at 376M, 776M, 1B, and 3B scales show that mixing NoPE attention with position-biased attention (RoPE, SWA, GLA/GDN) enhances length extrapolation and long-context training. The team found that hybrid placement improves long-context performance.
LoopVL open-source VLM shows data efficiency with 260x less training data
Researchers from Renmin University, Baidu, and others released LoopVL, an open-source vision-language model based on the Loop Transformer. Despite using only 0.14T training tokens versus about 36T for Qwen3.5-2B and InternVL3.5-2B, LoopVL matches or exceeds them on benchmarks like LogicVista, MMMU-Pro, and MathVision, demonstrating high data efficiency. The team also observed cross-loop visual attention reorganization, termed 'Visual Aha Moment'.
Kaiming He's Team Unveils VISTA, a Visual Harness for Multimodal Reasoning
Kaiming He's team released a paper introducing VISTA, a visual-native harness framework that lets multimodal models observe environments, store raw frames, and revisit them during reasoning without retraining. In evaluations, VISTA enabled Claude Opus 5.0 to complete all 25 public ARC-AGI-3 games with a perfect human-relative efficiency score of 100, using 57.4% fewer actions than first-time human baselines; GPT-5.6 Sol also cleared all games with a score of 99.
1 reports · 1 sources
News updates every hour. Subscribe in your RSS reader to get new stories automatically.
Subscribe