Pitt Researchers Uncover 3D Scene Topology in VLM Latent Spaces
University of Pittsburgh researchers analyzed the latent representations of vision-language models (VLMs) and found that intermediate layers encode spatial structures such as rings, grids, and random layouts, corresponding to 3D scene topology. Their paper, accepted as a NeurIPS 2026 spotlight, introduces methods to separate spatial signals from appearance attributes in high-dimensional activations.
Coverage timeline
机器之心机器之心
本研究由匹兹堡大学电气与计算机工程系的 Haoming Wang 与 Wei Gao 完成。 看一段房间视频，视觉语言模型（VLM）能够认出桌子、椅子和柜子。但在它的内部，这些物体是按颜色、形状组织在一起，还是形成了一张包含邻接关系的空间地图？ 为了回答这个问题，匹兹堡大学的研究人员将研究视角从模型的最终答案转向隐空间，也就是模型在处理输入时形成的内部数值表征。通过跨场景特征分离与低维可视化，研究人员在模型中间层观察到了更清晰的空间结构：圆环、网格和随机布局，都能在提取后的表征中找到对应。这使「模型究竟如何理解空间」变成了一个可以直接观察和检验的问题。本文已被 NeruIPS 2026 录用为 spotlight paper。 论文标题：Uncovering and Shaping the Latent Representation of 3D Scene Topology in Vision-Language Models 文章链接：https://arxiv.org/abs/2605.07148 代码链接：https://github.com/pittisl/vlm-latent-shaping 人类通常能够把不断变化的第一视角画面整合为相对稳定的认知地图。它保留的核心信息包括「谁挨着谁」，「谁位于谁的哪一侧」等三维位置关系。本文将物体间的位置与邻接结构称为场景拓扑，并借此考察 VLM 是否也形成了某种不局限于二维图像网格的内部表示。 这一问题很难仅靠答题准确率解决，因为模型可能借助物体共现、语言先验或其他线索给出答案；attention map 则显示模型关注了哪些图像区域。它们都不足以回答不同物体的信息是否在模网络内部组成了连贯的三维结构，以及这种结构出现在哪些层。 初步分析结果表明，颜色、形状等属性比三维位置更容易从模型激活 (activation) 中读出。直接分析原始表征，往往首先看到明显的外观差异。因此，研究的关键不只是寻找空间信号，更多的是解决一个可解释性的问题：当多种视觉信息混在模型的同一套高维表征中时，怎样把关心的部分分离出来？ 方法介绍 论文的分析过程从物体级表征出发，先聚合分散的视觉信息，再剔除稳定的身份属性，最后将剩余空间结构可视化。 步骤 1 从视觉标记中提取物体表征 一个物体通常对应多个图像小块，还会出现在视频的不同帧中。研究利用合成
Related stories
VisInteract: Interactive Text-to-Visualization for Imperfect Queries, NeurIPS 2026 Spotlight
Researchers from Hong Kong Polytechnic University and ByteDance propose VisInteract, a new paradigm for dynamic interactive text-to-visualization that handles imperfect user queries. They introduce VisInteract-Bench, a benchmark acknowledging ambiguous, incomplete, or erroneous inputs, and Vis-MCTS, a Monte Carlo tree search-based method that branches and backtracks between data retrieval, chart generation, and clarification. The work was accepted as a Spotlight paper at NeurIPS 2026 (292 out of 30,709 submissions).
1 reports · 1 sources
Apple and Duke propose LensVLM, compressing long context into images with on-demand zoom
Apple's foundation model team and Duke University introduced LensVLM, a method that compresses long context into images and lets the model zoom into relevant pages on demand. In seven text QA benchmarks, it achieved 68.9% average accuracy with 4.3x effective compression, close to the 72.4% of full-text reference. The work was accepted at NeurIPS 2026, and a public 9B demo is available.
LoopVL open-source VLM shows data efficiency with 260x less training data
Researchers from Renmin University, Baidu, and others released LoopVL, an open-source vision-language model based on the Loop Transformer. Despite using only 0.14T training tokens versus about 36T for Qwen3.5-2B and InternVL3.5-2B, LoopVL matches or exceeds them on benchmarks like LogicVista, MMMU-Pro, and MathVision, demonstrating high data efficiency. The team also observed cross-loop visual attention reorganization, termed 'Visual Aha Moment'.
STILL: Content-Aware Intra-Layer Hybrid Attention Cuts Memory 80%, Speeds Prefill 83%
Researchers from Harbin Institute of Technology (Shenzhen) and collaborators present STILL, a NeurIPS 2026-accepted architecture that routes tokens to Softmax or linear attention within the same layer based on content importance via a self-saliency score. It achieves 80% memory reduction and 83% prefilling speedup at 16K context, with only 0.04B training tokens for linearization, and up to 86.2% improvement over prior linearization methods on long-context benchmarks.
News updates every hour. Subscribe in your RSS reader to get new stories automatically.
Subscribe