NVIDIA and Waterloo Open-Source PixelUMM, an Encoder-Free Unified Image-Video Model
NVIDIA and the University of Waterloo released PixelUMM, an encoder-free unified multimodal model that handles image and video understanding and generation directly in pixel space, bypassing ViT and VAE. It uses spatial patches for images and spatiotemporal tubelets for videos, feeding raw pixels through a linear projection into a shared Mixture-of-Transformers backbone with separate parameters for understanding and generation but shared attention.
Coverage timeline
机器之心机器之心
统一多模态模型（Unified Multimodal Models, UMMs）通常为理解和生成分别使用不同的视觉表示：用 ViT 提取理解所需的语义特征，用 VAE 提供生成所需的潜变量。这样一来，同一张图像要以两种 token 进入上下文，视觉上下文随之变长；这种双流接口也不同于视觉-语言模型（VLM）常用的单一视觉流，难以直接沿用成熟的 VLM 预训练流程。 近期的像素空间建模提供了另一条路：JiT 等工作表明，图像生成可以不依赖 VAE、直接在像素上进行。但将这一范式扩展到 视频 并非易事：视频生成通常依赖 3D VAE，视频理解则通常用图像编码器逐帧处理，两者表示时间维度的方式不同，如何设计理解与生成共用的视频接口仍是一个开放问题。 为此，英伟达与滑铁卢大学提出 PixelUMM，一个直接在像素空间中统一图像与视频理解和生成的无编码器模型 。 论文标题：PixelUMM: Encoder-Free Unified Image and Video Understanding and Generation 项目主页：https://nv-tlabs.github.io/PixelUMM 论文：https://arxiv.org/abs/2609.38597 开源代码：https://github.com/nv-tlabs/PixelUMM 开源模型：https://huggingface.co/nvidia/PixelUMM 论文作者：魏聪（Cong Wei）、Xuanchi Ren、Bryan Chu、Weiming Ren、Huan Ling、Jiahui Huang、Laura Leal-Taixé、Sanja Fidler、Wenhu Chen、Zian Wang、Jay Zhangjie Wu 作者单位：NVIDIA、滑铁卢大学 PixelUMM 将图像切分为空间 patch、将视频切分为时空 tubelet，理解和生成都使用这种表示，并且仅通过一层线性投影就把原始像素接入共享的多模态主干网络。主干采用 Mixture-of-Transformers 架构，理解与生成各有一套参数，但共享同一个注意力。模型以自回归方式预测文本，同时把 JiT 式的干净像素预测扩展到视频，用像素空间的 flow matching 生成图像和视频。 实验表明
Related stories
Open-source projects unlock hidden Tensor Core throughput on GeForce GPUs, doubling RTX 4090 performance
Open-source developers released projects on GitHub that unlock higher Tensor Core throughput on NVIDIA GeForce GPUs by modifying boot-time configuration, without driver swaps or BIOS flashes. Tests reportedly doubled the FP16/FP32 and BF16/FP32 throughput of an RTX 4090 from about 165 TFLOPS to over 330 TFLOPS, with gains also observed on RTX 3090 Ti, RTX 4070 Ti SUPER, and RTX 2080 Ti. The projects target RTX 20, 30, and 40 series, with some experiments done on Windows 11.
1 reports · 1 sources
Google open-sources TorchTPU, making TPU a native PyTorch device
At PyTorch Conference 2026, Google announced the open-sourcing of TorchTPU, a software stack enabling native PyTorch execution on TPUs, developed with Meta and integrated by vLLM, SGLang, Hugging Face, Ray, and TorchTitan. The project, available on GitHub with a 23-page technical report on arXiv, allows deployment to TPU via a single parameter in Hugging Face Transformers v5, positioning TPU alongside GPU and CPU as a native PyTorch option.
VAST open-sources EdenLive real-time world model, 24+ FPS at 720P on consumer GPU
VAST released and open-sourced EdenLive, a real-time world model that generates interactive visuals at over 24 FPS at 720P on a single Nvidia RTX 5090 consumer GPU. It supports arbitrary reference inputs, customizable art styles, and long-duration continuous generation, with end-to-end latency in the hundreds of milliseconds. The release includes model weights, inference code, a technical report, and an end-to-end demo.
1 reports · 1 sources
Google launches TorchTPU, vLLM adds TPU backend
Google announced TorchTPU, an open-source version of PyTorch that lets AI applications run on TPUs and Google Cloud via simple configuration changes without rewriting code. Concurrently, vLLM introduced vllm-torchtpu, a new backend for Google Cloud TPUs built on TorchTPU, featuring Pallas kernels, multi-host parallelism, KV cache offloading, and disaggregated serving. These moves, following Amazon's similar tool, challenge Nvidia's dominance in AI development.
3 reports · 3 sources
News updates every hour. Subscribe in your RSS reader to get new stories automatically.
Subscribe