# Vision as Unified Multimodal Generation

> Vision tasks can be unified as multimodal text-and-image generation, matching specialized models while preserving general abilities and enabling zero-shot task variants.

- **Source:** [arXiv](https://arxiv.org/abs/2607.06560)
- **Published:** 2026-07-09
- **Permalink:** https://picx.dev/p/cr9gjl

## Summary

## Summary (Overview)

- **Unified Formulation**: The paper proposes formulating computer vision as **unified multimodal generation**, where heterogeneous visual tasks (detection, OCR, keypoints, segmentation, depth, normals, 3D reconstruction, camera pose) are expressed through the native text and image generation spaces of a unified multimodal model (UMM), without task-specific architectures or prediction heads.
- **SenseNova-Vision Model**: Built from the off-the-shelf UMM **Bagel-7B-MoT**, the model matches or approaches leading task-specialized systems across four perception families: structured visual understanding, dense geometric prediction, segmentation, and multi-view visual geometry.
- **SenseNova-Vision Corpus (SN-VC)**: A large-scale computer-vision instruction-response corpus (with a released 50M-example subset, SN-VC-50M) that converts heterogeneous public annotations into decodable text, image, and mixed text-image targets.
- **Language-Defined Task Variants**: The model supports zero-shot task variants not explicitly in the training corpus, including referring-style interactive segmentation, visual grounded (VGD) segmentation with text-specified cues, free-form color-coded masks, and text/letter-level segmentation.
- **General Capabilities Preserved**: After mixed-task fine-tuning, the model retains core multimodal abilities (MMVP: 79.0 vs. 83.3 for Bagel; GenEval: 0.85 vs. 0.82), demonstrating that vision-task training can be absorbed without catastrophic forgetting.

---

## Introduction and Theoretical Foundation

### Background and Motivation

Traditional computer vision relies on **task-specialized systems** with distinct architectures, losses, decoding rules, and evaluation protocols for each task family (detection, segmentation, depth, etc.). This task-specific organization makes visual supervision difficult to **share, reuse, and compose** across tasks.

Prior unification efforts fall into complementary but incomplete routes:

| Approach | Representative Works | Limitation |
|---|---|---|
| **Sequence-format unifications** | Pix2Seq, Pix2Seq v2, UniTAB, OFA, Uni-Perceiver, Unified-IO 2, Florence-2 | Different output types still require serialization/parsing conventions; unnatural for dense maps |
| **Representation-centric models** | SAM, Painter, SegGPT, Depth Anything, VGGT | Generalize within visually coherent families but lack unified output space and language control |
| **Image-generative methods** | Marigold, Lotus/Lotus-2, DICEPTION, Vision Banana | Handle dense maps but struggle with symbolic records |
| **MLLM-based systems** | LISA, PixelLM, X-SAM, G²VLM, VisionLLM v2 | Add language/reasoning but route dense outputs through task-specific decoders |

### Theoretical Foundation

The core premise: **heterogeneous computer vision tasks can be cast as unified multimodal generation**, analogous to what GPT did for NLP tasks. The formulation exploits two complementary native generation spaces of UMMs:

- **Text generation** expresses symbolic visual answers: categories, spatial references, OCR strings, camera parameters.
- **Image generation** is natural for dense prediction since masks, depth maps, surface normals, and point maps are spatially aligned with the input image.

The formulation is: 

$$\text{Response} = \text{Generate}(\text{Instruction}, \text{Visual Inputs})$$

where the response can be text, image, or mixed text-image, and the instruction specifies the task, target regions/views, output schema, and decoding convention.

---

## Methodology

### 3.1 Data Protocol

The protocol defines a common sample schema: **one or more visual inputs + natural-language instruction + decodable target response**, organized across four families:

**1. Structured Visual Understanding** — outputs decoded as sparse symbolic records (detection, referring, pointing, keypoints, OCR, layout, GUI grounding):
- Spatial fields written as normalized image coordinates
- Lightweight markers: `<p>`, `<bbox>`, `<point>` delimit phrases and coordinate fields
- All tasks share the same text-generation space, separated by instructions and schemas

**2. Dense Geometric Prediction** — pixel-aligned targets (depth, surface normals) represented as **image outputs**:
- Depth: metric values → inverse depth → normalized grayscale images
- Surface normals: rendered as RGB maps with channels encoding normal components

**3. Segmentation** — single-target vs. multi-region formats:
- Single-target (referring, reasoning, interactive): binary mask image with fixed foreground/background colors
- Multi-region (generic, panoptic, GCG): **mixed text-image response** — text lists regions with `<color>` markers for RGB palette values; image renders the color-coded mask

**4. Multi-View Visual Geometry** — ordered image sets:
- Dense scene geometry: per-view dense XYZ point maps stored in RGB channels
- Camera pose: structured sequences with reserved tokens `<frame>`, `<quat>`, `<offset>`, `<scale>` encoding quaternion rotation, translation direction, and scale relative to reference frame

### 3.2 Corpus Construction (SN-VC)

- **Source composition**: 46.7M structured perception, 32.3M multi-view 3D, 22.6M segmentation, 3.1M dense geometry examples
- **SN-VC-50M**: Released 50M-example subset with generated/curated targets (using Rex-Omni pipeline for detection/OCR, MoGe-2 for depth/normal densification, LingBot-Depth for sparse depth completion)
- Evaluation benchmark overlaps are excluded from training

### 4. Training Setup

- **Base model**: Bagel-7B-MoT (off-the-shelf UMM)
- **Mixed-task fine-tuning**: Joint sampling from converted vision samples + auxiliary multimodal data (VQA, text-to-image, image-to-image) to preserve general capabilities
- **Loss functions**:
  - Text outputs: standard **cross-entropy (CE)** loss under next-token prediction
  - Visual outputs: **rectified-flow objective** in VAE latent space
- **High-resolution**: SigLIP2 input up to 980 pixels for fine spatial conditioning
- **Multi-view**: At most 10 views per sample; 2,009 reserved vocabulary entries for pose tokens (2,001 quantized pose parameters + 8 structural placeholders)
- **Hyperparameters**: AdamW, LR $2.5 \times 10^{-5}$, no weight decay, 50K steps, EMA ratio 0.995, VAE frozen

---

## Empirical Validation / Results

### 5.1 Structured Visual Understanding

**Table 1: Quantitative comparison (F1@mIoU unless noted)**

| Method | COCO-Com. | RefCOCOg V/T | LVIS | Dense200 | VisDrone (F1@Point) | HierText | ICDAR15 | ScreenSpot-V2 | COCO-Kpt. |
|---|---|---|---|---|---|---|---|---|---|
| Grounding DINO-Swin-T | 56.6 | 25.2/45.9/46.8 | 38.8 | 33.1 | 38.5 | – | – | – | – |
| Bagel | 50.2 | 74.6/76.4/77.8 | 46.8 | 42.4 | 23.0 | 36.9 | 7.1 | 15.8 | 81.1 |
| Rex-Omni | 52.9 | 79.9/73.6/74.3 | 46.9 | 58.3 | 35.8 | 58.9 | 28.0 | 28.1 | 88.4 | 32.6 |
| **SenseNova-Vision** | **56.6** | **80.2/79.6/80.5** | **54.8** | **66.8** | **43.3** | **62.9** | **31.2** | **49.5** | 85.9 | **34.6** |

SenseNova-Vision **leads on all benchmarks** except ICDAR15, achieving the best overall structured understanding performance.

### 5.2 Dense Geometric Prediction

**Table 2: Depth (abs rel. ↓ / δ₁ ↑) and Normal (mean err. ↓ / δ₁₁.₂₅ ↑)**

| Method | NYUv2 | KITTI | ETH3D | ScanNet | DIODE | ScanNet (Norm) | iBims-1 | NYUv2 (Norm) |
|---|---|---|---|---|---|---|---|---|
| DepthAnything | 4.3/98.1 | 7.6/94.7 | 12.7/88.2 | 4.3/98.1 | 26.0/75.9 | – | – | – |
| ∗MoGe-2 | 3.5/98.0 | 5.5/97.7 | 3.4/98.8 | 3.4/98.3 | 23.0/82.3 | 12.8/68.4 | 14.7/70.4 | 14.7/62.3 |
| Lotus-2 | 4.1/97.6 | 6.7/94.5 | 4.6/98.1 | 4.2/97.6 | 22.1/75.2 | 14.2/66.8 | 15.4/70.4 | 16.9/59.0 |
| **SenseNova-Vision** | **4.0/98.1** | **5.9/95.9** | **4.3/97.4** | **3.9/98.0** | **20.6/76.4** | **12.8/68.9** | 15.4/69.1 | **14.4/62.7** |

SenseNova-Vision **outperforms generation-based baselines** (Marigold, DICEPTION, FE2E, Lotus-2) and remains competitive with geometry-specialized models (MoGe-2, DepthAnything).

### 5.3 Segmentation

**Table 3: Segmentation comparison**

| Method | Pan./Sem. | RefCOCO /+/g | Rea. Seg. | GCG Seg. | Inter. Seg. |
|---|---|---|---|---|---|
| LISA-7B | – | 74.9/65.1/67.9 | 52.9/47.3 | 62.0/61.7 | – |
| X-SAM | 54.7/66.5 | 85.1/78.0/83.8 | 56.6/57.8 | 69.4/69.0 | 65.4/70.0 |
| **SenseNova-Vision** | 48.8/64.0 | 81.3/76.0/80.3 | **63.2/60.7** | 65.7/66.2 | 60.9/73.9 |

Strong results on **reasoning and GCG segmentation**; specialized models (X-SAM) remain stronger on generic/referring metrics due to SAM/Mask2Former mask priors.

### 5.4 Multi-View Visual Geometry

**Table 4: Reconstruction (Acc. ↓ / Comp. ↓ / F1 ↑) and Camera Pose (RRA↑/RTA↑/AUC@30↑)**

| Method | 7Scenes | ETH3D | Re10K | CO3Dv2 |
|---|---|---|---|---|
| DUSt3R | 0.026/0.034/87.1 | 0.359/0.531/66.6 | 99.8/84.9/67.6 | 97.7/93.4/78.3 |
| VGGT | 0.023/0.032/88.4 | 0.177/0.155/80.9 | 100.0/93.5/79.3 | 98.3/96.6/89.2 |
| **SenseNova-Vision** | 0.028/0.026/**87.9** | 0.301/0.175/72.2 | 99.8/94.2/77.3 | 97.4/95.4/80.1 |

Approaches the leading specialist (VGGT) and **outperforms generalist geometric approaches** (G2VLM, MapAnything). A performance gap remains on several metrics, highlighting the benefit of geometry-focused training and inductive biases.

### 5.5 Comparison with Generalist Vision Models

- **vs. Youtu-VL**: SenseNova-Vision wins on detection (mAP 53.7 vs. 47.1), semantic segmentation (71.2 vs. 70.4), referring segmentation (80.3 vs. 76.5 cIoU), and depth (δ₁ 98.1 vs. 90.4)
- **vs. Vision Banana**: Competitive or better on semantic segmentation (71.2 vs. 69.9), referring segmentation (80.3 vs. 73.8), and several depth/normal metrics

### 5.6 Additional Analysis

**Convergence patterns** (Fig. 6): Depth/normals converge fastest (spatially aligned targets); camera pose converges slowest (cross-view alignment + new tokens); dense detection is slowest among structured tasks.

**Zero-shot task variants demonstrated**:
1. **Referring-style interactive segmentation**: Text-encoded point coordinates `<p><point>[x,y]</point></p>` → binary masks (combines referring segmentation + interactive segmentation + coordinate localization)
2. **Dense instance segmentation**: Detection instructions adapted to request instance-level masks — extracts up to ~100 tiny objects in dense/aerial scenes (Dense200, VisDrone)
3. **VGD segmentation**: Text-specified reference point/box → segment all same-class instances
4. **Free-form color-coded masks**: Natural-language color descriptions (e.g., "solid green") without fixed RGB tags
5. **Text/letter-level segmentation**: Segmenting the word "coke" or individual letters without text-segmentation training data

---

## Theoretical and Practical Implications

### Theoretical Significance

1. **Unification of output spaces**: The work demonstrates that symbolic records, dense spatial targets, and mixed outputs can coexist in one native multimodal generation framework — a first across prior sequence-format, representation-centric, image-generative, and MLLM-based routes.

2. **Cross-modal correspondences**: Joint training strengthens correspondences such that the same information (e.g., spatial location, color, text) can be flexibly expressed across modalities — text coordinates ↔ image masks, OCR strings ↔ text-shaped masks, RGB palettes ↔ natural-language color descriptions.

3. **Emergent task composition**: The model recombines capabilities from separate domains to handle tasks never explicitly trained (e.g., coordinate-to-mask, text segmentation), suggesting that unified generation enables **programmable perception** beyond fixed task taxonomies.

### Practical Implications

- **No task-specific heads**: A single model covers detection, OCR, keypoints, segmentation, depth, normals, point maps, and camera pose — simplifying deployment and reducing engineering overhead.
- **Scalable data pipeline**: The SN-VC protocol converts heterogeneous public annotations into unified instruction-response examples, providing a reproducible recipe for training vision-capable foundation models.
- **Language-defined variants**: Users can specify task variants combining category, color, region, and other cues through natural language, enabling flexible, on-the-fly task definition.
- **Public release**: Model and SN-VC-50M corpus are publicly available (GitHub + HuggingFace), enabling reproducible research.

---

## Conclusion

### Main Takeaways

SenseNova-Vision demonstrates that **computer vision can become a native generative capability of unified foundation models**, moving beyond isolated task-specific systems. The key contributions are:

1. A **unified multimodal generation formulation** casting heterogeneous vision tasks into UMM native input-output spaces
2. The **SenseNova-Vision Corpus** (SN-VC), a large-scale instruction-response corpus with decodable text, image, and mixed targets
3. A trained model achieving **competitive performance across four perception families** while supporting **language-defined task variants** beyond the training set

### Future Directions

1. **Stronger in-context learning**: Reducing boundaries between task domains so new visual tasks can be specified by examples, prompts, or mixed demonstrations
2. **Video extension**: Bringing temporal dynamics and web-scale video supervision into foundation-model training
3. **Scaling**: Larger corpus and model capacity, deeper integration with stronger language models, moving toward **world models** that perceive, reason about, and interact with the physical world

The work suggests that generative modeling of diverse 2D and 3D perception tasks may cultivate implicit spatial understanding, connecting naturally to the emerging frontiers of **physical intelligence**.

---

_Markdown view of https://picx.dev/p/cr9gjl, served by PicX — AI-generated visual whiteboard summaries of research papers._