# Video Generation Models are General-Purpose Vision Learners

> GenCeption repurposes a text-to-video diffusion model as a unified perception model, achieving state-of-the-art across multiple vision tasks with orders of magnitude less training data.

- **Source:** [arXiv](https://arxiv.org/abs/2607.09024)
- **Published:** 2026-07-14
- **Permalink:** https://picx.dev/p/nJG15A
- **Whiteboard:** https://picx.dev/p/nJG15A/image

## Summary

## Summary (Overview)

- **GenCeption** repurposes a large-scale text-to-video diffusion model (WAN 2.1) as a feed-forward, unified perception model, achieving state-of-the-art performance across multiple vision tasks (depth, surface normals, camera pose, foreground segmentation, expression-referring segmentation, 3D keypoints) without task-specific architectures.
- The model is trained predominantly on synthetic data (7,500 videos, human-centric) yet generalizes remarkably to real-world footage and out-of-distribution categories (animals, robots), exhibiting emergent sim-to-real transfer and multi-instance handling.
- Compared to alternative pretraining paradigms (V-JEPA, VideoMAE V2), the video generative backbone yields significantly better downstream performance under identical finetuning data and model size, with preliminary data and model scaling properties.
- GenCeption demonstrates exceptional data efficiency: it matches or surpasses specialized models like D4RT and VGGT-Ω using 7× to 500× less training data.
- The work advocates a paradigm shift from task-specific computer vision models to a unified generalist architecture, analogous to LLMs, where task specification is handled by data format rather than architectural changes.

## Introduction and Theoretical Foundation

The paper draws inspiration from the evolution of NLP, where generative pre-training (next-token prediction) collapsed diverse linguistic tasks into a single foundation model. In contrast, computer vision remains in a "specialized model" stage, with task-specific architectures such as Segment Anything (for segmentation) and Depth Anything (for geometry). The authors argue that the missing element is a universal pre-training objective analogous to next-token prediction, which must satisfy three core imperatives:

1. **Spatio-Temporal Evolution** – The model must internalize the 4D temporal causality and physics of a moving world.
2. **Vision-Language Alignment** – Visual features should be natively aligned with linguistic semantics to enable instruction-following.
3. **Scalability** – The paradigm should scale in both data and compute, enabling emergent intelligence.

The paper contends that **large-scale text-to-video generation** uniquely satisfies all three requirements: it forces the model to learn spatiotemporal priors (3D geometry, object permanence, physics), natively conditions on text, and has been scaled to massive datasets and compute. This forms the basis for **GenCeption**, which treats a pre-trained video diffusion backbone as a rich feature extractor and adapts it via a lightweight post-training phase to perform diverse perception tasks.

## Methodology

### Feed-Forward Reformulation
The core of GenCeption is to convert the iterative diffusion process into a single-step feed-forward prediction. The pre-trained DiT (Diffusion Transformer) originally operates on noisy latents $x_t$ and predicts the velocity $v = \epsilon - x_0$ (Rectified Flow). In GenCeption, the input is the **clean latent** of the input video, and the timestep is fixed to $t=0$ (the end of the generative process). The DiT output is then **negated** to produce $ -v = x_0 - \epsilon$, which aligns more closely with the target latent. This simple reformulation allows the model to extract rich features from the final layer without architectural modifications.

### Unified Task Representation
**Dense tasks** (depth, normals, segmentation, DensePose, camera raymap) are represented in standard 3-channel RGB space ($[0,1]$). For example, depth is normalized via a nonlinear mapping: $d' = \text{clip}(\alpha \log(d+1), 0, 1)$, where $\alpha$ balances near-field details and far-field structures. Camera poses are encoded as a "Rothko" raymap: ray origins in the center, ray directions in the periphery, fitting 6-channel data into 3 channels by spatial partitioning.

**Sparse tasks** (2D/3D keypoints) are enabled by appending $T$ learnable tokens (one per frame) to the video latents. After passing through the DiT, an MLP decodes each token to a $K$-dimensional target. The tokens are positioned using 3D RoPE with learnable spatial positions and temporal position interpolation to stay within pre-training bounds.

### Training Recipe
- **Unified loss**: Only a standard $L_2$ loss is used, applied in latent space for dense tasks and output space for sparse tasks. Task-specific customizations (e.g., scale-invariant depth) are handled at the data level (median normalization, log mapping).
- **Synthetic data generation**: 7,500 human-centric videos are rendered using 800 RenderPeople assets, 200 motions (CMU MoCap), varied backgrounds (HDRI, full scenes), and camera trajectories. Ground truth includes normals, depth, segmentation masks, 2D/3D joint positions, and camera poses, generated via Blender render passes.
- **Optimization**: Adam optimizer, learning rate $5e-5$, batch size 64 on 256 v6e TPUs, 15,000 steps with linear warmup over 250 steps. Gradient clipping and gradient dropping are used for stability.

## Empirical Validation / Results

### Comparison to State-of-the-Art (Table 1)
| Task | Benchmark | Metric | SOTA (Specialist) | GenCeption (L, 14B) |
|------|-----------|--------|-------------------|---------------------|
| Normals | Sintel | mAE ↓ | Lotus-2: 30.3 | **29.7** |
| Depth | KITTI | AbsRel ↓ | VGGT-Ω: 0.041 | **0.048** (close) |
| Cam. Pose | Sintel | ATE ↓ | D4RT: 0.148 | **0.130** |
| Foreground Seg. | V.Mat. | MSE ↓ | RVM: 0.0010 | **0.0027** (competitive) |
| Expression-Ref. Seg. | Ref-DAVIS | J&F ↑ | SAM3+Gemini: 64.5 | **76.4** |
| 3D Keypoint | EMDB | MPJPE ↓ | TRAM: 74.4 | **71.8** |

GenCeption (specialist L) achieves state-of-the-art on normals, camera pose, expression-referring segmentation, and 3D keypoints, and is highly competitive on depth and foreground segmentation. The generalist (joint training) shows mixed results: dense tasks sometimes degrade, but segmentation benefits.

### Ablation Studies (Table 2)
| Backbone | Model Size | Training Data | Sintel AbsRel ↓ | KITTI AbsRel ↓ | ETH3D AbsRel ↓ |
|----------|------------|---------------|-----------------|----------------|-----------------|
| V-JEPA (H) | 0.6B | 7.5K videos | 0.422 | 0.226 | 0.196 |
| VideoMAE V2 (G) | 1B | 7.5K videos | 0.260 | 0.105 | 0.099 |
| WAN 2.1 (S) | 1.3B | 7.5K videos | **0.201** | **0.099** | **0.068** |
| WAN 2.1 (L) | 14B | 7.5K videos | **0.181** | **0.060** | **0.039** |

The video generative backbone (WAN 2.1) significantly outperforms V-JEPA and VideoMAE V2 under the same finetuning data. The 14B model trained on only 7.5K videos achieves comparable results to D4RT (1B, ~86M frames) and VGGT-Ω (1B, ~600M frames), demonstrating exceptional data efficiency (7× to 500× less data).

### Emergent Behaviors
- **Sim-to-real transfer**: Trained purely on synthetic human videos, the model generalizes to real-world footage with fine-grained details (e.g., cat whiskers, hair segmentation) that exceed the quality of the synthetic training data.
- **Multi-instance generalization**: Trained on videos with a single object, it handles multiple objects in zero-shot (Fig. 10a).
- **Out-of-distribution categories**: Generalizes from humans to animals, robots, and anthropomorphic characters (Fig. 10b).

## Theoretical and Practical Implications

- **Paradigm shift**: The paper reframes video generation as a pre-training objective for generalist vision, similar to how next-token prediction unified NLP. This suggests that future vision models should be built on top of generative video backbones rather than task-specific architectures.
- **Data efficiency**: The ability to achieve SOTA with orders of magnitude less training data highlights the richness of the pre-trained representations. This has practical implications for domains where labeled data is scarce.
- **Unified architecture**: The success of a single backbone, head, and loss across diverse tasks (geometric, semantic, sparse) demonstrates that task specification can be moved from architecture to data format, enabling scalable multi-task learning.
- **Emergent properties**: The model's ability to generalize beyond its training distribution (synthetic → real, humans → animals) suggests that video generative models learn a universal "world model" that captures fundamental physical and visual priors.

## Conclusion

GenCeption shows that large-scale text-to-video generation is a powerful pre-training paradigm for computer vision, satisfying the requirements of spatiotemporal learning, vision-language alignment, and scalability. By repurposing a pre-trained video diffusion model into a feed-forward perception model, it achieves state-of-the-art results across multiple tasks, exhibits data efficiency, and demonstrates emergent behaviors. The authors argue that this paradigm shift—from task-specific engineering to scalable generative pre-training—paves the way for truly unified and generalist vision intelligence. Future directions include extending the approach to more tasks, scaling further, and exploring alternative pre-training objectives that natively support diverse downstream outputs.

---

_Markdown view of https://picx.dev/p/nJG15A, served by PicX — AI-generated visual whiteboard summaries of research papers._