Summary (Overview)

• GenCeption repurposes a large-scale text-to-video diffusion model (WAN 2.1) as a feed-forward, unified perception model, achieving state-of-the-art performance across multiple vision tasks (depth, surface normals, camera pose, foreground segmentation, expression-referring segmentation, 3D keypoints) without task-specific architectures.
• The model is trained predominantly on synthetic data (7,500 videos, human-centric) yet generalizes remarkably to real-world footage and out-of-distribution categories (animals, robots), exhibiting emergent sim-to-real transfer and multi-instance handling.
• Compared to alternative pretraining paradigms (V-JEPA, VideoMAE V2), the video generative backbone yields significantly better downstream performance under identical finetuning data and model size, with preliminary data and model scaling properties.
• GenCeption demonstrates exceptional data efficiency: it matches or surpasses specialized models like D4RT and VGGT-Ω using 7× to 500× less training data.
• The work advocates a paradigm shift from task-specific computer vision models to a unified generalist architecture, analogous to LLMs, where task specification is handled by data format rather than architectural changes.

Introduction and Theoretical Foundation

The paper draws inspiration from the evolution of NLP, where generative pre-training (next-token prediction) collapsed diverse linguistic tasks into a single foundation model. In contrast, computer vision remains in a "specialized model" stage, with task-specific architectures such as Segment Anything (for segmentation) and Depth Anything (for geometry). The authors argue that the missing element is a universal pre-training objective analogous to next-token prediction, which must satisfy three core imperatives:

1. Spatio-Temporal Evolution – The model must internalize the 4D temporal causality and physics of a moving world.
2. Vision-Language Alignment – Visual features should be natively aligned with linguistic semantics to enable instruction-following.
3. Scalability – The paradigm should scale in both data and compute, enabling emergent intelligence.

The paper contends that large-scale text-to-video generation uniquely satisfies all three requirements: it forces the model to learn spatiotemporal priors (3D geometry, object permanence, physics), natively conditions on text, and has been scaled to massive datasets and compute. This forms the basis for GenCeption, which treats a pre-trained video diffusion backbone as a rich feature extractor and adapts it via a lightweight post-training phase to perform diverse perception tasks.

Methodology

Feed-Forward Reformulation

The core of GenCeption is to convert the iterative diffusion process into a single-step feed-forward prediction. The pre-trained DiT (Diffusion Transformer) originally operates on noisy latents $x_t$ and predicts the velocity $v = \epsilon - x_0$ (Rectified Flow). In GenCeption, the input is the clean latent of the input video, and the timestep is fixed to $t=0$ (the end of the generative process). The DiT output is then negated to produce $-v = x_0 - \epsilon$, which aligns more closely with the target latent. This simple reformulation allows the model to extract rich features from the final layer without architectural modifications.

Unified Task Representation

Dense tasks (depth, normals, segmentation, DensePose, camera raymap) are represented in standard 3-channel RGB space ($[0,1]$). For example, depth is normalized via a nonlinear mapping: $d' = \text{clip}(\alpha \log(d+1), 0, 1)$, where $\alpha$ balances near-field details and far-field structures. Camera poses are encoded as a "Rothko" raymap: ray origins in the center, ray directions in the periphery, fitting 6-channel data into 3 channels by spatial partitioning.

Sparse tasks (2D/3D keypoints) are enabled by appending $T$ learnable tokens (one per frame) to the video latents. After passing through the DiT, an MLP decodes each token to a $K$-dimensional target. The tokens are positioned using 3D RoPE with learnable spatial positions and temporal position interpolation to stay within pre-training bounds.

Training Recipe

• Unified loss: Only a standard $L_2$ loss is used, applied in latent space for dense tasks and output space for sparse tasks. Task-specific customizations (e.g., scale-invariant depth) are handled at the data level (median normalization, log mapping).
• Synthetic data generation: 7,500 human-centric videos are rendered using 800 RenderPeople assets, 200 motions (CMU MoCap), varied backgrounds (HDRI, full scenes), and camera trajectories. Ground truth includes normals, depth, segmentation masks, 2D/3D joint positions, and camera poses, generated via Blender render passes.
• Optimization: Adam optimizer, learning rate $5e-5$, batch size 64 on 256 v6e TPUs, 15,000 steps with linear warmup over 250 steps. Gradient clipping and gradient dropping are used for stability.

Empirical Validation / Results

Comparison to State-of-the-Art (Table 1)

GenCeption (specialist L) achieves state-of-the-art on normals, camera pose, expression-referring segmentation, and 3D keypoints, and is highly competitive on depth and foreground segmentation. The generalist (joint training) shows mixed results: dense tasks sometimes degrade, but segmentation benefits.

Ablation Studies (Table 2)

The video generative backbone (WAN 2.1) significantly outperforms V-JEPA and VideoMAE V2 under the same finetuning data. The 14B model trained on only 7.5K videos achieves comparable results to D4RT (1B, ~86M frames) and VGGT-Ω (1B, ~600M frames), demonstrating exceptional data efficiency (7× to 500× less data).

Emergent Behaviors

• Sim-to-real transfer: Trained purely on synthetic human videos, the model generalizes to real-world footage with fine-grained details (e.g., cat whiskers, hair segmentation) that exceed the quality of the synthetic training data.
• Multi-instance generalization: Trained on videos with a single object, it handles multiple objects in zero-shot (Fig. 10a).
• Out-of-distribution categories: Generalizes from humans to animals, robots, and anthropomorphic characters (Fig. 10b).

Theoretical and Practical Implications

• Paradigm shift: The paper reframes video generation as a pre-training objective for generalist vision, similar to how next-token prediction unified NLP. This suggests that future vision models should be built on top of generative video backbones rather than task-specific architectures.
• Data efficiency: The ability to achieve SOTA with orders of magnitude less training data highlights the richness of the pre-trained representations. This has practical implications for domains where labeled data is scarce.
• Unified architecture: The success of a single backbone, head, and loss across diverse tasks (geometric, semantic, sparse) demonstrates that task specification can be moved from architecture to data format, enabling scalable multi-task learning.
• Emergent properties: The model's ability to generalize beyond its training distribution (synthetic → real, humans → animals) suggests that video generative models learn a universal "world model" that captures fundamental physical and visual priors.

Conclusion

GenCeption shows that large-scale text-to-video generation is a powerful pre-training paradigm for computer vision, satisfying the requirements of spatiotemporal learning, vision-language alignment, and scalability. By repurposing a pre-trained video diffusion model into a feed-forward perception model, it achieves state-of-the-art results across multiple tasks, exhibits data efficiency, and demonstrates emergent behaviors. The authors argue that this paradigm shift—from task-specific engineering to scalable generative pre-training—paves the way for truly unified and generalist vision intelligence. Future directions include extending the approach to more tasks, scaling further, and exploring alternative pre-training objectives that natively support diverse downstream outputs.