Summary (Overview)

• Unified pixel-space diffusion: PixWorld reformulates 3D scene reconstruction and generation under a single pixel-space diffusion paradigm, eliminating the need for a pretrained VAE or RAE by supervising diffusion directly on multi-view renderings of a pixel-aligned 3D Gaussian representation.
• Geometry perception loss: Introduces a novel loss that aligns rendered views with ground-truth observations in the geometry-aware feature space of a frozen 3D foundation model, providing 3D structural supervision beyond 2D photometric and perceptual losses.
• Superior performance: A single PixWorld model outperforms prior latent-space generation methods on RealEstate10K, DL3DV-10K, and WorldScore, while matching or exceeding state-of-the-art feed-forward reconstruction methods on novel view synthesis.
• Key insight: Partitioning multi-view inputs into clean and noisy subsets within a two-stream diffusion transformer naturally unifies both tasks—clean views anchor reconstruction, noisy views are generated conditioned on the clean ones—with both outputs decoded into a shared 3D Gaussian representation.

Introduction and Theoretical Foundation

Background and Motivation

3D scene reconstruction and generation have historically been tackled by separate paradigms:

• Reconstruction is dominated by feed-forward methods that regress 3D representations directly from multi-view images (e.g., pixelsplat, MVSplat, DepthSplat).
• Generation has moved from per-scene score distillation to latent-space diffusion, where a diffusion model operates in the compressed feature space of a VAE or RAE, followed by a learned decoder to 3D Gaussians.

Recent work (Gen3R) attempted to unify both tasks in a latent-space framework, but inherits fundamental drawbacks:

• The diffusion objective is defined on intermediate latent features, not on the underlying 3D representation.
• Both branches suffer from information loss due to latent encoding/decoding.
• Requires pretraining of a VAE or RAE, adding computational cost.

Theoretical Basis: Pixel-Space Diffusion

The paper builds on the observation that latent diffusion inserts an autoencoding bottleneck between the diffusion variable and the final image supervision. Instead, following JiT, they adopt image prediction in pixel space:

Given a clean image $x \in \mathbb{R}^{H \times W \times 3}$, noise $\epsilon \sim \mathcal{N}(0,I)$, and timestep $t \in [0,1]$, the noisy input is $x_t = t x + (1-t)\epsilon$. The denoiser is parameterized as an image predictor:

$$f_\theta: (x_t, t, c) \mapsto \hat{x}, \quad (1)$$

where $c$ is conditional information. The predicted image is converted to a velocity field $\hat{v} = (\hat{x} - x_t)/(1-t)$, yielding the flow-matching objective:

$$\mathcal{L}_{\text{FM}} = \mathbb{E}_{x,\epsilon,t}\left[ \|\hat{v} - v\|_2^2 \right] = \mathbb{E}_{x,\epsilon,t}\left[ \frac{\|\hat{x} - x\|_2^2}{(1-t)^2} \right], \quad (2)$$

where $v = x - \epsilon$ is the ground-truth velocity.

Key insight: By keeping the diffusion variable and the rendered output in the same RGB domain, the diffusion objective can be supervised directly on rendered images from a 3D representation, aligning optimization with 3D scene fidelity rather than an intermediate latent target.

Methodology

Task Formulation

Given a scene with $N$ posed views, partition indices into a clean subset $\Omega_c$ and a noisy subset $\Omega_n$ such that $\Omega_c \cap \Omega_n = \emptyset$, $|\Omega_c| \geq 1$. When $\Omega_n = \emptyset$, the task reduces to multi-view reconstruction. Otherwise, the model predicts the noisy views conditioned on the clean ones (generation). An optional text prompt $y$ may condition the scene.

Mixed multi-view input is:

$$\tilde{I}^n_t = \begin{cases} I^n & n \in \Omega_c \\ t I^n + (1-t)\epsilon^n & n \in \Omega_n \end{cases}, \quad \epsilon^n \sim \mathcal{N}(0,I), \quad (4)$$

Two-Stream Diffusion Transformer

A 24-layer DiT with SD3-style MMDiT design has two parallel streams:

• Clean stream: processes conditioning views ($\Omega_c$), receives time embedding at $t=1$.
• Noise stream: processes noisy views ($\Omega_n$), receives the sampled timestep $t$.

The two streams share cross-attention to text tokens and a single timestep embedder. Camera parameters are injected via PRoPE. The output heads predict per-pixel depth and 3D Gaussian attributes for every view (clean and noisy).

3D Gaussian Decoding and Rendering

Gaussian centers are obtained by unprojecting each pixel using its predicted depth: $\mu^n_p = \Pi^{-1}(p, \hat{d}^n_p, T^n)$. The scene representation $\hat{G}$ is rendered via differentiable renderer $\mathcal{R}: (\hat{G}, T^n) \mapsto \bar{I}^n$.

Losses

Rendering and depth objectives:

$$\mathcal{L}_{\text{render}} = \frac{1}{|\Omega_c|} \sum_{n \in \Omega_c} \|\bar{I}^n - I^n\|_2^2 \;+\; \frac{\mathbb{1}[|\Omega_n| > 0]}{|\Omega_n|} \sum_{n \in \Omega_n} \|\bar{v}^n - v^n\|_2^2 \;+\; \lambda_{\text{lpips}} \mathcal{L}_{\text{lpips}}, \quad (6)$$

where $\bar{v}^n = (\bar{I}^n - \tilde{I}^n_t)/(1-t)$ and $v^n = (I^n - \tilde{I}^n_t)/(1-t)$. The LPIPS term is active only when $t > t_{\text{th}}$.

Depth loss (using pseudo-depth from DA3):

$$\mathcal{L}_{\text{depth}} = \frac{1}{N} \sum_{n=1}^N \rho\left( \log \hat{D}^n - \log D^{\star n} \right), \quad (7)$$

where $\rho(\cdot)$ is Huber loss.

Geometry perception loss: Using a frozen 3D foundation model $\Psi$ (VGGT or $\pi_3$), extract multi-view geometric features:

$$\bar{H} = \Psi(\bar{I}, T), \quad H^\star = \Psi(I, T), \quad (8)$$

Then minimize average per-location cosine distance:

$$\mathcal{L}_{\text{geo}} = \mathbb{1}[t > t_{\text{th}}] \frac{1}{N H' W'} \sum_{n=1}^N \sum_p \left[ 1 - \frac{\langle \bar{h}^n_p, h^{\star n}_p \rangle}{\|\bar{h}^n_p\|_2 \|h^{\star n}_p\|_2} \right], \quad (9)$$

Gradients are stopped on the reference branch $H^\star$; only the rendered branch backpropagates.

Overall objective: $\mathcal{L} = \mathcal{L}_{\text{render}} + \lambda_{\text{depth}} \mathcal{L}_{\text{depth}} + \lambda_{\text{geo}} \mathcal{L}_{\text{geo}}$ with $\lambda_{\text{depth}}=1.0$, $\lambda_{\text{lpips}}=\lambda_{\text{geo}}=0.1$, and $t_{\text{th}}=0.3$.

Empirical Validation / Results

Training Details

• PixWorld has ~1.04B parameters, trained from scratch on RealEstate10K + DL3DV-10K (~67K scenes), augmented with 10M single images from BLIP-3o.
• Optimized with AdamW for ~200K steps at 336×448 resolution on 32 NVIDIA A800 GPUs.
• Frozen 3D critic: $\pi_3$ (Wang et al., 2025d).

Reconstruction Results (Table 1)

PixWorld is compared against MVSplat, DepthSplat, AnySplat, and YoNoSplat on novel view synthesis from 4/8 input views.

Table 1: Novel-view synthesis on RealEstate10K and DL3DV-10K.

PixWorld achieves best PSNR and LPIPS across all settings, demonstrating strong cross-view consistency.

Generation Results (Tables 2 & 3)

Compared against LVSM, GF, Gen3C, FlashWorld, and Gen3R on single- and two-view conditioned generation.

Table 2: Single-view 3D scene generation (averaged over First Frame and Bidirectional).

Table 3: Two-view 3D scene generation (averaged over Interpolation and Extrapolation).

PixWorld leads on perceptual and camera control metrics (especially strict AUC@5), indicating geometrically faithful trajectories.

WorldScore Benchmark (Table 4)

Table 4: WorldScore results.

PixWorld achieves highest average, camera control, 3D consistency, and photometric consistency.

Ablation Study (Table 5)

Table 5: Ablation of geometry perception loss on RealEstate10K (1-view).

Removing $\mathcal{L}_{\text{geo}}$ degrades PSNR by 1.13 dB and AUC@5 by ~12.5%, while generation quality metrics (I2V, I.Q.) hardly shift, confirming that 2D losses maintain individual frame plausibility but fail to enforce 3D geometry.

Theoretical and Practical Implications

• Removing the latent bottleneck: By supervising diffusion directly on rendered images, PixWorld eliminates information loss inherent to VAEs/RAEs and aligns the generative objective with actual 3D scene fidelity. This is particularly critical for reconstruction, which demands pixel-level precision.
• Unified paradigm: Partitioning views into clean/noisy subsets within a single forward pass provides a natural unification—the same model serves reconstruction (all-clean) and generation (mixed), without explicit task switching.
• Geometry-aware supervision: The proposed geometry perception loss leverages pretrained 3D foundation models to inject 3D structural signals beyond photometric consistency, solving depth drift and floater artifacts that 2D-only losses cannot address.
• Practical significance: PixWorld achieves state-of-the-art generation quality while matching reconstruction methods, demonstrating that a single model can replace two specialized pipelines, reducing engineering overhead and training cost.

Conclusion

PixWorld presents an end-to-end pixel-space diffusion framework that unifies 3D scene generation and reconstruction in a single model. By applying flow matching objectives directly on multi-view renderings of a 3D Gaussian representation, it eliminates the need for intermediate latent autoencoders and aligns optimization with 3D scene fidelity. A geometry perception loss provides additional 3D structural supervision. Extensive experiments on three benchmarks show that PixWorld outperforms prior latent-space generation methods and matches state-of-the-art reconstruction methods. Future directions include scaling to higher resolution, extending to more diverse scene types, and accelerating inference through distillation and quantization.