# PixWorld: Unifying 3D Scene Generation and Reconstruction in Pixel Space

> PixWorld unifies 3D scene reconstruction and generation via pixel-space diffusion, outperforming prior latent-based methods on both tasks without autoencoders.

- **Source:** [arXiv](https://arxiv.org/abs/2607.05373)
- **Published:** 2026-07-08
- **Permalink:** https://picx.dev/p/NUnw5e
- **Whiteboard:** https://picx.dev/p/NUnw5e/image

## Summary

## Summary (Overview)

- **Unified pixel-space diffusion**: PixWorld reformulates 3D scene reconstruction and generation under a single pixel-space diffusion paradigm, eliminating the need for a pretrained VAE or RAE by supervising diffusion directly on multi-view renderings of a pixel-aligned 3D Gaussian representation.
- **Geometry perception loss**: Introduces a novel loss that aligns rendered views with ground-truth observations in the geometry-aware feature space of a frozen 3D foundation model, providing 3D structural supervision beyond 2D photometric and perceptual losses.
- **Superior performance**: A single PixWorld model outperforms prior latent-space generation methods on RealEstate10K, DL3DV-10K, and WorldScore, while matching or exceeding state-of-the-art feed-forward reconstruction methods on novel view synthesis.
- **Key insight**: Partitioning multi-view inputs into clean and noisy subsets within a two-stream diffusion transformer naturally unifies both tasks—clean views anchor reconstruction, noisy views are generated conditioned on the clean ones—with both outputs decoded into a shared 3D Gaussian representation.

## Introduction and Theoretical Foundation

### Background and Motivation
3D scene reconstruction and generation have historically been tackled by separate paradigms:
- **Reconstruction** is dominated by feed-forward methods that regress 3D representations directly from multi-view images (e.g., pixelsplat, MVSplat, DepthSplat).
- **Generation** has moved from per-scene score distillation to latent-space diffusion, where a diffusion model operates in the compressed feature space of a VAE or RAE, followed by a learned decoder to 3D Gaussians.

Recent work (Gen3R) attempted to unify both tasks in a latent-space framework, but inherits fundamental drawbacks:
- The diffusion objective is defined on intermediate latent features, not on the underlying 3D representation.
- Both branches suffer from information loss due to latent encoding/decoding.
- Requires pretraining of a VAE or RAE, adding computational cost.

### Theoretical Basis: Pixel-Space Diffusion
The paper builds on the observation that latent diffusion inserts an autoencoding bottleneck between the diffusion variable and the final image supervision. Instead, following JiT, they adopt **image prediction in pixel space**:

Given a clean image $x \in \mathbb{R}^{H \times W \times 3}$, noise $\epsilon \sim \mathcal{N}(0,I)$, and timestep $t \in [0,1]$, the noisy input is $x_t = t x + (1-t)\epsilon$. The denoiser is parameterized as an image predictor:

$$f_\theta: (x_t, t, c) \mapsto \hat{x}, \quad (1)$$

where $c$ is conditional information. The predicted image is converted to a velocity field $\hat{v} = (\hat{x} - x_t)/(1-t)$, yielding the flow-matching objective:

$$\mathcal{L}_{\text{FM}} = \mathbb{E}_{x,\epsilon,t}\left[ \|\hat{v} - v\|_2^2 \right] = \mathbb{E}_{x,\epsilon,t}\left[ \frac{\|\hat{x} - x\|_2^2}{(1-t)^2} \right], \quad (2)$$

where $v = x - \epsilon$ is the ground-truth velocity.

**Key insight**: By keeping the diffusion variable and the rendered output in the same RGB domain, the diffusion objective can be supervised directly on **rendered images** from a 3D representation, aligning optimization with 3D scene fidelity rather than an intermediate latent target.

## Methodology

### Task Formulation
Given a scene with $N$ posed views, partition indices into a clean subset $\Omega_c$ and a noisy subset $\Omega_n$ such that $\Omega_c \cap \Omega_n = \emptyset$, $|\Omega_c| \geq 1$. When $\Omega_n = \emptyset$, the task reduces to multi-view reconstruction. Otherwise, the model predicts the noisy views conditioned on the clean ones (generation). An optional text prompt $y$ may condition the scene.

Mixed multi-view input is:
$$\tilde{I}^n_t = \begin{cases} I^n & n \in \Omega_c \\ t I^n + (1-t)\epsilon^n & n \in \Omega_n \end{cases}, \quad \epsilon^n \sim \mathcal{N}(0,I), \quad (4)$$

### Two-Stream Diffusion Transformer
A 24-layer DiT with SD3-style MMDiT design has two parallel streams:
- **Clean stream**: processes conditioning views ($\Omega_c$), receives time embedding at $t=1$.
- **Noise stream**: processes noisy views ($\Omega_n$), receives the sampled timestep $t$.

The two streams share cross-attention to text tokens and a single timestep embedder. Camera parameters are injected via PRoPE. The output heads predict per-pixel depth and 3D Gaussian attributes for every view (clean and noisy).

### 3D Gaussian Decoding and Rendering
Gaussian centers are obtained by unprojecting each pixel using its predicted depth: $\mu^n_p = \Pi^{-1}(p, \hat{d}^n_p, T^n)$. The scene representation $\hat{G}$ is rendered via differentiable renderer $\mathcal{R}: (\hat{G}, T^n) \mapsto \bar{I}^n$.

### Losses
**Rendering and depth objectives**:

$$\mathcal{L}_{\text{render}} = \frac{1}{|\Omega_c|} \sum_{n \in \Omega_c} \|\bar{I}^n - I^n\|_2^2 \;+\; \frac{\mathbb{1}[|\Omega_n| > 0]}{|\Omega_n|} \sum_{n \in \Omega_n} \|\bar{v}^n - v^n\|_2^2 \;+\; \lambda_{\text{lpips}} \mathcal{L}_{\text{lpips}}, \quad (6)$$

where $\bar{v}^n = (\bar{I}^n - \tilde{I}^n_t)/(1-t)$ and $v^n = (I^n - \tilde{I}^n_t)/(1-t)$. The LPIPS term is active only when $t > t_{\text{th}}$.

Depth loss (using pseudo-depth from DA3):

$$\mathcal{L}_{\text{depth}} = \frac{1}{N} \sum_{n=1}^N \rho\left( \log \hat{D}^n - \log D^{\star n} \right), \quad (7)$$

where $\rho(\cdot)$ is Huber loss.

**Geometry perception loss**:
Using a frozen 3D foundation model $\Psi$ (VGGT or $\pi_3$), extract multi-view geometric features:

$$\bar{H} = \Psi(\bar{I}, T), \quad H^\star = \Psi(I, T), \quad (8)$$

Then minimize average per-location cosine distance:

$$\mathcal{L}_{\text{geo}} = \mathbb{1}[t > t_{\text{th}}] \frac{1}{N H' W'} \sum_{n=1}^N \sum_p \left[ 1 - \frac{\langle \bar{h}^n_p, h^{\star n}_p \rangle}{\|\bar{h}^n_p\|_2 \|h^{\star n}_p\|_2} \right], \quad (9)$$

Gradients are stopped on the reference branch $H^\star$; only the rendered branch backpropagates.

Overall objective: $\mathcal{L} = \mathcal{L}_{\text{render}} + \lambda_{\text{depth}} \mathcal{L}_{\text{depth}} + \lambda_{\text{geo}} \mathcal{L}_{\text{geo}}$ with $\lambda_{\text{depth}}=1.0$, $\lambda_{\text{lpips}}=\lambda_{\text{geo}}=0.1$, and $t_{\text{th}}=0.3$.

## Empirical Validation / Results

### Training Details
- PixWorld has ~1.04B parameters, trained from scratch on RealEstate10K + DL3DV-10K (~67K scenes), augmented with 10M single images from BLIP-3o.
- Optimized with AdamW for ~200K steps at 336×448 resolution on 32 NVIDIA A800 GPUs.
- Frozen 3D critic: $\pi_3$ (Wang et al., 2025d).

### Reconstruction Results (Table 1)
PixWorld is compared against MVSplat, DepthSplat, AnySplat, and YoNoSplat on novel view synthesis from 4/8 input views.

**Table 1: Novel-view synthesis on RealEstate10K and DL3DV-10K.**
| Dataset | Views | Method | PSNR ↑ | SSIM ↑ | LPIPS ↓ |
|---------|-------|--------|--------|--------|---------|
| RealEstate10K | 4 | YoNoSplat | 25.86 | 0.841 | 0.143 |
| | 4 | **PixWorld** | **26.21** | **0.844** | **0.138** |
| | 8 | YoNoSplat | 28.35 | 0.889 | 0.107 |
| | 8 | **PixWorld** | **28.58** | **0.892** | **0.101** |
| DL3DV-10K | 4 | YoNoSplat | 22.89 | 0.710 | 0.228 |
| | 4 | **PixWorld** | **23.18** | **0.714** | **0.226** |
| | 8 | YoNoSplat | 21.92 | 0.678 | 0.262 |
| | 8 | **PixWorld** | **22.46** | **0.681** | **0.257** |

PixWorld achieves best PSNR and LPIPS across all settings, demonstrating strong cross-view consistency.

### Generation Results (Tables 2 & 3)
Compared against LVSM, GF, Gen3C, FlashWorld, and Gen3R on single- and two-view conditioned generation.

**Table 2: Single-view 3D scene generation (averaged over First Frame and Bidirectional).**
| Dataset | Method | PSNR ↑ | LPIPS ↓ | AUC@5 ↑ |
|---------|--------|--------|---------|---------|
| RealEstate10K | Gen3R | 17.59 | 0.382 | 0.147 |
| | FlashWorld | 16.51 | 0.403 | 0.546 |
| | **PixWorld** | **18.88** | **0.325** | **0.614** |
| DL3DV-10K | Gen3R | 15.75 | 0.495 | 0.117 |
| | FlashWorld | 15.42 | 0.461 | 0.420 |
| | **PixWorld** | **16.50** | **0.449** | **0.485** |

**Table 3: Two-view 3D scene generation (averaged over Interpolation and Extrapolation).**
| Dataset | Method | PSNR ↑ | SSIM ↑ | LPIPS ↓ | AUC@5 ↑ |
|---------|--------|--------|--------|---------|---------|
| RealEstate10K | LVSM | 23.61 | 0.819 | 0.215 | 0.611 |
| | FlashWorld | 21.48 | 0.770 | 0.257 | 0.637 |
| | **PixWorld** | **23.54** | **0.815** | **0.210** | **0.649** |
| DL3DV-10K | LVSM | 19.18 | 0.589 | 0.343 | 0.374 |
| | FlashWorld | 18.27 | 0.562 | 0.359 | 0.514 |
| | **PixWorld** | **19.37** | **0.594** | **0.340** | **0.534** |

PixWorld leads on perceptual and camera control metrics (especially strict AUC@5), indicating geometrically faithful trajectories.

### WorldScore Benchmark (Table 4)
**Table 4: WorldScore results.**
| Method | Camera Control | Object Control | Content Alignment | 3D Consist. | Photo. Consist. | Style Consist. | Subj. Qual. | Average |
|--------|----------------|----------------|-------------------|-------------|-----------------|----------------|-------------|---------|
| Wan-2.1 | 23.53 | 40.32 | 45.44 | 78.74 | 78.36 | 77.18 | 59.38 | 57.56 |
| WonderJourney | 84.60 | 37.10 | 35.54 | 80.60 | 79.03 | 62.82 | 66.56 | 63.75 |
| LucidDreamer | 88.93 | 41.18 | 75.00 | 90.37 | 90.20 | 48.10 | 58.99 | 70.40 |
| FlashWorld | 84.43 | 50.28 | 56.54 | 85.87 | 86.72 | 79.36 | 52.75 | 70.85 |
| **PixWorld** | **91.08** | 46.25 | 55.27 | **91.39** | **93.84** | 67.11 | 52.36 | **71.04** |

PixWorld achieves highest average, camera control, 3D consistency, and photometric consistency.

### Ablation Study (Table 5)
**Table 5: Ablation of geometry perception loss on RealEstate10K (1-view).**
| Variant | PSNR ↑ | SSIM ↑ | LPIPS ↓ | AUC@5 ↑ |
|---------|--------|--------|---------|---------|
| Full model | 19.12 | 0.717 | 0.310 | 0.642 |
| w/o Geometry Perception | 17.99 | 0.612 | 0.332 | 0.562 |

Removing $\mathcal{L}_{\text{geo}}$ degrades PSNR by 1.13 dB and AUC@5 by ~12.5%, while generation quality metrics (I2V, I.Q.) hardly shift, confirming that 2D losses maintain individual frame plausibility but fail to enforce 3D geometry.

## Theoretical and Practical Implications

- **Removing the latent bottleneck**: By supervising diffusion directly on rendered images, PixWorld eliminates information loss inherent to VAEs/RAEs and aligns the generative objective with actual 3D scene fidelity. This is particularly critical for reconstruction, which demands pixel-level precision.
- **Unified paradigm**: Partitioning views into clean/noisy subsets within a single forward pass provides a natural unification—the same model serves reconstruction (all-clean) and generation (mixed), without explicit task switching.
- **Geometry-aware supervision**: The proposed geometry perception loss leverages pretrained 3D foundation models to inject 3D structural signals beyond photometric consistency, solving depth drift and floater artifacts that 2D-only losses cannot address.
- **Practical significance**: PixWorld achieves state-of-the-art generation quality while matching reconstruction methods, demonstrating that a single model can replace two specialized pipelines, reducing engineering overhead and training cost.

## Conclusion

PixWorld presents an end-to-end pixel-space diffusion framework that unifies 3D scene generation and reconstruction in a single model. By applying flow matching objectives directly on multi-view renderings of a 3D Gaussian representation, it eliminates the need for intermediate latent autoencoders and aligns optimization with 3D scene fidelity. A geometry perception loss provides additional 3D structural supervision. Extensive experiments on three benchmarks show that PixWorld outperforms prior latent-space generation methods and matches state-of-the-art reconstruction methods. Future directions include scaling to higher resolution, extending to more diverse scene types, and accelerating inference through distillation and quantization.

---

_Markdown view of https://picx.dev/p/NUnw5e, served by PicX — AI-generated visual whiteboard summaries of research papers._