Summary (Overview)

• InfiniSplat v2 is a feed-forward 3D Gaussian Splatting (3DGS) reconstruction method that reconstructs Gaussian scenes from arbitrary input views for novel view rendering, achieving state-of-the-art perceptual quality.
• The method decouples geometry prediction from Gaussian attribute prediction by freezing a geometry branch (based on Depth Anything 3) while training a separate attribute branch, preventing geometric degradation during attribute learning.
• Three key improvements are introduced: (1) a dual-branch architecture with frozen geometry, (2) scale-aligned GT camera training to avoid pose-error-induced misalignment, and (3) large-angle target view supervision to better constrain Gaussian shapes.
• Across six datasets (DL3DV, ETH3D, NRGBD, Mip-NeRF 360, RE10K, DTU), the method achieves the lowest mean LPIPS and DISTS in both pose-conditioned and pose-free settings, with LPIPS reduced by 25.8% and DISTS by 32.9% relative to the strongest baselines in the pose-conditioned setting.
• A key finding: opacity learned with only image reconstruction supervision selectively suppresses geometrically inconsistent points, pruning 51.8% of inconsistent points while retaining 93.3% of surface-consistent points.

---

Introduction and Theoretical Foundation

Background

Image-based 3D scene reconstruction aims to recover renderable 3D scenes from input images, supporting applications such as simulation, content creation, and autonomous driving. 3D Gaussian Splatting (3DGS) (Kerbl et al., 2023) enables efficient rendering through explicit scene representation. Feed-forward methods directly predict Gaussian scenes from input images, eliminating per-scene optimization at inference time.

Motivation

Pretrained multi-view geometry models (e.g., DA3, DUSt3R, VGGT) predict depth maps and cameras, which can be back-projected into point clouds. However, these point clouds often contain:

• Spatially isolated clusters
• Clusters inconsistent across views

Existing feed-forward methods (e.g., DA3-GS) that combine geometry prediction with Gaussian attribute prediction suffer from blur, ghosting, and missing detail under large viewpoint changes.

Theoretical Basis

The scene is represented as M Gaussians, with each center $\pmb{\mu}_m$ separated from its remaining attributes $\pmb{\xi}_m$:

$$\mathcal{G} = \{(\pmb{\mu}_m, \pmb{\xi}_m) \}_{m=1}^{M}, \qquad \pmb{\xi}_m = (\alpha_m, \mathbf{s}_m, \mathbf{q}_m, \mathbf{c}_m). \tag{1}$$

Where $\pmb{\mu}_m \in \mathbb{R}^3$ is the position, and $\xi_m$ contains opacity $\alpha_m$, three-axis scale $\mathbf{s}_m$, rotation $\mathbf{q}_m$, and zeroth-order spherical-harmonic (SH) color $\mathbf{c}_m$. For camera $\Pi = (\mathbf{K}, \mathbf{P})$ with intrinsics K and camera-to-world pose P, the Gaussian renderer produces $\hat{\mathbf{I}} = \mathcal{R}(\mathcal{G}; \Pi)$.

---

Methodology

1. Dual-Branch Architecture for Attribute Learning

Two independently parameterized branches are initialized from the same pretrained DA3 weights:

$$\begin{array}{r} (\widehat{D}_{\mathcal{C}}, \widehat{\Pi}_{\mathcal{C}}) = \Phi_{\mathrm{geo}}(\mathcal{I}_{\mathcal{C}}; \mathbf{Q}_{\mathcal{C}}), \\ \mathbf{F}_{\mathcal{C}} = \Phi_{\mathrm{attr}}(\mathcal{I}_{\mathcal{C}}; \mathbf{Q}_{\mathcal{C}}). \end{array} \tag{2}$$

• Frozen geometry branch $\Phi_{\mathrm{geo}}$: predicts depths and cameras (including depth/camera heads)
• Trainable attribute branch $\Phi_{\mathrm{attr}}$: produces multi-layer patch features; only the last K Transformer blocks are fine-tuned
• A DPT Gaussian decoder maps features to per-pixel attributes

2. Scale-Aligned Training

To avoid camera-induced residuals from predicted pose errors, GT cameras are used for training-time back-projection with scale alignment:

Mean Radial Distance Normalization sets the reconstruction scale:

$$s_{\mathrm{norm}} = \frac{1}{|\mathcal{V}_{\mathrm{pts}}|} \sum_{j \in \mathcal{V}_{\mathrm{pts}}} \|\mathbf{x}_j\|_2, \qquad \bar{D}_i = \frac{\widehat{D}_i}{s_{\mathrm{norm}}}. \tag{3}$$

A Umeyama similarity transform fits GT input-camera centers to normalized predicted camera centers, producing scale $s_{\mathcal{C}}$. GT translations become $\widetilde{\mathbf{t}}_v^{\star} = s_{\mathcal{C}} \mathbf{t}_v^{\star}$. Back-projection uses scale-aligned GT cameras:

$$\boldsymbol{\mu}_{i,\mathbf{u}}^{\mathrm{cand}} = \bar{D}_i(\mathbf{u}) \widetilde{\mathbf{R}}_i^{\star}(\mathbf{K}_i^{\star})^{-1} \mathbf{u} + \widetilde{\mathbf{t}}_i^{\star}. \tag{4}$$

Redundant points are merged using the voxel-based strategy of AnySplat and WorldMirror 2.0.

3. Random Large-Angle View Supervision

Gaussians with opacity below threshold $\tau_\alpha$ are pruned. Target views are sampled with a bias toward large angular separation from input views:

$$a(v, \mathcal{C}) = \min_{i \in \mathcal{C}} \arccos(\mathbf{f}_v^{\top} \mathbf{f}_i). \tag{5}$$

Supervision is applied to renderings from both input views and sampled target views, with GT-depth warp masks restricting loss to valid regions.

4. Learned Rendering-Feedback Refinement

A learned refiner updates Gaussian attributes using input-view rendering feedback:

$$\begin{array}{l} \mathbf{g}^t = \text{RenderGrad}(\mathcal{G}^t; \mathcal{I}_{\mathcal{F}}, \Pi_{\mathcal{F}}^{\mathrm{fb}}), \\ \mathbf{h}^t = E(\mathcal{G}^t, \text{sg}(\mathbf{g}^t), \mathbf{f}). \end{array} \tag{6}$$

A shared recurrent KNN Transformer U aggregates features and predicts attribute residuals:

$$\begin{array}{c} (\Delta \boldsymbol{\xi}_m^t, \mathbf{b}_m^{t+1}) = U(\boldsymbol{\xi}_m^t, \mathbf{h}_m^t, \mathbf{b}_m^t), \\ \boldsymbol{\xi}_m^{t+1} = \text{Apply}(\boldsymbol{\xi}_m^t, \Delta \boldsymbol{\xi}_m^t), \qquad \boldsymbol{\mu}_m^{t+1} = \boldsymbol{\mu}_m^0. \end{array} \tag{7}$$

Gaussian centers and counts remain fixed; rendering gradients serve as inputs to learned updates rather than direct gradient steps.

5. Training Stages

• Stage I: Both branches receive input RGB and GT pose conditions; input and sampled target renderings supervised with equally weighted masked $L_1$ and LPIPS losses; only the attribute branch's last K Transformer blocks and Gaussian decoder are updated.
• Stage II: Reconstructor frozen; feature projection and refiner trained with input-view losses providing feedback.

---

Empirical Validation / Results

Quantitative Results (Table 1, selected rows)

Key improvements in pose-conditioned setting:

• LPIPS reduced by 25.8% over ReSplat (strongest LPIPS baseline)
• DISTS reduced by 32.9% over WorldMirror 2.0 (strongest DISTS baseline)

Ablation Study (Table 2)

Key ablation findings:

• Freezing geometry (vs. fine-tuning) reduces LPIPS by 16.6% (ETH3D) and 23.2% (DTU)
• GT-camera backprojection reduces DTU LPIPS by 36.1% relative to predicted cameras
• Fine-tuning last K=10 attribute blocks reduces DTU LPIPS by 20.8% vs. K=0

Opacity and Geometric Consistency

Using GT-depth labels on held-out Hypersim validation scenes:

• Pruning at threshold 0.01 removes 51.8% of geometrically inconsistent points
• Only 7.5% of all points are removed
• 93.3% of surface-consistent points are retained

This demonstrates that opacity learned with only image reconstruction supervision selectively suppresses geometrically inconsistent points across views.

---

Theoretical and Practical Implications

Theoretical Implications

1. Decoupling geometry from attributes: Jointly optimizing geometry and Gaussian attributes leads to geometric degradation, as positional updates compensate for attribute errors. Fixing centers blocks this pathway and forces the network to learn better attributes.

2. Camera alignment matters: Training with predicted poses introduces pose errors that misguide the network under large-angle supervision. Scale-aligned GT cameras provide a consistent reference frame.

3. Opacity as geometric consistency indicator: With only image reconstruction supervision, the network learns to assign low opacity to geometrically inconsistent points, effectively performing implicit outlier rejection.

Practical Implications

1. Single model, multiple settings: The same trained model works in both pose-conditioned and pose-free settings, simplifying deployment.

2. Robustness to imperfect geometry: The method produces high-quality renderings even when predicted point clouds contain numerous outliers (flying points), making it practical for real-world applications.

3. Efficient refinement: The learned rendering-feedback refinement uses only input views during both training and inference, with fixed centers allowing reuse of the spatial KNN graph across steps.

---

Conclusion

InfiniSplat v2 improves feed-forward Gaussian reconstruction by:

1. Freezing the geometry branch to decouple geometry prediction from attribute prediction
2. Using scale-aligned GT cameras during training to avoid pose-error-induced misalignment
3. Adopting large-angle target view supervision to better constrain Gaussian shapes

The method achieves state-of-the-art mean LPIPS and DISTS across six datasets in both pose-conditioned and pose-free settings. Ablations on ETH3D and DTU support the design choices of fixed centers and attribute adaptation under reconstruction supervision. A notable finding is that learned opacity preferentially suppresses geometrically inconsistent points, providing an implicit mechanism for handling imperfect predicted geometry.

Future directions (implied): Extending the approach to longer sequences, larger scenes, and more diverse real-world data; further analysis of the opacity-based outlier suppression mechanism; and integration with more advanced geometry foundation models as they emerge.