Summary (Overview)

• Decoupled 4D reconstruction: StudioRecon separately reconstructs static background and dynamic humans from as few as 4 low-overlap cameras (neighboring views ~90° apart), using complementary priors: video diffusion models for background, parametric body models (SMPL) for humans.
• Sparse-to-dense view synthesis: Hundreds of novel views are synthesized via a camera-controlled video diffusion model (GEN3C) to provide dense supervision for background Gaussians, compensating for sparse input coverage.
• Geometry-driven multi-view human estimation: Cross-view identity association combines spatial proximity and pose similarity (97.8% accuracy), and 3D keypoints are triangulated for robust SMPL parameter fitting, enabling accurate human initialization even under occlusions.
• Motion-adaptive diffusion enhancement: A single-step diffusion model with exponential moving average (EMA) blending and per-pixel confidence from optical flow warping harmonizes composited backgrounds and humans, removing artifacts while ensuring temporal consistency (23% reduction in warp error).
• State-of-the-art results: Outperforms Dyn-3DGS, MonoFusion, and STG on four real-world datasets (EgoHumans, Harmony4D, Mobile Stage, SelfCap) across both 360° and 180° camera configurations, with large LPIPS improvements (e.g., 0.251 vs. 0.551 on Legoassemble).

Introduction and Theoretical Foundation

High-fidelity 4D human capture is essential for entertainment, sports broadcasting, and virtual production. Professional volumetric systems require dense camera arrays (dozens to hundreds of cameras) in controlled environments. However, in real-world scenarios such as gymnasiums, homes, or sports venues, only a handful of uncalibrated, low-overlap cameras are available, often with multiple interacting people and frequent occlusions—referred to as in-the-wild studio capture.

Existing 4D Gaussian Splatting methods (e.g., 4DGS, STG) assume dense camera coverage with substantial view overlap. Sparse-view methods (e.g., DNGaussian, DropGaussian) still rely on shared visibility between neighboring cameras for correspondence matching. MonoFusion addresses low-overlap scenarios but produces noticeable artifacts in under-observed regions because errors in humans and backgrounds become entangled in a shared representation. Camera-controlled video diffusion models (e.g., GEN3C, ReCamMaster) can synthesize novel views but lack geometric consistency for moving humans.

Key insight: Backgrounds and humans benefit from different priors. Video diffusion models can plausibly synthesize static scenes from novel viewpoints by leveraging semantic priors from large-scale data. For humans, however, parametric body models like SMPL provide strong geometric priors that constrain shape and articulation, enabling robust reconstruction even from sparse observations. This motivates a decoupled reconstruction approach, avoiding the entanglement that causes artifacts in joint methods.

Methodology

The pipeline consists of four stages:

1. Sparse-to-Dense View Synthesis (Section 3.1)

• Preprocessing: Feed-forward 3D reconstruction (Pi³) provides initial point clouds, depth maps, and camera poses from the first frame. Monocular depth (MoGe) is aligned for subsequent frames. Humans are segmented using SAM3.
• Camera trajectory generation: Spherical linear interpolation (SLERP) for rotation and linear interpolation for translation produce L novel camera poses (L=481 in experiments).
• Novel view synthesis: A video diffusion model (GEN3C) takes first-frame images, depths, and target poses to synthesize dense novel views. Human masks are obtained for each synthesized view.

2. Multi-View Human Pose Estimation (Section 3.2)

• Per-view detection: Monocular pose estimator (CoMotion) provides SMPL pose $\boldsymbol{\theta}$, shape $\boldsymbol{\beta}$, 2D keypoints, and within-view tracking.
• Cross-view identity association: Affinity between detections $a$ and $b$: $$A(a,b) = w_p \cdot \exp\left(-\frac{\|\mathbf{p}_a - \mathbf{p}_b\|}{\sigma_p}\right) + w_\theta \cdot \exp\left(-\frac{\|\boldsymbol{\theta}_a - \boldsymbol{\theta}_b\|}{\sigma_\theta}\right)$$ where $\mathbf{p}$ is the 3D world position (unprojected pelvis), $w_p=0.9$, $w_\theta=0.1$. Hungarian algorithm assigns correspondences.
• 3D pose triangulation: Robust triangulation with Huber loss for each joint $k$: $$\mathbf{P}_k^w = \arg\min_{\mathbf{X}} \sum_{n=1}^N \rho \left\| \mathbf{x}_{n,k} - \pi_n(\mathbf{X}) \right\|$$
• SMPL parameter fitting (3D-to-3D optimization): $$\mathcal{L}_{fit} = \sum_j \| s^* \mathbf{J}_j^s(\boldsymbol{\theta},\boldsymbol{\beta}) - \mathbf{J}_j^w \|^2 + \sum_i \| s^* \mathbf{V}_i^s(\boldsymbol{\theta},\boldsymbol{\beta}) - \mathbf{V}_i^w \|^2 + \lambda_\beta \|\boldsymbol{\beta}\|^2$$ where $s^*$ is the scale factor from bone-length matching.

3. Decoupled Gaussian Reconstruction (Section 3.3)

• Background reconstruction: Optimizes static Gaussians at $t=1$ using synthesized views, with masked human regions: $$\mathcal{L}_{bg} = (1-M) \odot (\mathcal{L}_1 + \lambda_s \mathcal{L}_{SSIM} + \lambda_l \mathcal{L}_{LPIPS}) + \lambda_d \mathcal{L}_{den}$$ Higher loss weights near original cameras. Iterative refinement adds views with height variation.
• Human reconstruction: Each person is represented by canonical Gaussians initialized on SMPL. Deformation uses Linear Blend Skinning (LBS): $$\boldsymbol{\mu}^t = \sum_{b=1}^B w_b(\boldsymbol{\mu}^c) \cdot \mathbf{G}_b^t \cdot \boldsymbol{\mu}^c$$ Joint optimization of Gaussian attributes and SMPL pose using photometric, silhouette, density, and temporal smoothness losses.

4. Recursive Enhancement Module (Section 3.4)

• Motion-adaptive consistency injection: Blends rendered input $\hat{\mathbf{I}}^t$ with warped previous outputs using EMA: $$\tilde{\mathbf{I}}^t = (1 - w_{total}) \cdot \hat{\mathbf{I}}^t + w_{total} \cdot \sum_{i=0}^{K-1} \bar{w}_i \cdot c_i \cdot \text{warp}(\mathbf{O}^{t-1-i}, \mathbf{F}^{t \to t-1-i})$$ where $c_i = \max(0, 1 - e_i/\tau_e)$ is per-pixel confidence based on warp error, $\bar{w}_i$ are normalized EMA weights. Static regions get full injection; moving regions get minimal injection to prevent ghosting.

Empirical Validation / Results

Setup: Four real-world datasets: EgoHumans, Harmony4D (360° coverage), Mobile Stage, SelfCap (180° coverage). 4 training cameras spaced ~90° apart, 4 evaluation cameras at intermediate angles. Each sequence: 121 frames, 1-3 people.

Quantitative comparison:

Table 1: 360° scenes (best in red, second best in yellow)

Table 2: 180° scenes

Ablation studies:

• Cross-view association (Table 3): Hybrid (spatial + pose) achieves 97.8% accuracy vs. 93.3% (spatial only) and 81.4% (pose only).
• Pipeline components (Table 4): View synthesis (+2.4 PSNR, 36% LPIPS reduction), enhancement (+27% LPIPS reduction).
• Motion-adaptive injection (Table 5): Reduces warp error by 23% (0.119→0.092), improving temporal coherence.
• Seen/unseen fidelity (Table 6): Seen regions maintain 2.3 dB higher PSNR and better temporal consistency.
• Sensitivity to noise (Table 7): Robust to mask erosion, SMPL noise, and keypoint noise (≤0.2 dB PSNR drop).

Applications: Novel camera trajectories (dolly zoom, oscillating) and human replacement are demonstrated.

Theoretical and Practical Implications

• Theoretical: The work demonstrates that decoupling scene components with tailored priors (diffusion for static background, body models for articulated humans) is superior to joint reconstruction under extreme sparse-view, low-overlap conditions. The hybrid cross-view association using spatial and pose cues provides a principled approach for multi-person tracking in disjoint cameras.
• Practical: StudioRecon enables high-fidelity 4D capture with only 4 cameras, making it feasible for real-world venues (gyms, homes, studios) where dense camera arrays are impractical. The pipeline supports downstream applications like novel trajectory rendering and actor replacement without re-capturing. The recursive enhancement module is general and can improve baseline methods (LPIPS reduction of 20-31% on Dyn-3DGS, MonoFusion, STG).
• Limitations: Fine details (faces, hands) remain challenging; dynamic objects (basketballs, props) are not reconstructed; shadows baked into the static background do not follow human motion.

Conclusion

StudioRecon is a pipeline for high-fidelity 4D human-scene reconstruction from sparse, low-overlap cameras. The key insight is that backgrounds and humans benefit from different priors: diffusion models synthesize dense supervision for backgrounds, while parametric body models (SMPL) constrain human geometry. The decoupled approach, combined with geometry-driven multi-view human estimation and motion-adaptive diffusion enhancement, achieves state-of-the-art results on four real-world datasets. The work brings high-fidelity 4D human-scene capture from minimal cameras closer to practical deployment. Future directions include incorporating hand-held object models and dynamic shadow synthesis.