Summary (Overview)

• Rest2Art is a novel framework that reconstructs articulated objects (with part-level geometry and joint parameters) from a single rest-state (closed) configuration, without requiring any observed motion.
• The method uses an explicit mesh as an intermediate representation to ground, verify, and fuse noisy outputs from multiple pretrained models (VLM, SAM3, video diffusion models).
• A key innovation is an iterative co-refinement loop between a Vision-Language Model (VLM) and a segmentation model (SAM3), where their disagreements drive mutual correction.
• For joint estimation without observed motion, the framework uses a video diffusion model to synthesize articulation hypotheses, then fits rigid-body joint models (revolute and prismatic) by minimizing reprojection errors, constrained by mesh geometry.
• The method achieves competitive or superior performance compared to both motion-observing reconstruction baselines and generation-based methods, while being input-agnostic (accepting multi-view captures, single-image 3D generation, or existing 3D assets).

---

Introduction and Theoretical Foundation

Background and Motivation

Humans can often infer how unfamiliar objects articulate from their appearance alone—e.g., anticipating how a closed cabinet's doors swing or drawers slide. However, this inference is not always reliable. Existing work on articulated object reconstruction focuses on well-constrained scenarios that require observing objects in multiple articulation states or with explicitly visible motion.

The Rest-State Problem

The paper introduces a rest-state formulation: reconstructing articulated objects from a single closed configuration. This is an inherently ill-posed setting where geometry, semantics, and motion priors must compensate for the absence of motion cues. Objects are most often encountered in their closed configuration in practice (e.g., online product photos, internet imagery, scene-level datasets like ScanNet and Replica).

Theoretical Foundation

The key insight is that pretrained models can supply rich semantic and motion priors for this task. However, individual model outputs are noisy and may disagree:

• A VLM may predict an incomplete part hierarchy
• A segmentation model may miss parts in certain views
• A video diffusion model may hallucinate nonexistent components

The paper addresses this by grounding all predictions in an explicit mesh, whose surface connectivity enables:

• Spatially consistent part boundaries
• Reliable occlusion reasoning
• Confidence-weighted aggregation of noisy, view-dependent predictions

---

Methodology

Problem Statement

Given a rest-state object mesh, the framework outputs part-wise meshes $\{ \mathcal{M}_k \}_{k=1}^{K}$ and joint parameters $\{ \varPsi_k \}_{k=1}^{K-1}$, where $K$ is the number of parts (automatically determined).

Joint Models:

• Prismatic joint: $\varPsi^p = \{ \mathbf{v} \in \mathbb{R}^3, d \in \mathbb{R} \}$ where $\mathbf{v}$ is the translation direction and $d$ is the scalar displacement.
• Revolute joint: $\varPsi^r = \{ \omega \in \mathbb{R}^3, \mathbf{q} \in \mathbb{R}^3, \theta \in \mathbb{R} \}$ where $\omega$ is the rotation axis direction, $\mathbf{q}$ is a point on the axis, and $\theta$ is the rotation angle.

Pipeline Overview

The pipeline consists of three main stages:

1. Part Segmentation (Sec. 3.2): Identifying movable parts through co-refinement of hierarchy predictions and segmentation masks
2. Articulation Estimation (Sec. 3.3): Synthesizing articulation videos and fitting rigid-body models
3. Mesh Completion (Sec. 3.4): Completing each part into a volumetric mesh for physics simulation

Part Segmentation

Co-Refinement of Hierarchy and Part Masks:

• A VLM proposes a candidate hierarchy tree $T$; SAM3 produces per-view part masks $M_v$
• Semantic prompts $\mathcal{S}(T) = \{s_1, s_2, \ldots, s_C\}$ are extracted (e.g., drawer, handle, door)
• The detected count $|M_{v,s}|$ is compared against the expected count $n(s, T)$:
  • Agreement: masks are cached as validated evidence
  • Excess detections: the VLM is re-queried with annotated images to revise $T$
  • Missing detections: decision is deferred, evidence aggregated across views

Confidence-Weighted Evidence Lifting: The face support set of mask $M_p^v$ is defined as:

$$\operatorname{supp}(M_p^v) = \{f \in \mathcal{F} \mid \exists \mathbf{x} \in M_p^v \text{ s.t. } \phi_v(\mathbf{x}) = f\} \tag{1}$$

Evidence is accumulated across selected views:

$$E(f, p) = \sum_{v \in \mathbf{V}_p} w_p^v \cdot c_p^v \cdot \mathbb{1}[f \in \operatorname{supp}(M_p^v)] \tag{2}$$

Each face is assigned the part with maximal evidence:

$$\ell(f) = \arg\max_p E(f, p) \tag{3}$$

Graph-Based Refinement: Labels are propagated from seed regions via multi-source shortest-path computation:

$$\ell(u) = \arg\min_p d_p(u), \quad u \in \mathcal{U} \tag{4}$$

where $d_p(u)$ is the shortest-path distance from $u$ to the nearest seed of part $p$.

Articulation Estimation

Articulation Video Synthesis:

• Part masks are overlaid with transparency onto the best view
• Wan2.2 [46] with VBVR LoRA [47] generates articulation videos
• Dense 2D point trajectories are extracted using CoTracker3 [17]
• Mesh provides metric depth for unprojecting frame-0 positions to 3D

Joint Parameter Fitting: For a revolute joint, each rest-pose anchor point $\mathbf{p}_0$ is rotated:

$$\mathbf{p}_t = R(\hat{\boldsymbol{\omega}}, \theta_t)(\mathbf{p}_0 - \mathbf{q}) + \mathbf{q} \tag{5}$$

For a prismatic joint:

$$\mathbf{p}_t = \mathbf{p}_0 + d_t \hat{\mathbf{v}} \tag{6}$$

The fitted axis is refined by:

• Penalizing axes that cause inter-part penetration
• Blending with the principal direction of the adjacent part boundary

Two videos with different random seeds are synthesized; if both agree on motion type, trajectory evidence is combined and refit jointly.

Mesh Completion

• Parts are converted to closed meshes by offsetting vertices inward along normals
• Boundary-aware normal smoothing prevents artifacts near cut edges
• Drawers receive explicit parametric inner trays for interior geometry

---

Empirical Validation / Results

Dataset and Implementation

• ACD (Articulated Containers Dataset) [15] from HSSD [19] and ABO [6], rendered in closed configuration
• MultiScan Dataset [32] for real-world robustness
• Meshes reconstructed with 2DGS [14]; GPT-5.2 [37] for hierarchy; SAM3 [3] for masks; Wan2.2 [46] + VBVR LoRA [47] for video; CoTracker3 [17] for trajectories

Part Segmentation Results

Baselines (PartField [28], Find3D [31], SAMesh [44]) struggle on rest-state furniture due to uniform colors and flush boundaries. The co-refinement loop ensures part counts and masks are cross-validated before evidence is committed to the mesh.

Joint Estimation Results

Table 2: Joint estimation comparison on the ACD dataset (Ang. = axis angular error (°), Pos. = position error (dm), Type = type accuracy (%), ↓ = lower is better, ↑ = higher is better):

Key findings:

• Our method achieves the highest type accuracy among methods applicable to rest-state inputs on both datasets
• Two-state methods (ArtGS, REArtGS, REArtGS++) struggle with synthesized multi-state inputs due to geometric inconsistencies
• Generative methods (Singapo, URDFormer) benefit from axis-aligned priors but require post-hoc scale alignment

Table 3: Joint estimation on MultiScan (real-world):

Our method outperforms baselines on type accuracy and angular error despite noisy real-world inputs.

Ablation Studies

Table 4: Co-refinement model selection (T Acc. = hierarchy tree accuracy, # Parts Acc. = per-part count accuracy):

The co-refinement loop yields significant improvements, particularly for the weaker VLM (Qwen3-VL), confirming that the two models provide complementary signals realized only through mutual correction.

Table 5: Mesh reconstruction backend ablation:

Video generation design: Mask overlay reduces hallucination from 23.1% to 11.5% and raises the accurate articulation rate from 69.2% to 80.8% on ACD-HSSD.

---

Theoretical and Practical Implications

Theoretical Significance

1. Ill-posed problem formulation: The paper formally establishes rest-state articulated object reconstruction as an ill-posed recovery problem, demonstrating that it can be solved through structured cross-validation of multiple pretrained models rather than requiring explicit motion observations.

2. Mesh as grounding mechanism: The explicit mesh representation serves as a "geometric anchor" that reconciles inconsistent outputs from different models. This is a principled approach to multi-modal fusion where disagreements between models are treated as signals for correction rather than noise to be averaged out.

3. Video diffusion as hypothesis generator: The framework reframes video generation models as sources of motion hypotheses rather than ground truth, with final parameters determined by geometric consistency against the mesh—a novel division of labor between generative and geometric methods.

Practical Implications

1. Input-agnostic design: The framework works with multi-view captures, single-image 3D generation, and existing 3D assets, making it applicable to diverse real-world scenarios including online product photos and scene scans.

2. Metric-scale fidelity: Unlike generation-based methods that produce normalized-scale outputs requiring post-hoc alignment, the reconstruction-based formulation preserves metric scale, which is critical for robotics, simulation, and spatial planning applications.

3. No part-count priors: The method automatically discovers the number of parts without requiring prior knowledge, unlike many existing methods that need this as input.

4. Digital twin creation: The output includes volumetric meshes suitable for physics simulators and URDF export, enabling interactive simulation and downstream robotics applications.

---

Conclusion

Main Takeaways

Rest2Art demonstrates that structured geometric verification can transform noisy, inconsistent outputs from multiple pretrained models into robust articulated object reconstructions from rest-state observations alone. The key principles are:

1. Cross-model verification (VLM + segmentation) resolves semantic ambiguities in closed configurations
2. Mesh-grounded evidence accumulation filters out unreliable predictions
3. Video-synthesized motion hypotheses + geometric fitting estimates physically plausible joint parameters without observed motion

Future Directions

The paper identifies two promising directions:

1. Extending to more complex kinematic structures beyond revolute and prismatic joints
2. Improving the physical plausibility of video generation models, which would further enhance the quality of synthesized articulation hypotheses

The framework's success suggests that the combination of generative models for hypothesis generation and geometric models for verification is a powerful paradigm for tackling other ill-posed 3D understanding problems.