WildDet3D: Scaling Promptable 3D Detection in the Wild - Summary

Summary (Overview)

• Unified Geometry-Aware Architecture: Introduces WildDet3D, a single model that natively accepts text, 2D point, and 2D box prompts for open-vocabulary monocular 3D detection and can incorporate optional depth signals at inference time via a novel dual-encoder and depth fusion design.
• Large-Scale Open-World Dataset: Presents WildDet3D-Data, a massive dataset with over 1M images and 13.5K object categories (a 138× increase over Omni3D), constructed via a multi-model candidate generation and human/VLM verification pipeline.
• State-of-the-Art Performance: Achieves new SOTA across multiple benchmarks: 22.6/24.8 AP³ᴰ (text/box) on WildDet3D-Bench, 34.2/36.4 AP³ᴰ on Omni3D, and 40.3/48.9 ODS zero-shot on Argoverse 2 and ScanNet.
• Substantial Gains from Depth: Demonstrates that incorporating depth cues at inference yields an average gain of +20.7 AP, highlighting the model's ability to leverage richer geometric information.
• Real-World Versatility: Showcases practical applications including an iPhone app, AR integration with Meta Quest, robotic manipulation, and a VLM-agent for 3D referring expression localization.

Introduction and Theoretical Foundation

Understanding objects in 3D from a single image is fundamental for spatial intelligence in robotics, autonomous driving, and AR/VR. A practical, general-purpose monocular 3D detector must satisfy three key requirements not fully addressed by prior work:

1. Generalize in the wild to long-tailed, open-ended categories.
2. Support multiple prompt modalities (text, 2D points, 2D boxes) within a unified architecture for flexible interaction.
3. Leverage extra geometric cues (e.g., sparse LiDAR, partial depth) when available to improve 3D localization.

Existing methods specialize in either text-based querying (open-vocabulary) or fixed geometric inputs (oracle prompts), lacking a flexible, unified framework. Furthermore, progress is hampered by limited datasets covering narrow categories in controlled environments. This work addresses both the model and data bottlenecks.

The theoretical motivation centers on the choice of input modality for generalized 3D detection (Figure 2). Pure LiDAR lacks reliable height and full 6-DoF rotation cues. Pure RGB suffers from inherent scale and occlusion ambiguity. The proposed approach combines RGB with optional depth, retaining dense visual semantics for open-vocabulary recognition while using depth to resolve metric scale ambiguity when available.

Methodology

2.1 Dual-Vision Encoder

The architecture decouples semantic and geometric feature extraction to avoid trade-offs.

• Image Encoder: A ViT-H with SimpleFPN neck, initialized from SAM 3, provides high-resolution semantic features. The first 28 of 32 blocks are frozen during training.
• RGBD Encoder: A DINOv2 ViT-L/14 accepts 4-channel RGBD input (depth optional). It produces depth latents $Z_d \in \mathbb{R}^{C_d \times 49 \times 49}$ via a ConvStack neck. The first 21 of 24 blocks are frozen.
• Depth Fusion Module: Injects depth latents into image features via a ControlNet-style residual design: $$V' = V + \text{Conv}_{1\times1}(\text{LN}(Z^{\uparrow}_d))$$ where $Z^{\uparrow}_d$ is bilinearly interpolated depth latents, LN is LayerNorm, and the convolution is zero-initialized to preserve pretrained features at training start.

2.2 Promptable Detector

Unifies four prompt types using encoders adapted from SAM 3:

• Text Prompt: Category name encoded by a CLIP-style tokenizer and Transformer.
• Point Prompt: 2D pixel coordinates $(u, v)$ with positive/negative label.
• Box Prompt: 2D bounding box $(x_1, y_1, x_2, y_2)$.
• Exemplar Prompt: 2D box used as a visual exemplar to detect similar objects. Training uses per-prompt batching, aggregating all images containing a unique text category into a batch entry.

2.3 Deeply-Supervised 3D Detection Head

Lifts 2D queries to 3D boxes with deep supervision (loss applied at every decoder layer).

• Multi-Source Information Aggregation: Query features are enriched sequentially with:
  1. Camera Ray Features: Encoded using 8th-order real spherical harmonics: $\phi(r) = \text{RSH}_8(r / \|r\|) \in \mathbb{R}^{81}$.
  2. Depth Latents: Fused via cross-attention.
• 3D Box Parameterization: Predicts a 12D encoding: $$p_{3d} = (\underbrace{\Delta c_x, \Delta c_y}_{\text{center offset}}, \underbrace{\hat{d}}_{\text{log depth}}, \underbrace{\hat{w}, \hat{h}, \hat{l}}_{\text{log dims}}, \underbrace{r_1, ..., r_6}_{\text{rotation}})$$ where $\hat{d} = s_d \cdot \log(d)$ with $s_d=2.0$, and $(\hat{w}, \hat{h}, \hat{l}) = s_{\text{dim}} \cdot \log(w, h, l)$ with $s_{\text{dim}}=2.0$.
• Unambiguous Rotation Normalization: Applied to ground truth and predictions to resolve rotation-dimension ambiguity: (1) Dimension ordering ensures $w \le l$, (2) Yaw folding restricts angle to $[0, \pi)$.
• 3D Confidence Prediction: Predicts a quality score $s_{3D} \in [0,1]$ with soft target $q^* = \beta \cdot q_{\text{depth}} + (1-\beta) \cdot \text{IoU}_{3D}$ ($\beta=0.7$). Final score combines 2D and 3D confidence: $s = s_{2D} + \alpha \cdot s_{3D}$ ($\alpha=0.5$).

2.4 Multi-Task Learning

The overall training loss aggregates 3D detection and auxiliary losses:

$$L = \underbrace{L_{3D} + L_{\text{conf}}}_{\text{3D detection losses}} + \underbrace{L_{\text{geom}} + L_{2D}}_{\text{auxiliary losses}}$$

• 3D Regression Loss ($L_{3D}$): L1 loss on encoded 3D parameters.
• 3D Confidence Loss ($L_{\text{conf}}$): IoU-aware focal BCE loss.
• Auxiliary Geometry Loss ($L_{\text{geom}}$): Includes metric depth L1, SILog loss, affine-invariant point-map losses, confidence mask BCE, and camera ray MSE.
• Auxiliary 2D Detection Loss ($L_{2D}$): Includes IoU-aware classification, box regression, per-category presence, and One-to-Many (O2M) matching (each GT matched to top-$k=4$ predictions).

Ignore-Region Suppression: During training, negative classification loss is suppressed for predictions with 2D IoU > 0.5 against an object marked as IGNORE (lacks valid 3D GT), aligning training with evaluation.

3 WildDet3D-Data Construction Pipeline

A three-stage pipeline creates large-scale 3D annotations from existing 2D datasets (COCO, LVIS, Objects365, V3Det).

1. Candidate Generation: Five complementary methods generate candidate 3D boxes per 2D annotation:
   • 3D-MOOD, DetAny3D, SAM-3D, RANSAC-PCA, LabelAny3D.
   • Each candidate undergoes translation and rotation optimization.
2. Rule-Based Filtering: Applies geometric criteria (edge contact, occlusion, size ratio), VLM-based depicted object filter, and LLM-estimated size/geometry filters.
3. Candidate Selection: Two parallel paths:
   • Human Selection: Crowdsourced annotators select the best candidate and rate quality (good_fit, acceptable, unacceptable).
   • VLM Selection: A fine-tuned Molmo2 model scores candidates on six perceptual criteria (category, scale, translation, shape, rotation, tilt); keeps highest-scoring candidate if total score > 10.

Table 1: WildDet3D-Data Statistics

Table 2: Pipeline Validation on Human-Annotated Train Set

Empirical Validation / Results

4.1 Experimental Setup

• Datasets: WildDet3D-Bench (proposed, 700+ categories), Omni3D, zero-shot on Argoverse 2 (AV2) & ScanNet, real depth on Stereo4D.
• Metrics: AP³ᴰ (3D IoU matching for Omni3D, center-distance matching for in-the-wild), ODS (Open Detection Score) for zero-shot.
• Training: Three stages on 32 GPUs (12 epochs each on Omni3D, then Omni3D+Others+WildDet3D-Data, final fine-tuning).

4.2 In-the-Wild Evaluation on WildDet3D-Bench

Table 3: WildDet3D-Bench Evaluation Results

Key Findings:

1. WildDet3D trained on Omni3D alone outperforms 3D-MOOD by 3.0× (6.8 vs. 2.3 AP).
2. Adding WildDet3D-Data yields a 9.8× improvement over 3D-MOOD (22.6 vs. 2.3 AP).
3. Providing ground-truth depth at test time gives massive gains (+19.0 AP for the full model).
4. Improvements are consistent across all category frequency groups.

4.3 Results on Omni3D

Table 4: Omni3D Evaluation Results

Key Findings:

1. WildDet3D surpasses prior SOTA in both text (+5.8 AP over 3D-MOOD) and box (+2.0 AP over DetAny3D) prompt settings.
2. Achieves superior results with 10× fewer training epochs (12 vs. 120 for 3D-MOOD).
3. Depth input provides substantial gains, especially on indoor datasets with depth sensors.

4.4 Zero-Shot Evaluation

Table 5: Zero-Shot Evaluation on Argoverse 2 and ScanNet