Vision Pretraining for Dense Spatial Perception

Summary (Overview)

• Core contribution: The paper proposes masked boundary modeling (MBM), a self-supervised pretraining paradigm that treats boundaries and shape discontinuities as native learning signals rather than downstream outputs, enabling Vision Transformers to learn both semantic and spatially structured representations from raw images alone.

• Key innovation: The method forces teacher-discovered boundary tokens into the student's masked set and routes masked tokens by geometry—boundary tokens receive categorical boundary-field supervision while other masked tokens follow standard semantic self-distillation (iBOT), turning the tension between semantic abstraction and geometric sensitivity into cooperation.

• Categorical reparameterization: Continuous boundary fields are reparameterized into per-pixel categorical distributions over discretized bins, which stabilizes dense self-distillation and connects to classical a-contrario detection theory, providing parameter-free boundary validation.

• Scaled results: LingBot-Vision (1B ViT-g/16) matches or surpasses foundation models up to 7× larger on dense spatial tasks—attaining the best NYU-Depth v2 accuracy (RMSE 0.296) of all compared models including the 7B DINOv3, while the 0.3B distilled student matches the 7B DINOv3's NYUv2 accuracy with ~23× fewer parameters.

• Downstream impact: Upgraded depth completion system (LingBot-Depth 2.0) achieves leading performance on 14 depth completion benchmarks, with the advantage of the LingBot-Vision encoder widening as downstream training data grows.

---

Introduction and Theoretical Foundation

Background and Motivation

Dense spatial perception—recovering structured, metric, and actionable representations from pixels—is essential for physical intelligence (robotics, embodied AI). Yet modern visual foundation models prioritize semantic invariance at the expense of fine-grained spatial understanding. The authors argue that boundaries and shape discontinuities are not merely outputs to be predicted but fundamental organizing signals for learning dense representations.

Key Theoretical Insights

The boundary field representation (from [58, 59]) is central to the method. Image boundaries are modeled as line segments, and each pixel near a segment stores a dense attribute vector:

$$\mathbf{a}(p) = \bigl(d_p, \theta_p, \phi_p^1, \phi_p^2\bigr)$$

where $d_p$ is the distance to the nearest segment, $\theta_p$ the direction toward it, and $\phi_p^1, \phi_p^2$ locate the segment's endpoints from pixel $p$. This representation is deliberately redundant: any single pixel in a segment's support region $S_\ell = \{p : d_p \leq \tau_d\}$ carries enough information to reconstruct the entire segment.

Finding 1 (Boundaries emerge from corner points): Given a sparse set of corner points, a boundary field sampled uniformly at random already decodes into corner-anchored line fragments; coherent segments emerge when the direction channel is guided by the parameter-free level-line orientation of the image gradient. This bootstrapping property enables boundary learning from scratch.

Why Line Segments Over Edge Maps?

• Edge pixels are isolated filter responses that cannot be statistically validated and are prone to texture noise.
• A line segment is a single hypothesis supported by many pixels, testable against a no-structure null hypothesis (uniform orientation distribution) via a-contrario theory.
• Segments carry endpoints and orientation, enabling corner-point anchoring.

---

Methodology

3.1 Self-Distillation Baseline

The method builds on the DINO/iBOT self-distillation paradigm. The teacher's weights are an EMA of the student:

$$\bar{\theta} \leftarrow \lambda \bar{\theta} + (1 - \lambda)\theta$$

Image-level distillation (DINO):

$$\mathbf{p}^{\mathrm{t}} = \operatorname{softmax}\left(\frac{h_{\bar{\theta}}(\mathbf{z}^{\mathrm{cls}}) - \mathbf{c}}{\tau_t}\right), \qquad \mathbf{p}^{\mathrm{s}} = \operatorname{softmax}\left(\frac{h_{\theta}(\mathbf{z}^{\mathrm{cls}})}{\tau_s}\right)$$ $$\mathcal{L}_{\mathrm{DINO}} = -\mathbf{p}^{\mathsf{t}}(\mathbf{x}^{(1)})^{\top} \log \mathbf{p}^{\mathsf{s}}(\mathbf{x}^{(2)})$$

Patch-level distillation (iBOT):

$$\mathcal{L}_{\mathrm{iBOT}} = -\frac{1}{|\mathcal{M}|} \sum_{i \in \mathcal{M}} \mathbf{q}_i^{\mathsf{t}^{\top}} \log \mathbf{q}_i^{\mathsf{s}}$$

3.2 Boundary-Forcing Masked Modeling

The key departure from random masking: the teacher's predicted boundaries determine the masked set.

Boundary token set (tokens whose patch intersects a predicted boundary):

$$\mathcal{B} = \{i \in \{1, \dots, N\}: \text{a predicted boundary intersects } \operatorname{patch}(i)\}$$

Boundary-forced mask:

$$\mathcal{M}^{+} = \mathcal{M} \cup \mathcal{B}$$

Boundary tokens receive dual supervision: the standard iBOT semantic objective plus the categorical boundary objective. This is well-posed precisely where semantic targets are ambiguous (at region boundaries).

3.3 Categorical Reparameterization of Boundary Fields

Direct regression of continuous boundary fields collapses in the teacher–student loop. Instead, each field channel is discretized into $K$ bins, and the teacher's validated value $a^c(p)$ is encoded as a soft categorical label:

$$\bar{y}_k^c(p) \propto \exp\left(-\delta^c(k, a^c(p))^2 / \tau_\ell\right), \quad k = 1, \dots, K$$

The boundary objective is cross-entropy over boundary positions:

$$\mathcal{L}_{\mathrm{bnd}} = -\frac{1}{|\mathcal{B}|} \sum_{p \in \mathcal{B}} \sum_{c} \bar{y}^c(p)^{\top} \log \hat{y}^c(p)$$

Key benefits:

• Inherits centering and sharpening mechanisms of semantic SSL
• "No boundary" = uniform distribution over bins (the a-contrario null hypothesis)
• Boundary validation is native to the representation

3.4 Online Generation of Boundary Targets

Four steps per iteration:

1. Teacher predicts a dense boundary field over a global view
2. Corner points localized by a frozen single-block ViT
3. Candidate segments decoded by vote aggregation (corner-point anchored)
4. A-contrario test discards unsupported segments; survivors re-rendered into a clean target field

3.5 Full Training Objective

$$\mathcal{L} = \mathcal{L}_{\mathrm{DINO}} + \lambda_{\mathrm{i}} \mathcal{L}_{\mathrm{iBOT}} + \lambda_{\mathrm{b}} \mathcal{L}_{\mathrm{bnd}} + \lambda_{\mathrm{k}} \mathcal{L}_{\mathrm{KoLeo}}$$

Proof of Concept (Table 1, ViT-L/16 on ImageNet-1K)

The categorical boundary target is the active ingredient; forcing boundaries with semantic-only reconstruction performs at or below baseline.

Scaling to LingBot-Vision

• Data: ~161M images curated from a 2B pool (vs. DINOv3's 1,689M)
• Model: ViT-g/16, ~1.1B parameters, SwiGLU FFN, RoPE, 4 register tokens
• Training: 300k pretraining + 100k Gram anchoring + 100k high-res (512px) adaptation, batch size 3,072 (less than ⅓ of DINOv3's samples)
• Distillation: Frozen ViT-g teacher distills into ViT-L (300M), ViT-B (86M), ViT-S (21M)

---

Empirical Validation / Results

Dense Tasks (Table 2)

Key results:

• NYUv2 depth: Best RMSE overall (0.296), beating 7B DINOv3 (0.309) and 2B V-JEPA 2.1 (0.307) with fewer parameters
• Segmentation: On par with distilled DINOv3 ViT-H+, leads VOC12 (87.5), trails ADE20k by 1.3 mIoU

Video Object Segmentation (Table 3)

Distilled Model Family (Table 5, selected)

The 0.3B ViT-L student achieves NYUv2 RMSE 0.310—on par with the 7B DINOv3 (0.309) with ~23× fewer parameters.

LingBot-Depth 2.0 (Encoder Initialization Study, Table 6)

The same masked depth modeling (MDM) pipeline trained from different encoder initializations shows LingBot-Vision is the strongest starting point at ViT-L across all benchmarks (e.g., DIODE-In: 0.094 vs. 0.152 for DINOv2).

Data scaling effect (Figure 8): As MDM training data grows from 3M → 20M → 150M samples, the gap between LingBot-Vision and DINOv2 initializations widens (D₁₀₂: 0.777 → 0.795 for LingBot-Vision vs. 0.752 → 0.755 for DINOv2), showing better pretraining compounds rather than washes out.

---

Theoretical and Practical Implications

Theoretical Contributions

1. Boundaries as native learning signals: The paper inverts the traditional relationship—boundaries are not outputs to be predicted but organizing signals for representation learning. The attraction field, made categorical, becomes the medium of pretraining.

2. Resolving the semantic–spatial tension: The geometry-routed masking (boundary tokens get geometric targets, others get semantic targets) provides well-posed objectives exactly where conventional masked modeling is weakest, allowing semantic and spatial representations to co-emerge.

3. A-contrario theory as native validation: The uniform distribution over categorical bins doubles as the no-structure null hypothesis, making boundary validation a free byproduct of the representation rather than a post-hoc step.

Practical Implications

• Training efficiency: LingBot-Vision uses a corpus an order of magnitude smaller than DINOv3's and less than ⅓ of its training samples, yet matches or surpasses it on dense tasks.
• Deployment flexibility: The distilled family (ViT-L/B/S) offers the best available depth accuracy in each size class with classification within a point of the best competitor.
• Robotics and embodied AI: Stable boundary tokens trackable through video by cosine similarity alone, and the LingBot-Depth 2.0 system with leading results on 14 benchmarks, demonstrate practical value for physical intelligence applications.

---

Conclusion

Main Takeaways

1. Boundary modeling, long confined to standalone contour or line-segment detection, can serve as a general and scalable pretraining principle for spatially structured visual representations.
2. The categorical reparameterization of boundary fields makes dense self-distillation stable and scalable, inheriting the machinery of modern SSL.
3. Better pretraining compounds with downstream data scale rather than washing out—a crucial property for foundation-model economics.

Future Directions

• The paper notes its data pipeline is deliberately simpler than DINOv2/DINOv3's; adopting more sophisticated curation could further improve results.
• The remaining gap to DINOv3 on image-level recognition suggests room for balancing boundary-oriented and invariance-oriented objectives.
• The success of boundary tokens as trackable entities hints at applications in video understanding and temporal consistency without explicit temporal supervision.

"Boundaries and shape discontinuities are not merely outputs to be predicted, but fundamental organizing signals for learning dense representations."