# Vision Pretraining for Dense Spatial Perception

> Masked boundary modeling forces self-supervised vision models to reconstruct geometric boundaries, achieving top depth accuracy with 7× fewer parameters.

- **Source:** [arXiv](https://arxiv.org/abs/2607.05247)
- **Published:** 2026-07-08
- **Permalink:** https://picx.dev/p/sygE2O
- **Whiteboard:** https://picx.dev/p/sygE2O/image

## Summary

## Summary (Overview)

- **New self-supervised paradigm**: The paper introduces **masked boundary modeling**, a method that dynamically learns sub‑pixel boundary representations and uses the discovered boundary‑bearing tokens as masked targets for dense visual token learning, without requiring any human annotations, external edge detectors, or pretrained backbones.
- **Categorical reparameterization of boundary fields**: To stabilise dense self‑distillation, continuous boundary fields are discretised into categorical distributions over bins. This makes the boundary objective compatible with centering and sharpening mechanisms that prevent collapse, and connects naturally to a‑contrario detection theory for parameter‑free validation.
- **LingBot‑Vision (1B ViT‑g)**: Scaled from scratch on a 161M‑image curated corpus, the model achieves the best NYU‑Depth v2 linear‑probing RMSE (0.296) among all compared models (including 7B DINOv3), matches or surpasses models up to 7× larger on dense tasks, and produces boundary tokens stable enough for video tracking via cosine similarity alone.
- **LingBot‑Depth 2.0**: By replacing only the encoder initialisation with LingBot‑Vision and scaling training data to 150M samples, the depth completion system attains leading performance on 14 benchmarks across block‑masked, sparse, and real‑sensor settings.
- **Distillation‑efficient**: The ViT‑L/B/S students inherit the teacher’s dense advantages, with the 0.3B student matching the 7B DINOv3’s NYUv2 accuracy under the same linear‑probing protocol (23× fewer parameters).

## Introduction and Theoretical Foundation

The paper argues that modern visual foundation models, built on objectives such as semantic self‑distillation, cross‑modal alignment, or masked image reconstruction, prioritise semantic invariance at the expense of fine‑grained spatial understanding. Boundaries and shape discontinuities – which reveal depth, motion, occlusion, and scene layout – are traditionally treated as outputs of perception, recovered by task‑specific heads under dedicated supervision. This “output‑centric” view ties them to expensive, ambiguous annotations, making them difficult to exploit in large‑scale pretraining.

The authors propose a **boundary‑centric view**: boundaries are not merely outputs but fundamental organising signals for learning dense representations. They build on dense, over‑parameterised field representations of boundary geometry (attraction‑field lines) that encode vectorised boundaries through pixel‑wise, distance‑transform‑like fields. These representations have been used in supervised line‑segment detection; the paper shows they can be bootstrapped from raw images during self‑supervised pretraining – usable boundary fields emerge even from an untrained model, given a sparse set of corner points (**Finding 1**).

The central tension: semantic self‑distillation may suppress fine‑grained boundary signals, while boundary modelling may disrupt invariance needed for strong semantics. The core contribution is a design that turns this tension into cooperation through **boundary‑forcing masked modelling**.

## Methodology

### Self‑Distillation Vision Pretraining (Baseline)

The work extends the teacher–student self‑distillation paradigm of DINO [9] and iBOT [28, 64]. The teacher’s weights are an exponential moving average (EMA) of the student: $\bar{\theta} \leftarrow \lambda \bar{\theta} + (1-\lambda)\theta$.

- **Image‑level (DINO)**: A projection head $h$ maps the class token to a distribution over $C$ prototypes. Teacher and student distributions are compared via cross‑entropy across mismatched view pairs:
  $$p_t = \text{softmax}\!\left(\frac{h_{\bar{\theta}}(z_{\text{cls}}) - c}{\tau_t}\right),\quad p_s = \text{softmax}\!\left(\frac{h_{\theta}(z_{\text{cls}})}{\tau_s}\right)$$
  $$\mathcal{L}_{\text{DINO}} = -p_{t}^{(x^{(1)})^\top} \log p_s^{(x^{(2)})}$$

- **Patch‑level (iBOT)**: A subset $\mathcal{M}$ of student patch tokens is masked and replaced by a mask token. The student predicts a distribution at each masked location, distilled from the teacher’s distribution at the corresponding unmasked token:
  $$\mathcal{L}_{\text{iBOT}} = -\frac{1}{|\mathcal{M}|}\sum_{i\in\mathcal{M}} q_i^{t\top} \log q_i^{s},\quad q_i = \text{softmax}\!\left(\frac{g(z_i^p) - c^{\text{iBOT}}}{\tau}\right)$$

  In the baseline, $\mathcal{M}$ is sampled randomly, independently of image content.

### Boundary‑Forcing Masked Modeling

The key idea: **let the teacher’s own boundary predictions decide what the student must reconstruct**.

1. **Boundary token set $\mathcal{B}$**: The teacher predicts a dense boundary field (see below). A token is a boundary token if a predicted line segment passes through its $P\times P$ patch (here $P=16$):
   $$\mathcal{B} = \{ i \in \{1,\dots,N\} : \text{a predicted boundary intersects patch}(i)\}$$

2. **Forced mask**: The student’s masked set is enlarged to $\mathcal{M}^+ = \mathcal{M} \cup \mathcal{B}$.

3. **Geometry‑routed supervision**:
   - All masked tokens follow the semantic objective $\mathcal{L}_{\text{iBOT}}$ (as in iBOT).
   - Boundary tokens additionally receive the **categorical boundary objective** $\mathcal{L}_{\text{bnd}}$.
   
   Thus boundary tokens carry a dual target: semantic reconstruction plus explicit geometric reconstruction. This resolves the tension because semantic supervision is weakest exactly where two regions meet, and the geometric target supplies structure precisely where it is needed.

### Categorical Reparameterization of Boundary Fields

The boundary field represents straight line segments (curved boundaries as chains of short segments) via dense per‑pixel attributes. For a pixel $p$ near a segment, the attribute vector is
$$\mathbf{a}(p) = (d_p,\; \theta_p,\; \phi_p^1,\; \phi_p^2)$$
where $d_p$ is the distance to the nearest segment, $\theta_p$ the direction toward it, and $\phi_p^1,\phi_p^2$ locate the segment’s endpoints as seen from $p$.

Direct regression of continuous fields in a teacher–student loop is prone to collapse. The paper discretises each channel into $K$ bins (we use $K=32$) and encodes the teacher value $a^c(p)$ as a soft categorical label:
$$\bar{y}^c_k(p) \propto \exp\!\left(-\frac{\delta^c(k, a^c(p))^2}{\tau_\ell}\right),\quad k=1,\dots,K$$
where $\delta^c$ is the distance from bin center $k$ to $a^c(p)$ (arc distance for circular $\theta$). The student predicts a distribution $\hat{y}^c(p)$ per channel, and the boundary objective is:
$$\mathcal{L}_{\text{bnd}} = -\frac{1}{|\mathcal{B}|}\sum_{p\in\mathcal{B}}\sum_c \bar{y}^c(p)^\top \log \hat{y}^c(p)$$

**Benefits**:
- The categorical form allows centering and sharpening (same as semantic branch), stabilising self‑distillation.
- The uniform distribution over bins corresponds to the no‑structure null hypothesis of a‑contrario detection. “No boundary” is thus a representable prediction, providing parameter‑free validation at no cost.

### Online Generation of Boundary Targets

Since the teacher is initialised from scratch, boundary targets must bootstrap from raw images. The process:

1. **Teacher predicts dense boundary field** over a global view.
2. **Corner points** are localised by a frozen, single‑block Vision Transformer (orders of magnitude smaller than the backbone).
3. **Candidate segments** are decoded via vote aggregation: because each segment is over‑determined by many support pixels, even noisy field values produce coherent segments anchored at corner points (Finding 1).
4. **A‑contrario validation** discards segment hypotheses that are likely under the uniform null (i.e., fewer than a threshold number of aligned pixels). The survivors are re‑rendered into a clean target field $\mathbf{a}(\cdot)$.

This pipeline runs end‑to‑end as batched GPU kernels. Only the re‑rendered validated field, not the raw prediction, supervises the student. No boundary annotations, edge detectors, or image‑classification pretraining are used.

### Full Training Objective

For each image, using global and local views:
$$\mathcal{L} = \mathcal{L}_{\text{DINO}} + \lambda_i \mathcal{L}_{\text{iBOT}} + \lambda_b \mathcal{L}_{\text{bnd}} + \lambda_k \mathcal{L}_{\text{KoLeo}}$$
with $\lambda_i = \lambda_b = 1$, $\lambda_k = 0.1$. The KoLeo regulariser spreads class‑token features within a batch [28]. All teacher quantities (semantic distributions and validated boundary targets) are produced under stop‑gradient; the teacher is updated via EMA.

### Scaling to LingBot‑Vision

- **Data**: 161M images curated from 2B raw web images (retrieval with DINOv2 ViT‑B encoder), comparable to DINOv2’s LVD‑142M, an order of magnitude smaller than DINOv3’s LVD‑1689M.
- **Model**: ViT‑g/16 with SwiGLU FFN, rotary position embeddings, four register tokens – roughly 1.1B parameters.
- **Training schedule (three stages)**:
  1. **Pretraining**: 300k iterations, global batch size 3072, images 256px (global crops) / 112px (local crops).
  2. **Gram anchoring**: 100k iterations to prevent dense‑feature quality degradation.
  3. **High‑resolution adaptation**: 100k iterations at 512px.
  
  Total samples processed: less than one third of DINOv3’s.

- **Distillation**: The frozen ViT‑g teacher is used to distill ViT‑L, ViT‑B, ViT‑S students (300k iterations each, plus 100k high‑resolution adaptation). No Gram anchoring phase needed with a frozen teacher.

## Empirical Validation / Results

### Proof of Concept on ImageNet‑1K (ViT‑L/16)

| Variant | IN‑1K k‑NN ↑ | NYUv2 (linear) δ₁ ↑ | RMSE ↓ |
|---|---|---|---|
| DINO+iBOT baseline | 81.6% | 81.4% | 0.474 |
| + categorical boundary target (geometric only) | 81.8% | 84.4% | 0.446 |
| + dual supervision (iBOT loss on boundary tokens) | 82.0% | 84.7% | 0.443 |
| + RoPE backbone (final recipe) | **82.4%** | **84.9%** | **0.440** |
| w/ boundary forcing, semantic target only | 81.4% | 81.2% | 0.481 |

The geometric target drives most of the dense improvement; dual supervision adds a small extra gain; forcing boundary tokens with a semantic target alone does not help.

### Dense Visual Tasks (Frozen Backbone, Linear Probe)

| Method | Param | NYUv2 ↓ | KITTI ↓ | ADE20k | Citysc. | VOC |
|---|---|---|---|---|---|---|
| DINOv3 | 7B/16 | 0.309 | 2.346 | 55.9 | 81.1 | 86.6 |
| V‑JEPA 2.1 ViT‑G | 2B/16 | 0.307 | 2.461 | 47.9 | 73.5 | 85.0 |
| LingBot‑Vision (Ours) | **1B/16** | **0.296** | **2.552** | 53.5 | 79.6 | **87.5** |
| DINOv2 | 1B/14 | 0.372 | 2.624 | 49.5 | 75.6 | 83.1 |
| DINOv3 ViT‑H+ | 0.8B/16 | 0.352 | 2.635 | 54.8 | 79.5 | 85.8 |
| AM‑RADIOv2.5 | 1B/14 | 0.340 | 2.918 | 53.0 | 78.4 | 85.4 |

LingBot‑Vision achieves the best NYUv2 RMSE overall, ahead of the 7B DINOv3, with 7× fewer parameters. On segmentation it matches distilled specialists.

### Video Object Segmentation (Training‑Free)

| Method | Param | DAVIS‑S | YT VOS‑S |
|---|---|---|---|
| DINOv3 | 7B/16 | 71.1 | 74.1 |
| LingBot‑Vision | 1B/16 | **70.0** | **73.5** |
| DINOv2 | 1B/14 | 63.9 | 65.6 |
| V‑JEPA 2.1 ViT‑g | 1B/16 | 68.1 | 72.3 |

On par with the 7B DINOv3, best among all sub‑7B models.

### ImageNet‑1K Classification (Frozen Features)

| Model | Linear Top‑1 | k‑NN Top‑1 |
|---|---|---|
| LingBot‑Vision | 86.32 | 83.39 |
| DINOv2 | 87.00 | 83.68 |
| DINOv3 | 87.87 | 85.68 |
| SigLIP2 | 87.33 | 84.75 |

LingBot‑Vision trails on image‑level recognition (expected, given the boundary orientation), while leading on dense tasks.

### Distilled Model Family

| Size | Model | IN‑1K Linear | NYU ↓ | KITTI ↓ | ADE | VOC | City |
|---|---|---|---|---|---|---|---|
| L | DINOv2 | 86.43 | 0.411 | 3.243 | 48.63 | 83.54 | 74.51 |
| L | DINOv3 | 87.31 | 0.351 | 2.643 | 55.00 | 88.11 | 79.28 |
| L | **LingBot‑Vision** | **86.38** | **0.310** | **2.574** | **52.75** | **87.40** | **78.75** |
| B | DINOv2 | 84.27 | 0.429 | 3.416 | 48.25 | 83.81 | 74.19 |
| B | DINOv3 | 84.79 | 0.371 | 2.826 | 51.74 | 88.20 | 77.30 |
| B | **LingBot‑Vision** | **85.05** | **0.339** | **2.793** | **51.44** | **87.10** | **77.17** |
| S | DINOv2 | 80.76 | 0.447 | 3.568 | 44.86 | 82.34 | 71.44 |
| S | DINOv3 | 80.37 | 0.405 | 2.851 | 46.51 | 84.30 | 73.38 |
| S | **LingBot‑Vision** | **82.22** | **0.383** | 3.784 | **47.01** | **84.17** | 64.34 |

The 0.3B ViT‑L student matches the 7B DINOv3’s NYUv2 RMSE (0.310 vs. 0.309) with ~23× fewer parameters.

### LingBot‑Depth 2.0 (Depth Completion)

**Key change**: Replace DINOv2 encoder with LingBot‑Vision; scale training data from 3M to 150M RGB‑D samples.

**Encoder initialisation study (MDM pipeline, same data)**:

| Init | Arch | DIODE‑In ↓ | DIODE‑Out ↓ | NYU (block) ↓ | NYU (sparse) ↓ |
|---|---|---|---|---|---|
| DINOv2 | ViT‑L/14 | 0.152/0.981 | 3.192/0.775 | 0.169/0.920 | 0.167/0.944 |
| DINOv3 | ViT‑L/16 | 0.114/0.988 | 3.065/0.735 | 0.154/0.922 | 0.167/0.949 |
| LingBot‑Vision | ViT‑L/16 | **0.094/0.990** | **2.771/0.794** | **0.145/0.930** | **0.167/0.952** |
| DINOv2 | ViT‑g/14 | 0.118/0.990 | 2.778/0.809 | 0.178/0.922 | 0.156/0.957 |
| LingBot‑Vision | ViT‑g/16 | **0.083/0.993** | 2.734/0.801 | **0.148/0.935** | **0.142/0.964** |

LingBot‑Vision initialisation gives the largest gains on the hardest block‑mask patterns.

**Data scaling effect**: As training data grows from 3M to 20M to 150M, both DINOv2 and LingBot‑Vision improve monotonically, but the gap widens: the LingBot‑Vision curve keeps improving while DINOv2 saturates (e.g., D102 at 150M: 0.795 vs. 0.755).

**Final depth completion benchmarks (selected)**:

| Method | DIODE‑In (block) ↓ | DIODE‑Out (

---

_Markdown view of https://picx.dev/p/sygE2O, served by PicX — AI-generated visual whiteboard summaries of research papers._