ReferTrack: Referring Then Tracking for Embodied Visual Tracking

Summary (Overview)

• Proposes ReferTrack, a vision-language-action (VLA) policy for embodied visual tracking (EVT) that formulates target identification as selecting an indexed bounding box from real-time detections, followed by trajectory prediction conditioned on this explicit image-grounded decision.
• Introduces Temporal-Viewpoint-Bbox Indicator (TVBI) tokens that inject geometric features of previously selected target bounding boxes into the visual history via a sliding-window queue, enabling target-specific motion cues without relying on abstract latent spaces.
• Co-trains on a custom Refer-QA dataset (1.3M samples) derived from SYNTH-PEDES, reinforcing the model’s ability to select the correct pedestrian from a discrete catalog using natural language instructions.
• Achieves state-of-the-art single-view performance on EVT-Bench: 89.4% SR on single-target tracking (STT), 73.3% SR on distracted tracking (DT), and 74.1% SR on ambiguity tracking (AT), matching or surpassing several multi-camera baselines on identification-heavy tasks.
• Deployed on Unitree Go2 and G1 robots, demonstrating robust sim-to-real transfer at 10.6 Hz with a single forward-facing camera.

Introduction and Theoretical Foundation

Embodied visual tracking (EVT) requires a mobile agent to continuously follow a specific target described in natural language, using only onboard vision. The task demands two tightly coupled capabilities: target identification (determining which pedestrian matches the instruction) and trajectory planning (generating collision-free motions to maintain a following distance of 1–3 meters).

Recent VLA models unify identification and planning via next-token prediction, often employing chain-of-thought (CoT) reasoning in abstract spatial latent spaces. However, these abstract latents are difficult to supervise and weakly aligned with explicit image-space detections, especially in crowded or ambiguous scenes. Early modular pipelines decouple perception (e.g., foundation models for detection) and control (e.g., imitation learning), but recognition errors propagate to planning.

ReferTrack addresses this by proposing a referring-then-tracking paradigm: target identification is cast as a constrained multiple-choice problem over indexed image-space bounding boxes (bboxes). This aligns with the natural grounding mechanisms of VLMs, enables direct supervision, and allows the use of large-scale referring data beyond scarce tracking trajectories.

Methodology

Problem Formulation

At each timestep $T$, given instruction $L$ and forward-view observations $O_{1:T} = \{O_1, \dots, O_T\}$, the agent predicts a trajectory $W_T = \{w_1, w_2, \dots\}$ where each waypoint $w_i = (x, y, \theta) \in \mathbb{R}^3$ is an egocentric displacement and heading change.

Model Architecture

ReferTrack uses a dual-branch architecture for navigation and question-answering. The forward-view history $O_{1:T}$ is encoded with a dual-encoder (SigLIP + DINOv2), followed by grid pooling to produce fine tokens $V_{\text{fine}} \in \mathbb{R}^{64 \times C}$ and coarse tokens $V_{\text{coarse}} \in \mathbb{R}^{4 \times C}$. A sliding window of $H$ frames is maintained, organized as $V_T = \{ V_{\text{coarse}}^{T-H}, \dots, V_{\text{coarse}}^{T-1}, V_{\text{fine}}^{T} \}$.

TVBI Tokens

Temporal-Viewpoint-Bbox Indicator (TVBI) tokens explicitly inject target bbox geometry into the visual history. For a normalized bbox $b_t \in [0,1]^4$ at frame $t$, the TVBI token is computed as:

$$E_{\text{TVBI}}(t) = E_{\text{TVI}}(t) + P_{\text{bbox}}(b_t)$$

where $E_{\text{TVI}}(t)$ is the original temporal-viewpoint indicator token (from NavFoM) and $P_{\text{bbox}}$ is a shared two-layer MLP projector. If the target is absent, $b_t = [0,0,0,0]$.

Candidate Catalog

At each timestep, a real-time detector (YOLO11+ByteTrack) extracts up to $K$ pedestrians from the forward view, organized into an indexed catalog:

$$C_T = \{ \langle \text{ped}_1 \rangle, \dots, \langle \text{ped}_K \rangle, \langle \text{NO EXIST} \rangle \}$$

Each entry is represented by a slot ID token followed by the bbox embedding $E_{\text{bbox}} = P_{\text{bbox}}(b_T^{(k)})$.

Refer-CoT and Trajectory Prediction

The model performs two sequential LLM forward passes:

$$\begin{aligned} E_{\text{refer}}^T &= \text{LLM}(L, C_T, E_{1:T}^V) \\ E_{A}^T &= \text{LLM}(L, C_T, E_{1:T}^V, E_{\text{refer}}^T) \\ W_T &= \text{ActionHead}(E_{A}^T) \end{aligned}$$

$E_{\text{refer}}^T \in \{ \langle \text{ped}_1 \rangle, \dots, \langle \text{ped}_K \rangle, \langle \text{NO EXIST} \rangle \}$ is a single-token classification decision. The selected bbox is pushed into a FIFO queue $\mathcal{Q}$ of capacity $H-1$ for future TVBI conditioning.

Training Objective

The model is fully fine-tuned (LLM parameters and auxiliary modules) with a weighted sum:

$$\mathcal{L} = \alpha \mathcal{L}_{\text{traj}} + \mathcal{L}_{\text{refer}} + \mathcal{L}_{\text{text}}, \quad \alpha = 10.$$

Trajectory loss (MSE over $M$ waypoints):

$$\mathcal{L}_{\text{traj}} = \sum_{i=1}^{M} \text{MSE}(\hat{w}_i, w_i^{\text{gt}}).$$

Refer loss (cross-entropy on target index):

$$\mathcal{L}_{\text{refer}} = -\log P(E_{\text{refer}}^{\text{gt},T} \mid L, C_T, E_{1:T}^V).$$

Training Data

• Navigation data: 1.3M expert tracking trajectories from EVT-Bench (Habitat 3.0 simulator).
• Refer-QA data: 1.3M referring samples synthesized from SYNTH-PEDES [18], using the same indexed-bbox selection interface.
• Co-trained at a 1:1 ratio via two-stage SFT: Stage 1 aligns vision projectors on general multimodal data; Stage 2 full fine-tunes on both sources.

Empirical Validation / Results

EVT-Bench Quantitative Results

Evaluation under the single forward-view setting on three splits: Single-Target Tracking (STT), Distracted Tracking (DT), and Ambiguity Tracking (AT). Metrics: Success Rate (SR), Tracking Rate (TR), Collision Rate (CR).

Table 1: Performance on EVT-Bench (key rows highlighted)

ReferTrack achieves best single-view results across all splits, despite being a compact 4B model with only SFT (no RL). On DT, it improves over TrackVLA++ by +6.8 SR and +13.0 TR; on AT, by +22.9 SR and +22.3 TR. It matches or exceeds several multi-camera baselines on identification-heavy splits.

Ablation Study

Table 2: Ablation on EVT-Bench DT (single forward-view)

Key findings:

• The oracle variant (ground-truth target bboxes, no Refer-CoT) achieves 81.5% SR, close to the expert policy (85.1% SR), showing that motion planning is not the bottleneck in distracted scenarios.
• Removing both Refer-CoT and TVBI drops SR to 55.7%, confirming that explicit image-space target selection is the primary source of robustness.
• Removing TVBI alone yields a smaller drop (70.4% SR), showing that Refer-CoT alone provides strong identification, while TVBI stabilizes tracking over time.

Real-World Deployment

• Platforms: Unitree Go2 (quadruped) and Unitree G1 (humanoid), each with a single forward-facing camera.
• Pipeline runs at ~10.6 Hz on a cloud GPU server; detection takes 12 ms per step.
• Qualitative results show successful target selection under multi-person interference and robust tracking even with partial occlusion (e.g., only lower body visible).

Theoretical and Practical Implications

• Theoretical contribution: Demonstrates that formulating target identification as an index-based bbox selection (rather than abstract spatial reasoning) provides a more direct and supervisable interface for grounding natural language instructions in embodied tracking. The TVBI mechanism effectively bridges discrete detection geometry with continuous trajectory prediction.
• Practical significance: Achieves state-of-the-art single-view EVT with a compact 4B model and only supervised fine-tuning, significantly reducing the need for costly reinforcement learning or multi-camera setups. This makes EVT more accessible for real-world robotic systems with limited compute and sensor budgets.
• Scalability: The indexed-catalog interface allows leveraging large-scale referring and grounding data (e.g., person ReID datasets, autonomous driving annotations) beyond scarce tracking trajectories, offering a scalable path to improve identification capability.
• Limitation: The oracle ablation reveals that target identification remains the primary bottleneck; future work could integrate stronger tracking or re-identification modules.

Conclusion

ReferTrack introduces a referring-then-tracking paradigm for embodied visual tracking, where target identification is cast as a single-token selection over an indexed set of detected bounding boxes. Key components include TVBI tokens for injecting target bbox history, a sliding-window queue for temporal motion cues, and co-training on a custom Refer-QA dataset. On EVT-Bench, ReferTrack achieves state-of-the-art single-view performance, matching or exceeding multi-camera methods on identification-heavy tasks, while using a compact 4B backbone and only SFT. Real-world deployments on legged and humanoid robots validate robust sim-to-real transfer. The results suggest that explicit image-space referring combined with temporal bbox memory can compensate for limited camera coverage and reduce reliance on costly RL fine-tuning.

Future directions: Integrating stronger person re-identification modules, extending to multi-target scenarios, and exploring online adaptation of the candidate catalog. Code is available at https://github.com/MedlarTea/referTrack.