ChronoReplay: One Answer Interface Is Not Enough for Streaming Spatial Intelligence

Summary (Overview)

• Core problem: Training a spatial VLM on a single answer interface (e.g., categorical options) causes interface lock-in—improvements concentrate on that interface while transfer to other readouts (metric, temporal) remains weak or degrades.
• Key proposal: ChronoReplay, a two-stage post-training recipe that (1) replays heterogeneous spatial supervision as chronologically ordered episodes with native-format targets, and (2) balances supervision across families via a weighted objective.
• Main results: Under identical observation access, a single ChronoReplay checkpoint improves all three benchmarks simultaneously: +13.41 points on OST-Bench, +7.44 on UCS-Bench, and +4.91 on OVO-S-Bench, achieving the highest three-benchmark mean (56.86) among compared supervision settings.
• Component ablations: Chronological replay beats a shifted-order control by ~2.41 OST points across paired seeds; family-balanced learning outperforms ordinary joint SFT and natural turn weighting; answer serialization redistributes gains across OST/UCS.
• Inference-time bonus: Adaptive frame selection adds +4.09 OVO-S points at lower visual cost (80.50 average frames vs. 128), separating observation access from training effects.

---

Introduction and Theoretical Foundation

Background and Motivation

A home robot following a user through an apartment may later be asked three different questions about the same walk: "Is the mirror on my left or right?" (categorical option), "How far is the sofa from here?" (metric value in meters), and "When did we pass the plant?" (timestamp). Answering all three from a single video stream requires associating objects across views, accumulating observer motion, and recovering relations in changing reference frames—all under causal (streaming) access.

Recent progress comes from two directions:

• Architectural work: visual memories, bounded caches, fast weights, maps, scene graphs.
• Data work: post-training VLMs on spatial instruction data.

Both are typically single-interface: a recipe supervises one answer format and reports only the benchmark that shares it. Even recipes that mix qualitative and metric questions train on single images, leaving untested whether one model can serve categorical, metric, and temporal queries over one causal history.

The Two Challenges

❶ Interface lock-in: Post-training on individual supervision families improves the dominant readout while transferring unevenly to others. Starting from Qwen3-VL-8B:

• Categorical-only training reaches 58.03 on OVO-S but lowers OST from 50.52 to 47.48.
• Trajectory-only training leaves OST nearly flat at 51.03.

❷ Target-history dependence: A metric or temporal target is valid only under the history that produced it. An observer moving forward after facing north vs. east shares the same recent observations and local displacement, yet a later question about a previously seen object has different answers. Supervision detached from its valid prefix is ambiguous.

Problem Formulation

At time $t$, a model receives the causal observation history $h_t = (o_1, \ldots, o_t)$. A spatial query $q$ arrives with a native answer interface $\kappa \in \mathcal{K}$ (categorical option set, metric value with units, or timestamp), and the VLM predicts:

$$z_t = f_\theta(h_t), \qquad \widehat{y}_t^{(\kappa)} = g_\theta(z_t, q, \kappa), \tag{1}$$

against the task-defined target $y_t^{(\kappa)} = A_\kappa(h_t, q) \in \mathcal{V}_\kappa$. The representation $z_t$ is reusable over a family of query–interface pairs when every native target in that family can be read out from the same $z_t$.

Two properties make this demanding:

1. Interfaces refine one another: a categorical target separates spatial regions, while a metric target resolves variation within a region—a representation keeping only the coarse partition cannot support the finer readout.
2. Origin dependence: histories with identical recent suffixes but different origins must map to different states.

---

Methodology

ChronoReplay consists of two stages (Figure 2 in the paper):

Stage I: Causal Replay Construction

Native-Format Targets. Each target is kept in the value space of its source interface—categorical options retain candidate structure, metric targets retain magnitude/sign/units/precision, temporal targets retain timestamps and event indices. The population risk for interface $\kappa$ is:

$$\mathcal{R}_\kappa(F_\theta, G_\kappa) = \mathbb{E}\left[-\log G_\kappa\left(Y^\kappa \mid Z, Q\right)\right]. \tag{2}$$

For a fixed representation and sufficiently expressive readout, the optimal risk equals the conditional entropy $H(Y^\kappa \mid Z, Q)$, so low risk requires $Z$ to retain the information needed to recover the native target.

Chronologically Aligned State Trajectories. Supervision is placed on the episode's video-time axis and replayed in original order, with each target attached to its legal causal prefix. An episode is represented as:

$$\mathcal{E}_i = (c_{ij}, q_{ij}, y_{ij}, \kappa_{ij})_{j=1}^{m_i}, \qquad c_{ij} \preceq c_{i,j+1}, \tag{3}$$

where $\preceq$ denotes causal prefix inclusion. Applying the VLM to successive prefixes induces an implicit state trajectory:

$$z_{ij} = F_\theta(c_{ij}), \qquad \mathcal{Z}_i = (z_{i1}, \ldots, z_{im_i}). \tag{4}$$

No map, memory module, or auxiliary state loss is added.

Stage II: Family-Balanced Learning

The preservation family $\mathcal{D}_P$ holds 33,180 single-turn examples (30,635 categorical, 2,545 numerical), while the trajectory family $\mathcal{D}_T$ holds 6,381 ordered episodes (45,006 supervised turns). To prevent one family from dominating, risks are normalized within each family:

$$\mathcal{L}_P = \frac{\sum_{r \in \mathcal{D}_P} w_r \ell_r^P}{\sum_{r \in \mathcal{D}_P} w_r}, \qquad \mathcal{L}_T = \frac{\sum_{(i,j) \in \mathcal{T}} \ell_{ij}^T}{|\mathcal{T}|}. \tag{5}$$

The balanced objective is:

$$\mathcal{L}_{\mathrm{replay}} = (1 - \lambda)\mathcal{L}_P + \lambda \mathcal{L}_T, \qquad \lambda = \frac{\alpha}{1 + \alpha}, \tag{6}$$

with $\alpha = 0.5$ in main experiments: trajectory receives $1/3$ of total mass, preservation receives $2/3$ (split equally between categorical and numerical targets). Answer-length normalization assigns each supervised turn an average token loss.

Experimental Setup

• Base model: Qwen3-VL-8B-Instruct (visual encoder frozen; language model and visual merger trained).
• Benchmarks: OVO-S-Bench (categorical), OST-Bench (agent state, visible info, spatial relations), UCS-Bench (8,114 timestamp-causal questions).
• Controls: SHIFTED (cyclic rotation of turn blocks), CANONICAL (typed value field targets), Uniform-81 (equal-cost frame control), single-family diagnostics.

---

Empirical Validation / Results

RQ1: Cross-Interface Transfer

OVO-S-Bench (categorical): ChronoReplay raises OVO-S from 48.98 to 53.89 under matched Uniform-128 access—numerically comparable to the published Qwen3-VL-235B-A22B reference (53.64). All twelve families improve; L4 rises from 35.16 to 40.17.

OST-Bench (continuous state): ChronoReplay improves OST from 50.52 to 63.93 (+13.41 points). The $\Delta$ row is positive in all fifteen subtypes. Largest gains:

UCS-Bench (timestamp-causal): ChronoReplay improves from 45.33 to 52.77 (+7.44 points), with positive transfer in all eight subcategories. Largest gains: quantity (+10.73), object trajectory (+8.46), observer trajectory (+7.38).

RQ2/RQ3: Diagnostic Analysis

Single-family supervision (Table 4a):

Mechanism controls (Table 4b):

• Format (CANONICAL): NATIVE is +2.92 higher on UCS; CANONICAL is +1.48 higher on OST—a crossed result showing answer format redistributes transfer.
• Order (SHIFTED): Chronological replay beats shifted order by 2.42 and 2.41 OST points across two paired seeds (mean 2.41).
• Mass (α = 1.0): α = 0.5 gives higher OVO-S–OST mean (61.20 vs. 60.92).
• Topology (4 GPUs): reproduces three-benchmark mean within 0.34 points.

Joint-training objectives: ChronoReplay (mean 56.86) beats ordinary token SFT (54.10) and natural turn weighting (55.57), with paired seed gains of $0.92 \pm 0.27$ (OVO-S), $1.43 \pm 0.19$ (OST), $1.17 \pm 0.25$ (UCS).

RQ4: Observation Access as an Orthogonal Factor

Training effect: +4.91 overall. Access effect: +4.09 overall, using 80.50 frames on average (vs. 128).

---

Theoretical and Practical Implications

Theoretical Contributions

1. Interface lock-in as an empirical pattern: The paper documents that single-family training transfers unevenly across spatial readouts—categorical-only training can degrade metric performance. This highlights that supervision format shapes what a shared representation preserves.

2. Information-theoretic grounding: Proposition 1 (Appendix A) shows that a low population risk $\mathcal{R}_\kappa$ requires the representation $Z$ to retain the information needed to recover the native target, with optimal risk equal to conditional entropy $H(Y^\kappa \mid Z, Q)$. Corollary 1 shows lossless re-encoding preserves this requirement—justifying the native/canonical control as a clean serialization intervention.

3. Chronological alignment as a data-construction principle: The origin-shift control (SHIFTED) demonstrates that target validity depends on the causal prefix; Proposition 2 characterizes the risk when changed context makes targets ambiguous.

Practical Implications

• One checkpoint, many interfaces: A single 8B model can serve categorical, metric, and temporal queries over one causal history—no new memory module or annotation required.
• Recipe simplicity: The method adds no architectural changes, only data organization (chronological episodes) and loss balancing (family-weighted objective).
• Inference-time adaptability: Adaptive frame selection provides an orthogonal, training-free gain at lower visual cost—useful for deployment with computational constraints.
• Benchmark transfer: The three-benchmark mean of 56.86 exceeds both single-family variants (50.74, 50.76) and the official base (48.28), demonstrating that balanced multi-interface training is feasible without sacrificing any endpoint.

---

Conclusion

ChronoReplay replays heterogeneous spatial supervision in source chronology, retains native targets, and improves OVO-S, OST, and UCS together under matched observation access. The key findings:

1. Interface lock-in is an empirical pattern: single-family training transfers unevenly across spatial readouts, with categorical-only training favoring OVO-S but degrading OST.
2. Chronological replay contributes a repeatable 2.41 OST-point gain over shifted-order control—a smaller but reliable component of the full 13.41-point improvement.
3. Answer format redistributes gains across UCS and OST rather than establishing a universally best encoding.
4. Observation access is orthogonal to training: adaptive frame selection adds +4.09 OVO-S points at lower visual cost.

The authors position ChronoReplay as a simple recipe for balanced cross-interface transfer without assigning its full gain to one component or claiming dominance on every spatial subcategory. Future directions include testing in other environments (e.g., outdoor scenes) and exploring whether the chronological alignment principle extends to other multi-interface learning settings.

Code: https://github.com/yihaohu0118/ChronoReplay
Project Homepage: https://yihaohu0118.github.io/ChronoReplay/