Summary (Overview)

• TimeLens2 is a generalist video temporal grounding MLLM that treats evidence as a first-class interval set throughout supervision and optimization, enabling single- and multi-span grounding across diverse video lengths, domains, query forms, and viewpoints.
• TimeLens2-93K is a novel data construction pipeline that replaces brittle global annotation with a staged process: caption-derived proposals, dual-agent independent localization, cross-agent consensus, semantic verification, and boundary-focused local refinement, yielding reliable multi-span supervision for long videos.
• The temporal Wasserstein reward computes the exact one-dimensional 1-Wasserstein distance between uniform distributions over merged predicted and target interval supports, providing dense, matching-free, fragmentation-invariant feedback for non-overlapping and unequal-cardinality predictions.
• TimeLens2-2B, 4B, and 8B variants outperform all size-matched baselines across seven benchmarks, with the 4B model surpassing even the 397B Qwen3.5-397B-A17B by 7.5 average mIoU points. The 2B, 4B, and 8B variants improve over their Qwen3-VL backbones by 14.2, 13.0, and 18.1 mIoU points, respectively.
• The temporal Wasserstein reward rescues 75.8% of all-zero-tIoU GRPO groups, providing ranked learning signals where overlap-based rewards provide none, and improves mIoU by 0.7 points over tIoU alone.

Introduction and Theoretical Foundation

Video multimodal large language models (MLLMs) can describe what happens in a video but rarely identify when the supporting evidence occurs. Temporal grounding is the video analogue of citation: it makes outputs traceable by requiring precise, potentially disjoint evidence intervals rather than fluent responses alone. The authors study generalist temporal grounding, where a single MLLM with a unified output interface localizes single- or multi-interval evidence across video lengths, domains, query forms, and viewpoints.

Two structural mismatches hinder progress:

1. Supervision mismatch: Temporal evidence is a set of intervals, yet long-video labels are often produced as a single global annotation decision. One-pass annotators may confuse similar occurrences, miss repeated evidence, or produce imprecise boundaries. Long-video annotation is an evidence-verification problem, not merely a scaling problem.

2. Optimization mismatch: Next-token prediction learns timestamp generation syntax but lacks an explicit interval-level objective. Reinforcement learning with temporal IoU (tIoU) better aligns training with evaluation but assigns zero to all disjoint predictions, leaving near misses indistinguishable from distant errors. Multi-span targets require one-to-one matching (e.g., NGIoU) which is fragile under fragmentation or unequal cardinality.

TimeLens2 treats evidence as a first-class interval set throughout training. The central idea is to make evidence verifiable when constructing supervision and geometry-aware when optimizing predictions.

Methodology

TimeLens2-93K Data Construction Pipeline

The pipeline separates candidate construction from label determination through six stages:

1. Long and diverse video source pool: 34,867 YouTube videos stratified across five duration ranges (under 1 minute to over 1 hour) and 15 visual domains.

2. Hierarchical temporally grounded captions: Videos are partitioned into semantically coherent clips (20–60s) using PySceneDetect. Qwen3-VL-235B-A22B generates both global captions (subjects, setting, actions, temporal progression) and segment captions (visually verifiable actions within bounded intervals).

3. Segment-aware query synthesis: Kimi-K2.5 jointly generates a non-redundant declarative query and selects all segments whose captions support it, forming a coarse proposal $P(q) \subseteq C(v)$.

4. Dual-agent local grounding: For each query-proposal pair $(q, c_i)$, Qwen3-VL-30B-A3B and TimeLens-8B independently return one or more intervals or an empty set, providing two views for consensus.

5. Cross-agent consensus and semantic verification:

   • Temporal consensus: $\text{IoU}_{\text{set}}(A, B) = \frac{|\text{merge}(A) \cap \text{merge}(B)|}{|\text{merge}(A) \cup \text{merge}(B)|} > 0.9$
   • Semantic verification: Query–clip cosine similarity $\geq 0.5$ using Qwen3-VL-Embedding

6. Boundary-focused local refinement: For each retained boundary, Qwen3-VL-235B-A22B observes a $\pm 3$s neighborhood and predicts a refined transition point. Adjacent spans separated by $\leq 1$s are merged.

The resulting corpus contains 23,793 videos and 93,232 grounding instances, including 12,091 with multiple supporting intervals.

TimeLens2 Model Training

Long-context supervised grounding: Fine-tune Qwen3-VL on long-context examples from TimeLens2-93K, TimeLens-100K, and Ego4D-NLQ. Retaining full-video context forces the model to isolate sparse evidence among competing events. Instruction and response-format diversity (27 instructions, 28 response specifications) encourages protocol-invariant interval semantics.

Rollout-guided hard-sample mining: Generate off-policy completions from the SFT checkpoint, using mean tIoU as an empirical estimate of model competence. Examples with lower scores receive higher sampling weights during GRPO.

Temporal Wasserstein reward: Each sampled completion is parsed into a predicted interval set $\hat{Y}$ and scored by:

$$R(\hat{Y}, Y) = R_{\text{tIoU}}(\hat{Y}, Y) + R_{\text{TW}}(\hat{Y}, Y) - \mathbb{1}_{\text{invalid}}$$

where:

• $R_{\text{tIoU}}(\hat{Y}, Y) = \frac{|\text{merge}(\hat{Y}) \cap \text{merge}(Y)|}{|\text{merge}(\hat{Y}) \cup \text{merge}(Y)|}$ measures support overlap
• $R_{\text{TW}}$ computes the 1-Wasserstein distance between uniform distributions over merged interval supports

Each non-empty interval set $A$ is lifted to a uniform temporal distribution:

$$\mu_A(t) = \frac{\mathbb{1}\{t \in \text{merge}(A)\}}{\ell(A)}$$

where $\ell(A) = |\text{merge}(A)|$. The 1-Wasserstein distance in one-dimensional time has the exact CDF form:

$$W(\hat{Y}, Y) = W_1(\mu_{\hat{Y}}, \mu_Y) = \int_{\mathbb{R}} |F_{\hat{Y}}(t) - F_Y(t)| \, dt$$

This is converted to a scale-normalized similarity:

$$R_{\text{TW}}(\hat{Y}, Y) = \exp\left(-\frac{W(\hat{Y}, Y)}{|\text{merge}(Y)| + \epsilon}\right)$$

The reward is matching-free, fragmentation-invariant, and provides dense, graded feedback for non-overlapping and unequal-cardinality predictions.

Empirical Validation / Results

Main Results (Table 1)

TimeLens2-4B outperforms all prior models (including 397B Qwen3.5) on six of seven benchmarks. TimeLens2-8B surpasses all prior methods in R1@0.5 on all seven benchmarks and mIoU on six (trailing Gemini 2.5 Pro on QVHighlights by only 0.2 points).

Data and Label Curation Ablation (Tables 2 & 3)

• TimeLens2-93K scaling: The first 5% of data delivers the largest gain (avg. mIoU from 34.7 to 42.8); full corpus reaches 45.8. Gains concentrate on harder settings (VUE-TR-V2, Ego4D-NLQ).
• Label curation cascade: Progressive curation (consensus + verification + refinement) improves mIoU from 42.0 (raw) to 45.8 with only 93.2K labels (one-eighth of raw 735.4K). Boundary refinement is the largest single stage (+1.7 mIoU).

Reward Design Ablation (Table 6)

Uniform support produces the best average, improving every benchmark. Temporal Wasserstein outperforms matched NGIoU by 0.6 mIoU, with larger gaps on multi-interval benchmarks (VUE-TR +1.4, VUE-TR-V2 +1.0).

Diagnostic: Zero-Overlap Misses (Table 8)

$R_{\text{TW}}$ recovers positive overlap for 21.9% of near misses, but only 5.7% of far misses, confirming distance-aware signal.

Diagnostic: Group-Relative Signal (Table 9)

Adding $R_{\text{TW}}$ reduces constant-reward groups from 13.8% to 3.6% and rescues 75.8% of all-zero-tIoU groups, turning silent groups into ranked training signals.

Theoretical and Practical Implications

• Theoretical: The paper demonstrates that treating temporal evidence as an interval set throughout supervision and optimization is more principled than treating it as a single moment or a sequence of independent boundaries. The temporal Wasserstein reward provides a theoretically grounded (optimal transport) approach to reward shaping for set-valued predictions, avoiding the correspondence issues of matching-based methods like NGIoU.
• Practical: TimeLens2 shows that compact models (2B–8B) can outperform extremely large models (up to 397B) on temporal grounding, suggesting that data quality and reward design are more important than scale for this task. The staged data construction pipeline (TimeLens2-93K) provides a scalable methodology for producing reliable multi-span supervision for long videos, which is a major bottleneck for video understanding. The instruction/response format diversity technique improves cross-benchmark transfer without additional data.
• The improvements on question-form grounding (MomentSeeker) from purely declarative training data indicate that the model learns to search for evidence rather than imitate query templates, suggesting that temporal grounding capability transfers across query types.

Conclusion

TimeLens2 addresses two core challenges in video temporal grounding: constructing trustworthy evidence labels for long videos, and providing geometry-aware optimization for interval-set predictions. The TimeLens2-93K pipeline replaces brittle global annotation with a staged evidence-verification process, while the temporal Wasserstein reward supplies dense, matching-free, fragmentation-invariant feedback. Across seven benchmarks covering long-video, multi-span, question-form, and egocentric settings, TimeLens2-2B, 4B, and 8B models outperform much larger open-source models. The key insight is that treating evidence consistently as an interval set from supervision through optimization turns temporal grounding from a specialized retrieval task into a native, auditable capability of generalist video MLLMs. Future work could extend the approach to more complex evidence structures (e.g., hierarchical or conditional intervals) and integrate temporal grounding with other video understanding tasks.