CollabVR: Collaborative Video Reasoning with Vision-Language and Video Generation Models

Summary (Overview)

• Core Contribution: Introduces CollabVR, a closed-loop framework that couples a Vision-Language Model (VLM) as a planner/verifier with a Video Generation Model (VGM) at a step-by-step granularity for video reasoning tasks.
• Key Mechanism: The VLM progressively plans the immediate next action and verifies each short clip generated by the VGM, diagnosing failures and folding corrective feedback into the next prompt. This addresses two key VGM failure modes: long-horizon drift and mid-clip execution errors.
• Empirical Results: CollabVR consistently improves performance over single-inference, Pass@k sampling, and prior test-time scaling methods (like VideoTPO) on two benchmarks (Gen-ViRe and VBVR-Bench) for both open-source (VBVR-Wan2.2) and closed-source (Veo 3.1) VGMs, at matched computational cost.
• Orthogonality: The framework's test-time reasoning supervision is orthogonal and stackable with reasoning-oriented fine-tuning of VGMs, yielding further improvements on a fine-tuned model.
• Validation: A human-annotated benchmark confirms the reliability of the VLM's supervisory decisions (plan depth, verification, and prompt evolution) aligned with expert judgment.

Introduction and Theoretical Foundation

Recent "Thinking with Video" paradigms use Video Generation Models (VGMs) to produce temporally coherent Chain-of-Frames as reasoning artifacts. However, VGMs exhibit complementary strengths and weaknesses compared to VLMs:

• VLMs excel at logical reasoning, planning, and abstract inference but are weak at direct visual simulation.
• VGMs excel at short-horizon visual simulation, detail, and physical coherence but are weak at abstract reasoning and long-range consistency.

VGMs trained for perceptual quality exhibit two recurring failure modes in goal-directed tasks:

1. Overloaded-prompt failure (Long-horizon drift): When a single prompt specifies a multi-step task, the VGM collapses it into one short rollout, deviating from the intended trajectory due to lack of planning capacity.
2. Execution failure (Mid-clip errors): Localized errors within a single clip (e.g., agent crossing a wall, object identity loss) propagate and contaminate the entire trajectory.

The root cause is the absence of an explicit, corrigible reasoning process on top of the VGM's strong short-horizon visual prior. A VLM can naturally serve as a reasoning supervisor. However, upfront planning commits before generation, and post-hoc critiques over whole videos intervene too late. CollabVR proposes step-level coupling to catch failures immediately and repair them.

Methodology

CollabVR is a closed-loop, training-free framework that treats video reasoning as a construction problem. The correct trajectory is assembled stepwise through VLM planning and VGM execution.

Problem Formulation

A task is specified by an input image $I_0$ and a task prompt $q$. The goal is to produce a video $V$ that realizes the reasoning. The framework uses:

• A VLM-based planner/verifier $\pi$ (queried as $\pi_{\text{plan}}$ and $\pi_{\text{verify}}$).
• An image-to-video generator $g$ that maps a conditioning frame $f$ and an action prompt $a_t$ to a short clip $c_t$.

The system maintains:

• $f$: the latest conditioning frame (initially $I_0$).
• $H$: the history of accepted clips.

The output is the concatenation of accepted clips: $V = c_1 \oplus \cdots \oplus c_N$.

The core algorithm is detailed in Algorithm 1 (provided in the paper). The loop iterates up to a maximum planning step $N_{\text{max}}$ with a per-step attempt budget $M$.

Core Modules

1. VLM-Driven Progressive Planning (Module 1): Addresses overloaded-prompt failure. Instead of pre-planning all steps upfront, the VLM adaptively decides the step count and plans only the immediate next action $a_t$ conditioned on previously generated frames and the task prompt.

$$a_t \gets \pi_{\text{plan}}(I_0, q, H)$$

This allows the plan to adapt to the VGM's actual output, mitigating long-horizon drift.

2. VLM-VGM Collaborative Reasoning (Module 2): Addresses execution failure. For each generated clip $c_t$, the VLM verifier $\pi_{\text{verify}}$ produces a structured judgment $(v, d)$ where:

• $v \in \{\text{accept}, \text{reject}\}$
• $d$ is a textual reason and actionable suggestion for repair.

If $v = \text{reject}$, the next action prompt is evolved using the diagnosis: $a_t \gets \text{evolve}(a_t, d)$. The VGM is re-sampled with this revised prompt (up to $M$ retries). This localizes and repairs errors before they compound.

Empirical Validation / Results

Experimental Setup

• Benchmarks: Gen-ViRe (72 samples, 6 categories, VLM-judged) and VBVR-Bench (500 samples, 5 categories, rule-based evaluation).
• VGMs: VBVR-Wan2.2 (open-source), Veo 3.1 (closed-source), Cosmos-Predict-2.5.
• VLM: Gemini 2.5 Pro as default planner/verifier.
• Baselines: Single Inference (Pass@1), Pass@k ($k=2,4$), VideoTPO.
• Cost Metric: Total seconds of video generated by the VGM per sample (VLM compute is negligible).
• CollabVR Configuration: $N_{\text{max}}=3$, per-step attempt budget $M=3$.

Main Results

Table 1: Benchmarking results on Gen-ViRe.

Table 2: Benchmarking results on VBVR-Bench.

Key Findings:

• CollabVR achieves the highest scores on both benchmarks for the tested VGMs at comparable or lower VGM cost than Pass@4.
• Gains are most pronounced on categories requiring multi-step reasoning (Planning, Algorithmic, Spatial, Transformation).
• CollabVR improves performance even on a reasoning-fine-tuned VGM (VBVR-Wan2.2), demonstrating orthogonality.
• A blind user study showed human annotators preferred CollabVR outputs (73.8%) over Pass@4 (19.7%) and Pass@1 (6.5%).

Ablation Study

Table 3: Per-module ablation on Gen-ViRe and VBVR-Bench.

• Module Dominance: On Gen-ViRe (dominated by multi-step tasks), Progressive Planning (M1) is the larger contributor. On VBVR-Bench (dominated by single-step tasks), Collaborative Reasoning (M2) is the larger contributor. The framework adapts to task character.
• Effect of $N_{\text{max}}$: Performance on Gen-ViRe rises as $N_{\text{max}}$ increases to the required level (up to 3), then plateaus or degrades with further splitting (Figure 7).
• VLM Choice: Performance degrades gracefully with weaker VLMs (Qwen3.5-27B, Qwen3.5-9B), but even Qwen3.5-9B with CollabVR surpasses all baselines using Gemini 2.5 Pro (Table 4).

Analysis

Category-wise Module Effectiveness (Figure 8):

• Planning tasks benefit most from progressive planning (M1) (+0.165).
• Analogy tasks (atomic transformations) benefit from verifier-driven re-sampling (M2) alone (+0.139).
• The full pipeline (M1+M2) yields positive gains in every category, and is crucial for improving long-horizon Spatial tasks.
• Smallest gains are on symbolic/abstract categories (Analogy, Abstract), where the VGM lacks the underlying capability.

Human-Annotated Benchmark for VLM Supervision Reliability: A benchmark was constructed to evaluate the VLM's decisions along three axes:

1. Plan-depth match: VLM's predicted step count $N$ vs. human annotation.
2. Clip-level verification agreement: VLM's accept/reject judgment vs. human annotation.
3. Evolution quality: Human rating (1-3 scale) of the verifier's suggested repair.

Figure 9 shows results:

• Gemini 2.5 Pro aligns most closely with human annotators on all axes.
• Plan-depth match: Exact-match accuracy 68.0%, MAE 0.366.
• Verification agreement: F1 score 0.750.
• Evolution quality: Mean rating 2.61 (scale 1-3).

Theoretical and Practical Implications

• Paradigm Shift: Redirects test-time compute from sampling more videos (Pass@k) to refining the one being constructed through step-level diagnosis and repair.
• Complementary Strengths: Provides a principled framework to leverage the complementary strengths of VLMs (reasoning, planning, verification) and VGMs (visual simulation) in a synergistic closed loop.
• Practical Effectiveness: Demonstrates consistent improvements over existing test-time scaling methods at matched compute, making it a cost-effective approach for enhancing video reasoning.
• Orthogonality to Training: The test-time reasoning supervision is orthogonal to reasoning-oriented VGM fine-tuning, suggesting both approaches can be combined for further gains.
• Generalizability: The framework is training-free and works with any off-the-shelf VGM and VLM, showing graceful degradation with weaker VLMs.

Conclusion

CollabVR is a closed-loop framework that couples a VLM and VGM at step-level granularity for collaborative video reasoning. It addresses key VGM failure modes through progressive planning and failure-aware collaborative reasoning. The framework consistently improves performance on benchmark tasks for both open- and closed-source VGMs over existing baselines. Human annotation confirms the reliability of the VLM as a supervisor. Limitations include inability to overcome fundamental VGM capability gaps (e.g., symbolic transformations) and imperfect verifier accuracy. Future directions include integrating reasoning-oriented VGM training and finer-grained failure localization into the test-time loop.