Summary (Overview)

• Diagnostic Framework: Video-Oasis is introduced as a sustainable diagnostic suite to audit existing video understanding benchmarks, systematically testing for visual dependency, temporal dependency, and annotation ambiguity.
• High Shortcut Prevalence: A large-scale audit of 14 diverse benchmarks reveals that 55% of samples can be solved without visual input or temporal context, inflating reported model performance.
• Capability Gap: After filtering shortcut-solvable samples, state-of-the-art Video-LLMs (e.g., Qwen2.5-VL, Video-R1) perform only marginally above random chance ($\approx 25.6\%$) on the remaining video-native challenges.
• Algorithmic Insights: Using the distilled challenges as a testbed, the paper shows that temporal grounding, adaptive reasoning depth control, and a balanced combination of SFT and RLVR are critical for robust video understanding.
• Practical Guidelines: Video-Oasis provides a reproducible, extensible pipeline for constructing rigorous video benchmarks and evaluating future Video-LLMs. Code is available at https://github.com/sejong-rcv/Video-Oasis.

Introduction and Theoretical Foundation

The rise of multi-modal language models has shifted video understanding from narrow tasks (action recognition, temporal localization) toward integrated perception and reasoning. However, it is difficult to determine whether benchmark performance stems from visual perception, linguistic reasoning, or knowledge priors. Existing benchmarks often include samples solvable through shortcuts—e.g., linguistic priors, single-frame perception, or transcript reliance—rather than true spatio-temporal understanding.

The paper argues for a unified set of video-centric criteria for both benchmark creators and users. Instead of introducing yet another benchmark, Video-Oasis re-examines the essential criteria: visual dependency (is raw visual evidence required?) and temporal dependency (is temporal ordering necessary?). It also addresses annotation ambiguity arising from the complexity of video data.

The theoretical contribution lies in formalizing a diagnostic taxonomy that decouples visual and temporal cues, enabling systematic identification of shortcut-solvable samples. This extends prior auditing efforts (e.g., EgoTempo, Cambrian-S, Apollo) by jointly covering visual, temporal, and ambiguity axes across 14 benchmarks with cross-model consensus and human verification.

Methodology

Video-Oasis consists of three main diagnostic components:

1. Visual Dependency Tests (Figure 2a):

   • Blind: Provide only the question and answer options (no video). Detects cases solvable via linguistic bias or world knowledge.
   • Audio: Transcribe the video’s audio track into text and provide it to the model instead of the video.
   • Summary: Replace the video with a concatenated sequence of captions extracted at fixed intervals.
   • If a sample can be answered correctly from these degraded inputs, it does not require grounded visual perception.

2. Temporal Dependency Tests (Figure 2b):

   • Center-Frame: Provide only the middle frame. Detects tasks functioning as spatial recognition only.
   • Frame Shuffling: Randomly permute frame order to disrupt temporal causality.
   • Bag-of-Frames (BoF): Use a frozen CLIP-based encoder that does not model temporal order to perform top-$k$ frame matching against the query. If successful, the task does not require temporal reasoning.

3. Ambiguity Verification (Figure 2c):

   • Consistency: Identify samples where multiple models fail to agree, indicating inherent ambiguity or non-unique answers.
   • Redundancy: Detect cases solvable via any arbitrary video segment, revealing flawed question designs.
   • Sensitivity: Manually verify cases where models succeed despite frame shuffling, correcting false positives.

The audit uses a consensus threshold ($c$): the number of diagnostic models (from Table 2) that must answer a sample correctly for it to be counted as a shortcut. The final video-native challenge set is obtained by filtering out all samples identified as shortcuts under rigorous consensus conditions.

Empirical Validation / Results

Diagnostic Test Results (Table 2)

Aggregate results over 14 benchmarks (random chance baseline: 25.6%).

Shortcut Ratio per Benchmark Category (Table 3)

Shortcut-solvable samples appear consistently across all task groups.

Ambiguity Verifications and Unique Contributions (Tables 4 & 5)

Table 4: Manual Refinement Statistics

Table 5: Diagnostic Test Distributions ($c=3$)

Validity of Shortcut Identification (Table 6)

Correlation rate (%) under standard evaluation: shortcut-identified samples are already well-solved by current models.

Video-Native Challenges

After filtering, 11,033 QA pairs remain from 24,416 (55% reduction), associated with 4,938 unique videos. The remaining samples are classified into five video-native challenge categories:

• Fine-Grained Perception: Grounding fine-grained recognition in spatio-temporal context.
• Spatial World Understanding: Synthesizing multi-view evidence across frames to infer 3D context.
• Temporal Dynamics & Tracking: Monitoring changes over time (object tracking, action sequencing, state transitions).
• Causality & Logical Reasoning: Deducing cause-and-effect, physical laws, and unobserved intentions.
• Global Narrative: Integrating events across the full timeline to infer long-term semantics.

Model Performance on Distilled Challenges (Table 7)

Bold rows indicate top performers. Random chance baseline: 25.6%.

Ablation Studies on Algorithmic Designs

Temporal Grounding (Table 8): Adding AKS-based temporal grounding yields modest improvements (Eagle2.5: 31.5→32.9; Qwen3-VL-Instruct: 27.8→30.1).

Oracle Grounding (Table 9): With oracle temporal regions, performance on distilled samples jumps from 35.0% to 50.8%, while shortcuts improve only from 78.0% to 80.8%.

Reasoning Depth (Table 10):

Adaptive thinking outperforms always-on thinking, and oracle selection nearly matches frontier models.

Training Paradigms (Table 11):

SFT and RLVR have complementary strengths; grounding rewards improve Global Narrative significantly.

Theoretical and Practical Implications

Theoretical Implications:

• The work reframes video understanding evaluation by formally decoupling visual and temporal dependencies, providing a principled way to measure true spatio-temporal reasoning.
• The five video-native challenge categories (fine-grained perception, spatial world understanding, temporal dynamics, causality, global narrative) offer a data-driven taxonomy grounded in empirical diagnostic filtering.
• The finding that 55% of benchmark samples are shortcut-solvable calls into question the reported progress of Video-LLMs and emphasizes the need for rigorous evaluation criteria.
• The strong correlation between shortcut prevalence and inflated accuracy (Figure 1b) suggests that current benchmarks may be measuring linguistic reasoning or static perception rather than video understanding.

Practical Implications:

• For benchmark creators: Use Video-Oasis to audit new datasets for shortcut samples before release. The three-test protocol (Summary, Center-Frame, Frame Shuffling) captures 65% of unique shortcuts and is efficient to implement.
• For model developers: The distilled challenges reveal that precise temporal grounding and adaptive reasoning depth are crucial for robust video understanding. Training with a combination of SFT (for overall accuracy) and RLVR with grounding rewards (for reasoning tasks) is promising.
• For evaluation: The 55% reduction in evaluation volume enables more efficient yet rigorous testing. The pipeline is extensible to new benchmarks and challenges.
• For future research: The open-source code and configurable diagnostic suite provide a sustainable foundation for community-wide auditing and improvement.

Conclusion

Video-Oasis rethinks the evaluation of video understanding by introducing a rigorous, multi-axis diagnostic suite that filters shortcut-solvable samples from existing benchmarks. The audit reveals that 55% of samples can be answered without visual or temporal context, inflating reported performance. After filtering, current Video-LLMs perform only marginally above random chance on the remaining video-native challenges, exposing a substantial capability gap in spatio-temporal reasoning.

Using these distilled challenges as a testbed, the paper explores algorithmic design choices and finds that:

• Temporal grounding is essential, especially when strong spatio-temporal dependencies are present.
• Adaptive reasoning depth (deciding when to think versus when to perceive) is as impactful as raw model scale.
• A balanced combination of SFT and RLVR with grounding rewards offers complementary strengths.

The work is fully reproducible, and the entire pipeline is open-sourced to enable large-scale auditing and provide an extensible evaluation protocol for the community. Video-Oasis aims to serve as a foundation for building more rigorous benchmarks and driving the next generation of models toward robust video understanding.