Summary

• VideoJev introduces a family of "System One" video understanding models that unify temporal grounding and multiple-choice video question answering (Video QA) through direct candidate scoring, replacing autoregressive answer decoding.
• The models reuse visual computation across queries via Cached inference (for arriving queries) and Joint inference (for multiple queries in one forward pass), achieving inference speedups of 2.8–3.1× for grounding and over 100× for QA compared to autoregressive baselines.
• VideoJev-4B and VideoJev-35B-A3B improve mean grounding mIoU over their backbones by 6.7 and 10.8 points, respectively, and mean QA scores by 8.0 and 5.7 points, while remaining competitive with Thinking-mode baselines.
• Ablations show gains extend beyond training data alone; joint training on both tasks improves all benchmarks, and Soft IoU supervision is critical for event coverage (+4.6 mean grounding mIoU).
• The paper proposes a foundation for visual agents that compose structured judgments into complex, open-ended workflows, with future work on streaming inputs and calibration-aware training.

Introduction and Theoretical Foundation

The paper addresses a core challenge in visual intelligence: video understanding must support timely decisions that guide reasoning and action. An agent may need to locate evidence, check progress, or assess possible actions, often using the same visual history—for example, a cooking assistant checking whether an ingredient has been added before choosing the next step.

The key premise is that structured visual judgments can support open-ended behavior without requiring open-ended generation at every step. Existing models have complementary strengths:

• Discriminative vision models are efficient but task-specific with fixed label spaces.
• Video multimodal large language models (MLLMs) offer broad semantic understanding but express decisions through slow autoregressive generation, even for compact structured outputs.

The paper builds on the System One paradigm (introduced by Jev), which produces typed probabilistic judgments that software can use directly without token-by-token generation. Video introduces an additional requirement: decisions must reflect temporal structure while sharing expensive visual context across queries.

VideoJev is built on two principles:

1. Direct candidate scoring – scores temporal intervals and textual options with a shared head.
2. Reusable visual computation – a shared video prefix is encoded once and reused across queries.

Methodology

Architecture

VideoJev adapts a pretrained MLLM (Qwen3.5-4B and Qwen3.6-35B-A3B) by removing the final MLP and replacing the language-model output head with a candidate scorer. Two new tokens, <|option_end|> and <|decide|>, provide candidate and query-level decision readouts. The scoring mechanism is defined by:

$$u = \frac {W _ {q} h _ {d} + b _ {q}}{\| W _ {q} h _ {d} + b _ {q} \| _ {2}}, \qquad v _ {i} = \frac {W _ {k} h _ {i} + b _ {k}}{\| W _ {k} h _ {i} + b _ {k} \| _ {2}}, \qquad s _ {i} = 1 0 u ^ {\top} v _ {i} + b.\tag{1}$$

where $h_d$ and $h_i$ are decision and candidate states, $W_q, W_k, b_q, b_k$ are learned projections, and $s_i$ is the logit score with fixed scale 10 and learned bias $b$.

Temporal Grounding

For temporal grounding, the timeline is partitioned into candidate intervals. Each candidate $i$ has label $y_i = \mathbb{1}\{t_i \in G\}$ (midpoint inside ground-truth event union $G$). Training uses multi-label binary cross-entropy (BCE):

$$\mathcal {L} _ {\mathrm{BCE}} = - \frac {1}{K} \sum_ {i = 1} ^ {K} \left[ y _ {i} \log p _ {i} + (1 - y _ {i}) \log (1 - p _ {i}) \right].\tag{2}$$

A Soft IoU loss adds event-level coverage supervision:

$$\mathcal {L} _ {\mathrm{IoU}} = 1 - \frac {\sum_ {i} d _ {i} p _ {i} y _ {i}}{\sum_ {i} d _ {i} [ y _ {i} + p _ {i} (1 - y _ {i}) ]}.\tag{3}$$

Decoding selects the contiguous range with the largest sum of threshold-adjusted logits (Kadane's algorithm, $O(K)$ time):

$$(a ^ {\star}, b ^ {\star}) = \arg \max _ {1 \leq a \leq b \leq K} \sum_ {i = a} ^ {b} \left[ s _ {i} - \log \frac {\tau}{1 - \tau} \right].\tag{4}$$

Video QA

For multiple-choice QA, softmax normalizes option logits, trained with cross-entropy:

$$p _ {i} = \frac {\exp (s _ {i})}{\sum_ {j = 1} ^ {K} \exp (s _ {j})}, \qquad \mathcal {L} _ {\mathrm{CE}} = - \log p _ {y},\tag{5}$$

A reproducible, answer-independent permutation reduces position bias in training.

Training Recipe

• Two-stage training: Stage one trains only the head on 30% of videos with frozen backbone; stage two adapts the language backbone, marker embeddings, and head on 70% of videos. The visual encoder and merger remain frozen throughout.
• Training data: TimeLens-100K/93K for grounding; ~240K internally curated multiple-choice questions over 19K videos for QA.
• Optimization: AdamW, BF16, cosine decay, DeepSpeed ZeRO-2/3.

Empirical Validation / Results

Temporal Grounding (Table 1)

Video QA (Table 2, selected)

| Model | Latency (↓) | LVBench | Video-MME | LongVideoBench | MLVU | Avg. | |---|---|---|---|---|---| | Qwen3.5-4B Instruct | 1.355 | 60.0 | 70.6 | 61.2 | 73.3 | 66.3 | | Qwen3.5-4B Thinking | 144.929 | 66.4 | 76.9 | 73.6 | 82.8 | 74.9 | | VideoJev-4B | 1.123 | 67.9 | 75.0 | 73.6 | 80.6 | 74.3 | | Qwen3.6-35B-A3B Instruct | 2.379 | 66.0 | 77.1 | 72.4 | 79.5 | 73.8 | | Qwen3.6-35B-A3B Thinking | 149.777 | 71.4 | 82.5 | 74.9 | 86.2 | 78.8 | | VideoJev-35B-A3B | 1.493 | 73.4 | 81.8 | 77.1 | 85.9 | 79.5 |

Key Ablations

• Architectural gains (Table 3): VideoJev-4B exceeds an autoregressive SFT control on identical data by +1.3 mean grounding mIoU and +1.9 mean QA score.
• Joint inference slightly improves over Cached inference (0.8/0.5 points at 4B; 0.3/0.7 at 35B-A3B), likely due to training alignment.
• Joint training on grounding and QA improves all benchmarks (+0.7 mIoU, +1.3 QA mean).
• Soft IoU is crucial: +4.6 mean grounding mIoU and +0.4 mean QA score.
• Removing the final MLP maintains performance (simpler readout adopted).
• Two-stage adaptation matches single-stage quality at lower cost.

Efficiency

• At ten queries per video, Joint inference adds only 55 ms (grounding) and 31 ms (QA) compared to one query.
• VideoJev-4B: 0.39 s for ten grounding queries at 60.1 mIoU vs. Qwen3.5-4B Instruct's 5.98 s at 52.3 mIoU.
• Qwen Thinking takes 1,393 s for ten QA queries at 66.4% vs. VideoJev's 1.64 s at 67.9%.

Confidence Calibration

• Higher confidence correlates with higher accuracy (e.g., 95.8% accuracy on high-confidence LVBench predictions).
• However, probabilities overestimate correctness: mean ECE is 8.65 (4B) and 8.13 (35B-A3B) percentage points.

Theoretical and Practical Implications

• Unified decision interface: VideoJev demonstrates that a single scoring head can handle both temporal and semantic decisions, suggesting a path toward general-purpose visual decision systems.
• Efficiency for agents: The 100×+ speedups enable interactive, real-time video agents that would be impractical with autoregressive generation.
• System One/System Two complementarity: The paper envisions a higher-level agent (System Two) decomposing goals into candidate checks for VideoJev (System One) to evaluate, enabling complex open-ended workflows.
• Training efficiency: Two-stage adaptation reduces compute cost while maintaining quality, and joint task training provides mutual benefits.

Conclusion

VideoJev unifies temporal grounding and multiple-choice Video QA through direct candidate scoring with shared visual computation. Dense 4B and MoE 35B-A3B models achieve strong quality and efficiency across seven benchmarks. Ablations confirm gains extend beyond data alone and that joint learning benefits both tasks.

Future directions include:

• Extending the candidate interface to more tasks (action recognition, event detection, scene-state classification).
• Building open-ended agent workflows where System Two plans and System One evaluates.
• Developing streaming inputs, calibration-aware training, and selective prediction for reliability.

The work establishes a foundation for visual agents that compose structured judgments into complex, open-ended workflows.