# VideoJev: System One Model for Video Understanding and Interaction

> VideoJev unifies temporal grounding and video QA via direct candidate scoring, achieving 100x faster inference while surpassing autoregressive baselines in accuracy.

- **Source:** [arXiv](https://arxiv.org/abs/2610.18615)
- **Published:** 2026-10-15
- **Permalink:** https://picx.dev/p/OKU5tw
- **Whiteboard:** https://picx.dev/p/OKU5tw/image

## Summary

## Summary

- **VideoJev** introduces a family of "System One" video understanding models that unify temporal grounding and multiple-choice video question answering (Video QA) through direct candidate scoring, replacing autoregressive answer decoding.
- The models reuse visual computation across queries via **Cached inference** (for arriving queries) and **Joint inference** (for multiple queries in one forward pass), achieving inference speedups of **2.8–3.1×** for grounding and **over 100×** for QA compared to autoregressive baselines.
- VideoJev-4B and VideoJev-35B-A3B improve mean grounding mIoU over their backbones by **6.7 and 10.8 points**, respectively, and mean QA scores by **8.0 and 5.7 points**, while remaining competitive with Thinking-mode baselines.
- Ablations show gains extend beyond training data alone; joint training on both tasks improves all benchmarks, and Soft IoU supervision is critical for event coverage (+4.6 mean grounding mIoU).
- The paper proposes a foundation for visual agents that compose structured judgments into complex, open-ended workflows, with future work on streaming inputs and calibration-aware training.

## Introduction and Theoretical Foundation

The paper addresses a core challenge in visual intelligence: video understanding must support **timely decisions** that guide reasoning and action. An agent may need to locate evidence, check progress, or assess possible actions, often using the same visual history—for example, a cooking assistant checking whether an ingredient has been added before choosing the next step.

The key premise is that **structured visual judgments can support open-ended behavior without requiring open-ended generation at every step**. Existing models have complementary strengths:
- **Discriminative vision models** are efficient but task-specific with fixed label spaces.
- **Video multimodal large language models (MLLMs)** offer broad semantic understanding but express decisions through slow autoregressive generation, even for compact structured outputs.

The paper builds on the **System One paradigm** (introduced by Jev), which produces typed probabilistic judgments that software can use directly without token-by-token generation. Video introduces an additional requirement: decisions must reflect **temporal structure** while sharing expensive visual context across queries.

VideoJev is built on two principles:
1. **Direct candidate scoring** – scores temporal intervals and textual options with a shared head.
2. **Reusable visual computation** – a shared video prefix is encoded once and reused across queries.

## Methodology

### Architecture

VideoJev adapts a pretrained MLLM (Qwen3.5-4B and Qwen3.6-35B-A3B) by removing the final MLP and replacing the language-model output head with a **candidate scorer**. Two new tokens, `<|option_end|>` and `<|decide|>`, provide candidate and query-level decision readouts. The scoring mechanism is defined by:

$$
u = \frac {W _ {q} h _ {d} + b _ {q}}{\| W _ {q} h _ {d} + b _ {q} \| _ {2}}, \qquad v _ {i} = \frac {W _ {k} h _ {i} + b _ {k}}{\| W _ {k} h _ {i} + b _ {k} \| _ {2}}, \qquad s _ {i} = 1 0 u ^ {\top} v _ {i} + b.\tag{1}
$$

where $h_d$ and $h_i$ are decision and candidate states, $W_q, W_k, b_q, b_k$ are learned projections, and $s_i$ is the logit score with fixed scale 10 and learned bias $b$.

### Temporal Grounding

For temporal grounding, the timeline is partitioned into candidate intervals. Each candidate $i$ has label $y_i = \mathbb{1}\{t_i \in G\}$ (midpoint inside ground-truth event union $G$). Training uses **multi-label binary cross-entropy (BCE)**:

$$
\mathcal {L} _ {\mathrm{BCE}} = - \frac {1}{K} \sum_ {i = 1} ^ {K} \left[ y _ {i} \log p _ {i} + (1 - y _ {i}) \log (1 - p _ {i}) \right].\tag{2}
$$

A **Soft IoU loss** adds event-level coverage supervision:

$$
\mathcal {L} _ {\mathrm{IoU}} = 1 - \frac {\sum_ {i} d _ {i} p _ {i} y _ {i}}{\sum_ {i} d _ {i} [ y _ {i} + p _ {i} (1 - y _ {i}) ]}.\tag{3}
$$

Decoding selects the contiguous range with the largest sum of threshold-adjusted logits (Kadane's algorithm, $O(K)$ time):

$$
(a ^ {\star}, b ^ {\star}) = \arg \max _ {1 \leq a \leq b \leq K} \sum_ {i = a} ^ {b} \left[ s _ {i} - \log \frac {\tau}{1 - \tau} \right].\tag{4}
$$

### Video QA

For multiple-choice QA, softmax normalizes option logits, trained with cross-entropy:

$$
p _ {i} = \frac {\exp (s _ {i})}{\sum_ {j = 1} ^ {K} \exp (s _ {j})}, \qquad \mathcal {L} _ {\mathrm{CE}} = - \log p _ {y},\tag{5}
$$

A reproducible, answer-independent permutation reduces position bias in training.

### Training Recipe

- **Two-stage training**: Stage one trains only the head on 30% of videos with frozen backbone; stage two adapts the language backbone, marker embeddings, and head on 70% of videos. The visual encoder and merger remain frozen throughout.
- **Training data**: TimeLens-100K/93K for grounding; ~240K internally curated multiple-choice questions over 19K videos for QA.
- **Optimization**: AdamW, BF16, cosine decay, DeepSpeed ZeRO-2/3.

## Empirical Validation / Results

### Temporal Grounding (Table 1)

| Model | Latency (↓) | Charades TL mIoU | ActivityNet TL mIoU | QVHighlights TL mIoU | Avg. |
|---|---|---|---|---|---|
| Qwen3.5-4B (backbone) | 0.763 | 52.3 | 55.6 | 67.9 | 58.6 |
| **VideoJev-4B** | **0.276** | **60.1** | **62.8** | **73.2** | **65.3** |
| Qwen3.6-35B-A3B (backbone) | 1.054 | 47.3 | 56.0 | 67.5 | 56.9 |
| **VideoJev-35B-A3B** | **0.336** | **63.7** | **65.5** | **74.0** | **67.7** |

### Video QA (Table 2, selected)

| Model | Latency (↓) | LVBench | Video-MME | LongVideoBench | MLVU | Avg. |
|---|---|---|---|---|---|
| Qwen3.5-4B Instruct | 1.355 | 60.0 | 70.6 | 61.2 | 73.3 | 66.3 |
| Qwen3.5-4B Thinking | 144.929 | 66.4 | 76.9 | 73.6 | 82.8 | 74.9 |
| **VideoJev-4B** | **1.123** | **67.9** | **75.0** | **73.6** | **80.6** | **74.3** |
| Qwen3.6-35B-A3B Instruct | 2.379 | 66.0 | 77.1 | 72.4 | 79.5 | 73.8 |
| Qwen3.6-35B-A3B Thinking | 149.777 | 71.4 | 82.5 | 74.9 | 86.2 | 78.8 |
| **VideoJev-35B-A3B** | **1.493** | **73.4** | **81.8** | **77.1** | **85.9** | **79.5** |

### Key Ablations

- **Architectural gains (Table 3)**: VideoJev-4B exceeds an autoregressive SFT control on identical data by **+1.3 mean grounding mIoU** and **+1.9 mean QA score**.
- **Joint inference** slightly improves over Cached inference (0.8/0.5 points at 4B; 0.3/0.7 at 35B-A3B), likely due to training alignment.
- **Joint training** on grounding and QA improves all benchmarks (+0.7 mIoU, +1.3 QA mean).
- **Soft IoU** is crucial: +4.6 mean grounding mIoU and +0.4 mean QA score.
- **Removing the final MLP** maintains performance (simpler readout adopted).
- **Two-stage adaptation** matches single-stage quality at lower cost.

### Efficiency

- At ten queries per video, Joint inference adds only **55 ms** (grounding) and **31 ms** (QA) compared to one query.
- VideoJev-4B: 0.39 s for ten grounding queries at 60.1 mIoU vs. Qwen3.5-4B Instruct's 5.98 s at 52.3 mIoU.
- Qwen Thinking takes 1,393 s for ten QA queries at 66.4% vs. VideoJev's 1.64 s at 67.9%.

### Confidence Calibration

- Higher confidence correlates with higher accuracy (e.g., 95.8% accuracy on high-confidence LVBench predictions).
- However, probabilities overestimate correctness: mean ECE is 8.65 (4B) and 8.13 (35B-A3B) percentage points.

## Theoretical and Practical Implications

- **Unified decision interface**: VideoJev demonstrates that a single scoring head can handle both temporal and semantic decisions, suggesting a path toward general-purpose visual decision systems.
- **Efficiency for agents**: The 100×+ speedups enable interactive, real-time video agents that would be impractical with autoregressive generation.
- **System One/System Two complementarity**: The paper envisions a higher-level agent (System Two) decomposing goals into candidate checks for VideoJev (System One) to evaluate, enabling complex open-ended workflows.
- **Training efficiency**: Two-stage adaptation reduces compute cost while maintaining quality, and joint task training provides mutual benefits.

## Conclusion

VideoJev unifies temporal grounding and multiple-choice Video QA through direct candidate scoring with shared visual computation. Dense 4B and MoE 35B-A3B models achieve strong quality and efficiency across seven benchmarks. Ablations confirm gains extend beyond data alone and that joint learning benefits both tasks.

**Future directions** include:
- Extending the candidate interface to more tasks (action recognition, event detection, scene-state classification).
- Building open-ended agent workflows where System Two plans and System One evaluates.
- Developing streaming inputs, calibration-aware training, and selective prediction for reliability.

The work establishes a foundation for visual agents that compose structured judgments into complex, open-ended workflows.

---

_Markdown view of https://picx.dev/p/OKU5tw, served by PicX — AI-generated visual whiteboard summaries of research papers._