Summary (Overview)

• JoyAI-VL-Interaction is an 8B-scale, vision-first VL-interaction model that is proactive by design: it continuously watches a live video stream and decides, at every second, whether to stay silent, speak, or delegate a hard task to a background model.
• The model makes "when to act" a learned, per-second decision, treating silence as a first-class action alongside speaking and delegating — moving beyond turn-based dialogue paradigms.
• The authors release a complete, deployable system with pluggable ASR/TTS, memory, visualization UI, and a background bridge to any API/model/agent, supporting hours of continuous video at sub-second latency.
• In head-to-head human evaluation across six real-world streaming scenarios, JoyAI-VL-Interaction wins 77.6% of comparisons against Doubao's video-call assistant and 87.9% against Gemini's, with 100% win rates in monitoring/alerting tasks.
• This is the first open, vision-driven interaction model released with its training recipe, data, and complete deployable system.

---

Introduction and Theoretical Foundation

The Turn-Based Limitation

The paper's central motivation is that most real-world moments that need AI do not wait for a user to ask. A fire starts on a security monitor, a toddler wanders toward a stove, a decisive play happens before anyone can cheer. The authors argue that today's large models are structurally turn-based: they "open their eyes only at the moment they are prompted" and cannot perceive "now" by construction.

Inadequacy of Existing Approaches

The paper identifies three lines of work that fail to solve the problem:

1. Real-time omni models (GPT-Realtime-2, Qwen3.5-Omni): Genuinely end-to-end with low latency, but their optimization target remains conversational turn-taking — responding quickly after the user speaks. They are not designed to continuously watch a visual stream and decide unprompted.

2. Consumer video-call products (Doubao, Gemini): Doubao chases proactivity via periodic background polling (external ExternalTextToLLM triggers), but reaction is locked to the polling interval. Gemini is strictly one-question-one-answer with no background monitoring at all.

3. Streaming-video understanding research: Advances one property at a time (responsiveness, proactivity, or memory), rarely combines them, and is typically evaluated offline rather than in live deployment.

The Interaction Model Paradigm

The authors adopt the term "interaction model" from Thinking Machines Lab (TML) and make its criterion explicit:

An interaction model judges for itself, moment to moment, when the time is right to respond to the ongoing stream, the way a person does, choosing when to speak and when to stay silent rather than answering only when addressed.

The key distinction: The difference is not how fast a model answers, but whether it decides for itself when answering is worth it.

Position vs. Related Work

---

Methodology

3.1 Base Model and Architecture

JoyAI-VL 1.0 is the base model: language model initialized from Qwen3-8B, visual encoder from Qwen3-VL ViT, trained through representation alignment, vision-language pre-training, and post-training with On-Policy Distillation and RL.

AdaCodec Video Encoding: Instead of re-encoding every frame, AdaCodec (a predictive visual code) transmits only what prediction cannot explain:

• Full ViT tokens only on reference frames (at scene changes)
• Compact P-tokens (~16 tokens) on predictable frames, built from motion and residuals
• A predictive-cost reset opens a new reference frame when prediction becomes costly

This yields ~16× fewer tokens per predictable frame, so the token budget grows with scene change rate rather than frame count.

3.2 Data Construction for VL-Interaction

Core format: At each one-second step, the model takes one of three actions:

• </silence> — stay silent and keep watching
• </response> — speak (with textual reply)
• Delegate — send a hard subtask to an asynchronous background model (hidden delegate token + query, folded back into context when result returns)

Six data families (>4M time-aligned streaming clips):

1. Proactive alerting and anomaly detection on live feeds
2. Time-aligned QA in backward, present, and forward timing
3. Counting and perception over time
4. Live commentary and narration
5. Multi-turn casual chat (egocentric video, long-video dialogue, companionship)
6. Delegation episodes — video-grounded knowledge questions, STEM problems, video reasoning routed to background

Construction pipeline: Multi-stage with verifier agents checking both content (what to say) and timing (exact second to act). Each example passes global checks (all frames + full annotation) and local checks (frames at annotated timestamp + reply). Silence is a first-class label, not the absence of one.

Delegation episode anatomy: The model emits a brief holding reply ("let me look into that"), then a hidden delegate token + query. A random delay simulates background reasoning time, forcing the model to stay present while delegation is pending. Episodes are synthesized three ways: (i) inserted pure-text hard problems, (ii) converted offline video-reasoning problems woven into multi-turn chat, (iii) fully synthesized multi-role agent pipelines (Planner → Verifier → Background → Foreground Rewriter).

3.3 Training Recipe

Continue training: Single supervised stage mixing time-aligned interaction data with conventional turn-based data.

Weighted cross-entropy objective: Since silence tokens vastly outnumber response tokens, the loss is role-weighted:

$$L(\theta) = -\frac{1}{|A|} \sum_{j \in A} w_j \log p_\theta(y_j | y_{<j})$$

where $A$ is the supervised assistant-token positions, $C \subseteq A$ is the control-token positions (each either </silence> or </response>), $w^{\text{first}}_{\text{silence}} = 1$ for the first silence in a run, $w^{\text{repeated}}_{\text{silence}} = 0.4$ for continued silence, and $w_{\text{response}} = 1.5$ for response onsets. Delegation needs no separate weight (it rides inside a response).

Reinforcement learning (GRPO): Optimizes the per-second policy against stream-level rewards using answer-centered window sampling to keep rollouts tractable. Rewards: correct responses within the right window, appropriate silence, well-judged delegation; penalizes false alarms, mistimed responses, and degenerate always-respond behavior. Delegation scored in two parts: (1) handing off genuinely hard subtasks vs. easy ones, (2) using returned results well while staying responsive during pending delegation. Response content scored by an LLM judge against task-specific rubrics.

---

Empirical Validation / Results

Benchmark Design

Six everyday scenarios (58 cases total) comparing against Doubao and Gemini in-app video-call assistants:

1. Monitoring and alerting (10 cases) — flagging events the instant they occur
2. Real-time counting (10 cases) — objects/events over time
3. Real-time translation (10 cases) — on-screen content
4. Time awareness (10 cases) — acting on elapsed time
5. Live commentary and guidance (9 cases) — narrating unfolding scenes
6. Long-horizon memory (9 cases) — answering about things seen far earlier

Protocol: Human raters score each system on quality (correctness, relevance, well-formedness) and timing (arriving at the right moment, appropriate silence), each on a three-level scale (good/fair/poor). Equal-weight average per case → win/tie/loss. System identities hidden, order randomized, five university-educated LLM researchers rated with high inter-rater agreement.

Results vs. Doubao

Results vs. Gemini

Key Case Study Findings

• Fall Detection: JoyAI-VL-Interaction alerts at the instant of collapse; Doubao reacts 4–5 seconds later; Gemini cannot monitor at all.
• Dart Throw Counting: All six throws counted on time vs. Doubao's two (with delay) and Gemini's single "let me check."
• Street Interview Translation: Continuous translation even with pinyin in subtitles; both baselines translate only the moment of request then stop.
• Timed Cooking Scene (20-second interval): JoyAI-VL-Interaction off by 1–2 seconds; Doubao fails entirely; Gemini signals at ~40 seconds.
• Pet Livestream Commentary: All pets narrated as they appear vs. Doubao's 3/13 (inaccurate) and Gemini's single prompted comment.
• Phone App Delegation: Produces working HTML reproducing the observed interface via background delegation — a capability baselines don't offer at all.

Emergent Capabilities (Never Explicitly Trained For)

• Shopping App Guidance: Follows a changing phone interface through swipes, guiding the user to the intended item — despite no app-interface video in the training data.
• Travel Scene Commentary with 4-second cadence: Holds the cadence throughout with strong content — a combination of timed action + live commentary that never co-occurs in training data.

---

Theoretical and Practical Implications

The Scaling of Interactivity as a Capability

The paper's central thesis: interactivity should scale as a capability of the model itself, alongside the scaling of intelligence. Scaling interactivity means scaling three things simultaneously:

1. The model's freedom to speak on its own when a moment is worth a word
2. Its ability to interact in real time
3. Its sense of elapsed time

The "Watch-and-Do" Paradigm

The system operationalizes a new mode of human-AI collaboration: from "submit a request and wait" to "watch-and-do." The model observes the physical world and, when warranted, delegates tasks that a background agent carries out in the digital world — closing a loop from seeing to acting.

Design Principles with Broad Applicability

1. Decision in the model, the rest replaceable: The interaction model alone decides when to speak/delegate; everything else (ASR/TTS, memory, background brain, UI) is transduction and orchestration.

2. Vision-first, speech-pluggable: Rather than fusing audio and vision into the model, speech is interchangeable I/O, fitting a watch-and-interact setting where voice is a replaceable interface.

3. Compact scale (~8B): Deliberately chosen over TML's 276B MoE (12B active) to enable local, low-cost deployment, fine-tuning, and reproduction by the broader community.

4. Background-agnostic delegation protocol: Any external model, agent, or API (e.g., Hermes Agent, OpenClaw) can serve as the background brain.

Two Concurrent Loops Architecture

• Real-time loop: WebRTC/RTSP ingestion → 1 Hz sampling → model → TTS → UI
• Asynchronous loop: Delegate action → background bridge → background model/agent → result folded back into stream
• Long-horizon memory: Three-tier hierarchy (short-term raw vision tokens $T_s=100$s, mid-term text summaries $T_m = M T_s$, long-term compressed blocks $T_l = LM T_s$) reaching ~2 hours of context, designed for KV-cache prefix reuse in vLLM.

---

Conclusion

Main Takeaways

1. The turn-based paradigm is structurally insufficient for real-world events that don't wait for a user to ask. An interaction model that decides for itself when to act is not a speed improvement but a fundamentally different capability.

2. A compact 8B model can outperform far larger products (Seed 2.0 behind Doubao, Gemini-3.1-flash-live behind Gemini) precisely in the event-driven regime, because interactivity — not scale — is what matters there.

3. Interactivity is data-efficient to learn: Even modest time-aligned data yields strong interaction behavior, and capabilities never explicitly trained for emerge (app guidance, timed commentary).

4. The full open release (model weights, training recipe, data, complete system) is designed to lower the barrier for the community to build on this direction together.

Limitations and Future Directions

• Scale gap in general ability: The baselines are stronger on general turn-based ability (broader knowledge, more polished chat, robustness on complex inputs). The claim is narrow: interaction models hold a natural advantage in event-driven scenarios.
• Data at an early stage: The mixture is not yet tuned, corpus not yet scaled to the intended volume; sparse commentary data + compact scale cause occasional hallucination during live narration.
• Evaluation preliminary: Six scenarios, 58 human-rated cases against two products — a larger, more fine-grained study is planned.
• Future work: Closing the gap in general ability, widening the scenario set, scaling the corpus, and exploring what interaction models can do in AI glasses, accessibility, home robots, elder care, and companionship.

The Vision

"You come home worn out after a long day, and before you have said anything, a quiet voice notices and offers, 'I can see you're tired; today must have been hard on you.' Presence like that, given unasked, is what an interaction model makes possible and a turn-based one, waiting to be addressed, never can."

The paper's larger hope is to move the field from turn-based dialogue toward genuine streaming interaction, in the open.