# JoyAI-VL-Interaction: Real-Time Vision-Language Interaction Intelligence

> JoyAI-VL-Interaction is an 8B-scale vision-first model that learns per-second decisions to speak, stay silent, or delegate, beating Doubao and Gemini in 77.6% and 87.9% of human evaluations.

- **Source:** [arXiv](https://arxiv.org/abs/2606.14777)
- **Published:** 2026-06-17
- **Permalink:** https://picx.dev/p/4gaI78

## Summary

## Summary (Overview)

- **JoyAI-VL-Interaction** is an 8B-scale, vision-first VL-interaction model that is *proactive by design*: it continuously watches a live video stream and decides, at every second, whether to stay silent, speak, or delegate a hard task to a background model.
- The model makes "when to act" a learned, per-second decision, treating silence as a first-class action alongside speaking and delegating — moving beyond turn-based dialogue paradigms.
- The authors release a complete, deployable system with pluggable ASR/TTS, memory, visualization UI, and a background bridge to any API/model/agent, supporting hours of continuous video at sub-second latency.
- In head-to-head human evaluation across six real-world streaming scenarios, JoyAI-VL-Interaction wins **77.6%** of comparisons against Doubao's video-call assistant and **87.9%** against Gemini's, with 100% win rates in monitoring/alerting tasks.
- This is the first open, vision-driven interaction model released with its training recipe, data, and complete deployable system.

---

## Introduction and Theoretical Foundation

### The Turn-Based Limitation

The paper's central motivation is that **most real-world moments that need AI do not wait for a user to ask**. A fire starts on a security monitor, a toddler wanders toward a stove, a decisive play happens before anyone can cheer. The authors argue that today's large models are *structurally* turn-based: they "open their eyes only at the moment they are prompted" and cannot perceive "now" by construction.

### Inadequacy of Existing Approaches

The paper identifies three lines of work that fail to solve the problem:

1. **Real-time omni models** (GPT-Realtime-2, Qwen3.5-Omni): Genuinely end-to-end with low latency, but their optimization target remains *conversational turn-taking* — responding quickly *after the user speaks*. They are not designed to continuously watch a visual stream and decide unprompted.

2. **Consumer video-call products** (Doubao, Gemini): Doubao chases proactivity via periodic background polling (external `ExternalTextToLLM` triggers), but reaction is locked to the polling interval. Gemini is strictly one-question-one-answer with no background monitoring at all.

3. **Streaming-video understanding research**: Advances one property at a time (responsiveness, proactivity, or memory), rarely combines them, and is typically evaluated offline rather than in live deployment.

### The Interaction Model Paradigm

The authors adopt the term **"interaction model"** from Thinking Machines Lab (TML) and make its criterion explicit:

> An interaction model judges for itself, moment to moment, when the time is right to respond to the ongoing stream, the way a person does, choosing when to speak and when to stay silent rather than answering only when addressed.

The key distinction: *The difference is not how fast a model answers, but whether it decides for itself when answering is worth it.*

### Position vs. Related Work

| System | Interaction Decision | Modality Focus | Scale | Open-Source |
|---|---|---|---|---|
| GPT-Realtime-2 | Turn-based | Speech | Large | No |
| Qwen3.5-Omni | Turn-based | Omni | Large | Yes |
| Doubao video call | External polling | Vision+Speech | Large | No |
| Gemini video call | Turn-based | Vision+Speech | Large | No |
| TML-Interaction-Small | In-model | Audio+Vision (fused) | 276B MoE (12B active) | Research preview |
| MoshiRAG | In-model (speech) | Speech full-duplex | — | Yes |
| **JoyAI-VL-Interaction** | **In-model (per-second)** | **Vision-first, speech pluggable** | **~8B** | **Full stack** |

---

## Methodology

### 3.1 Base Model and Architecture

**JoyAI-VL 1.0** is the base model: language model initialized from Qwen3-8B, visual encoder from Qwen3-VL ViT, trained through representation alignment, vision-language pre-training, and post-training with On-Policy Distillation and RL.

**AdaCodec Video Encoding**: Instead of re-encoding every frame, AdaCodec (a predictive visual code) transmits only what prediction cannot explain:
- Full ViT tokens only on **reference frames** (at scene changes)
- Compact **P-tokens** (~16 tokens) on predictable frames, built from motion and residuals
- A predictive-cost reset opens a new reference frame when prediction becomes costly

This yields ~16× fewer tokens per predictable frame, so the token budget grows with scene change rate rather than frame count.

### 3.2 Data Construction for VL-Interaction

**Core format**: At each one-second step, the model takes one of three actions:
- `</silence>` — stay silent and keep watching
- `</response>` — speak (with textual reply)
- **Delegate** — send a hard subtask to an asynchronous background model (hidden delegate token + query, folded back into context when result returns)

**Six data families (>4M time-aligned streaming clips)**:
1. **Proactive alerting and anomaly detection** on live feeds
2. **Time-aligned QA** in backward, present, and forward timing
3. **Counting and perception** over time
4. **Live commentary and narration**
5. **Multi-turn casual chat** (egocentric video, long-video dialogue, companionship)
6. **Delegation episodes** — video-grounded knowledge questions, STEM problems, video reasoning routed to background

**Construction pipeline**: Multi-stage with verifier agents checking both *content* (what to say) and *timing* (exact second to act). Each example passes global checks (all frames + full annotation) and local checks (frames at annotated timestamp + reply). Silence is a first-class label, not the absence of one.

**Delegation episode anatomy**: The model emits a brief holding reply ("let me look into that"), then a hidden delegate token + query. A random delay simulates background reasoning time, forcing the model to stay present while delegation is pending. Episodes are synthesized three ways: (i) inserted pure-text hard problems, (ii) converted offline video-reasoning problems woven into multi-turn chat, (iii) fully synthesized multi-role agent pipelines (Planner → Verifier → Background → Foreground Rewriter).

### 3.3 Training Recipe

**Continue training**: Single supervised stage mixing time-aligned interaction data with conventional turn-based data.

**Weighted cross-entropy objective**: Since silence tokens vastly outnumber response tokens, the loss is role-weighted:

$$L(\theta) = -\frac{1}{|A|} \sum_{j \in A} w_j \log p_\theta(y_j | y_{<j})$$

where $A$ is the supervised assistant-token positions, $C \subseteq A$ is the control-token positions (each either `</silence>` or `</response>`), $w^{\text{first}}_{\text{silence}} = 1$ for the first silence in a run, $w^{\text{repeated}}_{\text{silence}} = 0.4$ for continued silence, and $w_{\text{response}} = 1.5$ for response onsets. Delegation needs no separate weight (it rides inside a response).

**Reinforcement learning (GRPO)**: Optimizes the per-second policy against stream-level rewards using answer-centered window sampling to keep rollouts tractable. Rewards: correct responses within the right window, appropriate silence, well-judged delegation; penalizes false alarms, mistimed responses, and degenerate always-respond behavior. Delegation scored in two parts: (1) handing off genuinely hard subtasks vs. easy ones, (2) using returned results well while staying responsive during pending delegation. Response content scored by an LLM judge against task-specific rubrics.

---

## Empirical Validation / Results

### Benchmark Design

Six everyday scenarios (58 cases total) comparing against Doubao and Gemini in-app video-call assistants:
1. **Monitoring and alerting** (10 cases) — flagging events the instant they occur
2. **Real-time counting** (10 cases) — objects/events over time
3. **Real-time translation** (10 cases) — on-screen content
4. **Time awareness** (10 cases) — acting on elapsed time
5. **Live commentary and guidance** (9 cases) — narrating unfolding scenes
6. **Long-horizon memory** (9 cases) — answering about things seen far earlier

**Protocol**: Human raters score each system on *quality* (correctness, relevance, well-formedness) and *timing* (arriving at the right moment, appropriate silence), each on a three-level scale (good/fair/poor). Equal-weight average per case → win/tie/loss. System identities hidden, order randomized, five university-educated LLM researchers rated with high inter-rater agreement.

### Results vs. Doubao

| Scenario | JoyAI-VL-Interaction | Tie | Doubao |
|---|---|---|---|
| Monitoring and alerting | **100.0%** | 0.0% | 0.0% |
| Real-time counting | **70.0%** | 30.0% | 0.0% |
| Real-time translation | **80.0%** | 20.0% | 0.0% |
| Time awareness | **80.0%** | 10.0% | 10.0% |
| Live commentary and guidance | **55.6%** | 22.2% | 22.2% |
| Long-horizon memory | **77.8%** | 22.2% | 0.0% |
| **Overall** | **77.6%** | 17.2% | 5.2% |

### Results vs. Gemini

| Scenario | JoyAI-VL-Interaction | Tie | Gemini |
|---|---|---|---|
| Monitoring and alerting | **100.0%** | 0.0% | 0.0% |
| Real-time counting | **100.0%** | 0.0% | 0.0% |
| Real-time translation | **100.0%** | 0.0% | 0.0% |
| Time awareness | **50.0%** | 40.0% | 10.0% |
| Live commentary and guidance | **100.0%** | 0.0% | 0.0% |
| Long-horizon memory | **77.8%** | 22.2% | 0.0% |
| **Overall** | **87.9%** | 10.3% | 1.7% |

### Key Case Study Findings

- **Fall Detection**: JoyAI-VL-Interaction alerts at the instant of collapse; Doubao reacts 4–5 seconds later; Gemini cannot monitor at all.
- **Dart Throw Counting**: All six throws counted on time vs. Doubao's two (with delay) and Gemini's single "let me check."
- **Street Interview Translation**: Continuous translation even with pinyin in subtitles; both baselines translate only the moment of request then stop.
- **Timed Cooking Scene** (20-second interval): JoyAI-VL-Interaction off by 1–2 seconds; Doubao fails entirely; Gemini signals at ~40 seconds.
- **Pet Livestream Commentary**: All pets narrated as they appear vs. Doubao's 3/13 (inaccurate) and Gemini's single prompted comment.
- **Phone App Delegation**: Produces working HTML reproducing the observed interface via background delegation — a capability baselines don't offer at all.

### Emergent Capabilities (Never Explicitly Trained For)

- **Shopping App Guidance**: Follows a changing phone interface through swipes, guiding the user to the intended item — despite *no app-interface video in the training data*.
- **Travel Scene Commentary with 4-second cadence**: Holds the cadence throughout with strong content — a combination of timed action + live commentary that *never co-occurs in training data*.

---

## Theoretical and Practical Implications

### The Scaling of Interactivity as a Capability

The paper's central thesis: **interactivity should scale as a capability of the model itself**, alongside the scaling of intelligence. Scaling interactivity means scaling three things simultaneously:
1. The model's freedom to speak on its own when a moment is worth a word
2. Its ability to interact in real time
3. Its sense of elapsed time

### The "Watch-and-Do" Paradigm

The system operationalizes a new mode of human-AI collaboration: from "submit a request and wait" to **"watch-and-do."** The model observes the physical world and, when warranted, delegates tasks that a background agent carries out in the digital world — closing a loop from seeing to acting.

### Design Principles with Broad Applicability

1. **Decision in the model, the rest replaceable**: The interaction model alone decides when to speak/delegate; everything else (ASR/TTS, memory, background brain, UI) is transduction and orchestration.

2. **Vision-first, speech-pluggable**: Rather than fusing audio and vision into the model, speech is interchangeable I/O, fitting a watch-and-interact setting where voice is a replaceable interface.

3. **Compact scale (~8B)**: Deliberately chosen over TML's 276B MoE (12B active) to enable local, low-cost deployment, fine-tuning, and reproduction by the broader community.

4. **Background-agnostic delegation protocol**: Any external model, agent, or API (e.g., Hermes Agent, OpenClaw) can serve as the background brain.

### Two Concurrent Loops Architecture

- **Real-time loop**: WebRTC/RTSP ingestion → 1 Hz sampling → model → TTS → UI
- **Asynchronous loop**: Delegate action → background bridge → background model/agent → result folded back into stream
- **Long-horizon memory**: Three-tier hierarchy (short-term raw vision tokens $T_s=100$s, mid-term text summaries $T_m = M T_s$, long-term compressed blocks $T_l = LM T_s$) reaching ~2 hours of context, designed for KV-cache prefix reuse in vLLM.

---

## Conclusion

### Main Takeaways

1. **The turn-based paradigm is structurally insufficient** for real-world events that don't wait for a user to ask. An interaction model that decides for itself when to act is not a speed improvement but a fundamentally different capability.

2. **A compact 8B model can outperform far larger products** (Seed 2.0 behind Doubao, Gemini-3.1-flash-live behind Gemini) precisely in the event-driven regime, because interactivity — not scale — is what matters there.

3. **Interactivity is data-efficient to learn**: Even modest time-aligned data yields strong interaction behavior, and capabilities never explicitly trained for emerge (app guidance, timed commentary).

4. **The full open release** (model weights, training recipe, data, complete system) is designed to lower the barrier for the community to build on this direction together.

### Limitations and Future Directions

- **Scale gap in general ability**: The baselines are stronger on general turn-based ability (broader knowledge, more polished chat, robustness on complex inputs). The claim is narrow: interaction models hold a natural advantage in event-driven scenarios.
- **Data at an early stage**: The mixture is not yet tuned, corpus not yet scaled to the intended volume; sparse commentary data + compact scale cause occasional hallucination during live narration.
- **Evaluation preliminary**: Six scenarios, 58 human-rated cases against two products — a larger, more fine-grained study is planned.
- **Future work**: Closing the gap in general ability, widening the scenario set, scaling the corpus, and exploring what interaction models can do in AI glasses, accessibility, home robots, elder care, and companionship.

### The Vision

> "You come home worn out after a long day, and before you have said anything, a quiet voice notices and offers, 'I can see you're tired; today must have been hard on you.' Presence like that, given unasked, is what an interaction model makes possible and a turn-based one, waiting to be addressed, never can."

The paper's larger hope is to move the field from turn-based dialogue toward genuine streaming interaction, in the open.

---

_Markdown view of https://picx.dev/p/4gaI78, served by PicX — AI-generated visual whiteboard summaries of research papers._