Summary (Overview)

• Novel Benchmark: Introduces StreamVibe-Bench, the first unified benchmark for intent-aware proactive interaction, containing 4.1K evaluation samples and 85K training samples spanning explicit, implicit, and dynamic user intents across both physical and digital environments.
• New Model Architecture: Develops StreamVibe-VL, a streaming proactive video model that jointly learns when and how to assist under implicit and evolving user intents, trained via adaptive streaming SFT and localized trajectory reinforcement learning.
• Agentic Harness Design: Proposes StreamVibe-Harness, which maintains user intents as persistent, updatable states with flexible lifecycles and hierarchical memory, separating state tracking from video-grounded response decisions.
• Key Findings: StreamVibe-VL improves response timing and answer quality by 10.7% over baselines; existing proactive models struggle with implicit/dynamic intents; StreamVibe-Harness yields a further 12.3% gain in proactive interaction quality.
• Dual Evaluation Protocol: Introduces both pointwise (local decision-making with gold history) and trajectory (sustained interaction with self-generated history) evaluation settings to comprehensively assess proactive behavior.

Introduction and Theoretical Foundation

Vision-language models are shifting from offline passive understanding toward online proactive interaction. However, existing approaches suffer from three critical limitations:

1. Implicit Intent Gap: Current methods rely heavily on explicit requests (e.g., "Notify me when a goal is scored") and fail to infer implicit user intent (e.g., "Help me finish this task whenever I get stuck").
2. State Tracking Deficiency: Prior interactions can change the environment and task progress, making previously appropriate responses outdated or incorrect. Existing streaming models struggle to maintain up-to-date state.
3. Intent Evolution: User intents evolve as interactions unfold—later feedback may introduce new goals, revise existing constraints, or shift focus. Existing methods fail to update which intents remain relevant.

The paper formulates proactive assistance as a causal streaming decision process. The agent observes a video stream $\mathcal{V} = \{ \upsilon_t \}_{t=1}^{T}$ while the user expresses intents $i_j$ over time. Each intent becomes available at time $\tau_j$, and the agent maintains an active intent set $\mathcal{A}_t \subseteq \mathcal{I}_{\leq t}$. At each timestep, the agent accesses context $c_t = \langle \mathcal{V}_{\leq t}, \mathcal{I}_{\leq t}, \mathcal{H}_{<t} \rangle$ and predicts $a_t = \pi(c_t) \in \{\text{silence}, \text{response}\}$.

Key theoretical concepts include evidence points $\xi_{jk}$ with valid response windows $[s_{jk}, e_{jk}]$ and reference responses $\hat{y}_{jk}$. An intent remains active until all its evidence points are covered, requiring the agent to determine both when and what to respond while avoiding premature, missed, repeated, or irrelevant responses.

Methodology

3-Level Hierarchical Difficulty

The benchmark characterizes proactive interaction along two dimensions—intent explicitness and intent dynamics:

• Level 1 (Explicit Intents): User specifies a condition, event, or state to monitor (e.g., "Let me know when the water starts boiling").
• Level 2 (Implicit Intents): User provides only a high-level goal, requiring inference from ongoing observations (e.g., "Help me keep an eye on this dish").
• Level 3 (Dynamic Intents): Intents evolve through Revision (RV, modifies existing intent), Composition (CP, adds new constraints), and Interruption (IT, introduces immediate requests).

StreamVibe-Harness Framework

The Harness maintains an explicit interaction state $S_t = \langle \mathcal{A}_t, \mathcal{P}_t, \mathcal{R}_t \rangle$ where:

• $\mathcal{A}_t$: currently active intents
• $\mathcal{P}_t$: task progress and environment state
• $\mathcal{R}_t$: previously delivered responses

Each intent is tracked through an independent lifecycle (persistent vs. one-time), and a hierarchical memory $M_{<t}$ preserves recent information in fine-grained form while compressing older content. Before each decision, the Harness integrates state and retrieved memory: $\tilde{c}_t = \phi(c_t, S_t, m_t)$.

Two-Stage Policy Learning

Stage 1: Adaptive Streaming SFT uses adaptive decision-token weights to balance silence/response ratios:

$$\mathcal{L}_{\mathrm{SFT}} = -\frac{1}{|\mathcal{Y}|} \sum_{j \in \mathcal{Y}} \omega_j \log \pi_\theta(z_j | c_{t(j)}, z_{<j}), \quad \omega_j = \begin{cases} \gamma_{B,c(j)}, & j \in \mathcal{A}, \\ \lambda_{\mathrm{text}}, & j \in \mathcal{R}, \end{cases} \tag{1}$$

Stage 2: Localized Trajectory RL constructs short causal rollouts around annotated help points, optimized with GRPO using:

(i) Point-level alignment reward:

$$R_{\mathrm{align}} = \frac{(1 + \alpha) \sum_{(g,p) \in \mathcal{M}} \tau(\Delta_{g,p}) \rho(g,p)}{N_{\mathrm{pred}} + \alpha N_{\mathrm{gt}}} \tag{2}$$

where $\Delta_{g,p} = t_p - t_g$ is the temporal offset, $\tau(\cdot)$ is timing score, $\rho(\cdot)$ is content quality, and $\alpha$ controls missed-evidence vs. over-response trade-off.

(ii) Trajectory-level reward $R_{\mathrm{glob}}$ evaluates overall coherence, with final reward $R = \lambda R_{\mathrm{align}} + (1-\lambda) R_{\mathrm{glob}}$, $\lambda = 0.7$.

Empirical Validation / Results

Main Results on StreamVibe-Bench

Pointwise Evaluation (gold history)—StreamVibe-VL (RL) achieves:

• Explicit: F1 = 60.88, ANS = 55.32, C-F1 = 33.68
• Implicit: F1 = 72.01, ANS = 85.11, C-F1 = 61.28
• Dynamic: F1 = 66.34, ANS = 59.25, C-F1 = 39.31

Trajectory Evaluation (self-generated history)—StreamVibe-VL (RL) shows:

• Explicit: F1 = 26.51, ANS = 40.86, C-F1 = 10.83
• Implicit: F1 = 26.26, ANS = 72.83, C-F1 = 19.13
• Dynamic: F1 = 19.96, ANS = 44.62, C-F1 = 8.91

Key Diagnostic Findings

1. Intent understanding remains difficult: Implicit and Dynamic intents are markedly harder than Explicit intents.
2. Response selectivity is insufficient: Higher Recall often correlates with more false triggers.
3. Sustained interaction amplifies problems: Timing F1 declines steadily with interaction length; Revision shows sharper late-stage Recall drops; Composition exhibits weaker content quality.
4. RL improves decision selectivity: Substantially reduces FPR while improving answer quality.
5. Harness adds sustained-interaction gains: Adding the Harness suppresses unnecessary responses and improves content quality, with clearest gains on Dynamic Intents.

Generalization Results

StreamVibe-VL achieves competitive results on EgoProactive and consistent gains on IPIBench, while outperforming its base model across all general video benchmarks (MVBench, Video-MME, OVO-Bench, OVBench), indicating proactive policy learning preserves general video understanding.

Theoretical and Practical Implications

Theoretical Contributions

• Formalization of proactive interaction: Defines proactive assistance as a causal streaming decision process with evidence points, response windows, and active intent tracking.
• Separation of concerns: Decomposes proactive interaction into evidence grounding (video-grounded response decisions) and state tracking (persistent intent maintenance), showing these are complementary capabilities.
• Reward design insights: Demonstrates that localized trajectory RL with point-level alignment and trajectory-level coherence rewards outperforms full-trajectory optimization, which suffers from diffuse credit assignment.

Practical Implications

• Benchmark as testbed: StreamVibe-Bench provides a unified evaluation protocol for intent-aware interaction, covering physical (egocentric activities) and digital (GUI/computer-use) environments with both pointwise and trajectory settings.
• Architecture guidance: The two-stage training pipeline (adaptive SFT + localized RL) offers a practical recipe for training streaming proactive models.
• Harness deployment: The StreamVibe-Harness demonstrates that explicit state tracking is crucial for long-horizon interaction stability, reducing outdated and redundant responses.

Conclusion

This work addresses three research questions in intent-aware proactive interaction: constructing a testbed (StreamVibe-Bench), internalizing intent-aware interaction as a native model capability (StreamVibe-VL), and flexibly managing context for state tracking (StreamVibe-Harness).

The paper demonstrates that:

1. Policy learning (SFT + RL) mainly improves local proactive decisions
2. State tracking (Harness) helps preserve valid intents and task progress over long-horizon interactions

Future directions include studying co-evolving interactions where new observations, task outcomes, and prior assistance jointly reshape user needs and the environment, particularly the complex feedback between intents and environmental states over longer horizons.