Summary (Overview)

• OneStreamer is a streaming video LLM that unifies query-independent evidence recording and task response through a single proactive generation process, jointly learning when to record/respond and what to generate.
• Proactive Hierarchical Caption Memory (PHCM) generates time-grounded local-detail captions (</Observe>) and semantic summaries (</Summary>) that complement a recent visual window, providing reusable factual context without revisiting historical visual features.
• Proactive State Transition Learning (PSTL) addresses state-label imbalance by preserving supervision at all output anchors and selecting representative state-change and state-persistence tokens, supervising only 27.5% of annotated state tokens.
• OneStreamer-1M is a broad-coverage streaming video interaction dataset (over 1 million records) built via a reusable streaming data synthesis pipeline that aligns output content and timing with available evidence.
• The 4B model achieves state-of-the-art results across all eight evaluated streaming video understanding benchmarks, with an average relative improvement of 25.0% over the Qwen3-VL baseline and 6.1% over the strongest competing method.

---

Introduction and Theoretical Foundation

Real-world video arrives as an open-ended stream of observations. In live-stream assistants, wearable agents, and security monitoring systems, a model must interpret incoming visual evidence without knowing which observations will matter to future questions. Key challenges:

1. Evidence retention: An event that initially appears unimportant may become relevant later, after its source frames have left the model's limited visual context.
2. Timely response: A user-visible response must be grounded in available evidence and delivered before the relevant moment passes.

Existing approaches face fundamental limitations:

• Memory compression/retrieval methods extend accessible history but retained historical visual tokens compete with recent observations for limited context capacity.
• Streaming thinking methods maintain evolving reasoning, but reasoning traces do not necessarily serve as reusable time-grounded factual records.
• State-token-based response control enables proactive outputs, but dense supervision overemphasizes repeated silence labels relative to sparse response decisions.

The central research question: How can a streaming video LLM turn incoming observations into reusable factual memory for timely responses that draw on both past and current evidence?

The key insight is to rethink the role of proactive generation: beyond producing user-visible responses, a streaming model can learn to turn observed evidence into reusable factual records, connecting perception learning with query-independent memory formation through a shared causal generation process.

---

Methodology

3.1 Overall Architecture

OneStreamer processes a live video stream incrementally as temporally ordered clips:

• A vision encoder extracts visual tokens from each incoming clip; a projector maps them into the LLM's embedding space.
• A Recent-N FIFO sliding window retains only visual tokens from the latest $N$ observed frames.
• Visual tokens are interleaved with a text history containing accumulated caption records and dialogue.
• The LLM predicts task-specific control tokens and generates associated text when the predicted state initiates an output.

Control tokens:

• </Observe>: introduces a local-detail caption of observed objects, actions, scenes, and state changes
• </Summary>: introduces a semantic summary of a completed event or segment
• </Standby>: relevant evidence is emerging but the model is not yet ready to answer
• </Response>: initiates user-visible answer generation
• </Silence>: continued observation without generating caption or response

3.2 Proactive Hierarchical Caption Memory (PHCM)

PHCM organizes observed evidence into two types of time-aligned records:

At inference, OneStreamer incrementally builds PHCM using only currently available visual-language context. Each generated record is appended to the text history in generation order, providing long-range context without retrieving or revisiting historical visual features.

3.3 Proactive State Transition Learning (PSTL)

PSTL addresses state-label imbalance where repeated silence tokens outnumber output-initiating tokens. The method:

1. Groups state tokens by the transition from the preceding control state to the current one.
2. Defines a supervision quota $q$ shared across all transition groups in a sequence:

$$q = \max_{X} \max_{a \in A_\tau} n_{X \to a}$$

where $A_\tau$ is the output-anchor set for task $\tau$, and $n_{X \to Y}$ is the number of occurrences of transition $X \to Y$ between consecutive annotated control states.

3. From each transition group $X \to Y$, samples state-token indices uniformly without replacement to form $I_{X \to Y}$ of size:

$$|I_{X \to Y}| = \min(n_{X \to Y}, q)$$

4. Only state tokens indexed by $I = \bigcup_{X,Y} I_{X \to Y}$ contribute to the state loss.

This preserves supervision for all output-initiating tokens while selecting examples of both state changes and state persistence. Unselected state tokens remain in the causal sequence but are excluded from the state loss.

4 Dataset Construction: OneStreamer-1M

Streaming Caption Synthesis (3 stages):

1. High-quality video curation: semantic retrieval, scene detection, visual-richness assessment
2. Multi-granularity caption annotation: Gemini and Seed generate timestamp-grounded captions at frame/clip, segment, and video levels, verified for factual and temporal consistency
3. Causal streaming sequence construction: frame/clip captions become </Observe> targets at interval ends; segment captions become </Summary> targets at segment boundaries

Streaming QA Synthesis (3 stages):

1. Task-directed streaming QA design: task-specific templates for streaming interaction capabilities
2. Coarse evidence interval localization: VLM localizes temporal interval containing supporting evidence; verification via clip-only answering
3. Fine-grained response-time calibration: candidate timestamps scored by conditional likelihood (reasoning QA) or visual-text similarity (perception tasks); earliest reliable timestamp selected as response time

OneStreamer-1M task composition includes: hierarchical captioning (52,500), flat captioning (7,500), alerts & reminders (173,930), continuous description (76,220), visual interaction (9,048), active counting (7,633), contextual feedback (4,642), procedural assistance (97), general video QA (162,866 + 97,229), gesture & expression QA (20,782 + 3,606), video dialogue QA (10,000), temporal QA (1,824), procedural QA (1,263 + 12,877), anomaly QA (849), spatial QA (360,603), temporal grounding (92,283), counting QA (26,149), future prediction (17,193), risk reasoning (13,081), answerability judgment (6,515), person identification (1,314).

---

Empirical Validation / Results

5.1 Main Results

Table 1: Online video benchmark results — OneStreamer achieves the highest score on all eight benchmarks:

5.2 Ablations

Finding 1: Proactively generated captions provide reusable temporal memory beyond the recent visual window.

Table 2: Effects of PHCM — Retaining generated captions improves historical QA:

Finding 2: Combining recent visual context with distant caption memory jointly improves perception and memory. PHCM exceeds FIFO's real-time perception (81.4 vs. 80.9) and Full's memory performance (ASI: 71.6 vs. 67.6) within a single representation.

Finding 3: PSTL outperforms dense state-token supervision.

Table 3: Ablation of state-token supervision:

5.3 Efficiency and Latency

Table 4: Answer-stage efficiency (360s OVOBench sample, single NVIDIA H200):

PHCM reduces context length by 93.1% and GPU memory by 60.4% relative to Full, while reducing TTFT from 4.560s to 0.124s. Compared to FIFO, it adds only 1,272 context tokens, 0.29 GB GPU memory, and 0.030s TTFT.

---

Theoretical and Practical Implications

1. Proactive generation as a shared learning interface: The results support proactive generation as a unified mechanism connecting visual perception, reusable factual memory, and timely task response within a single streaming model—challenging the view that memory and perception require separate architectural components.

2. Heterogeneous memory representation: PHCM demonstrates that text-based factual records can effectively replace retained visual history for long-range context, providing a more efficient alternative to visual token retention under limited context capacity.

3. Selective supervision beats dense supervision: PSTL shows that selectively supervising representative state transitions and persistence tokens outperforms both dense supervision and frequency-balanced focal loss, with only 27.5% of state tokens supervised—suggesting that what you supervise matters more than how much.

4. Practical deployment efficiency: The 93.1% context reduction and 36.8× TTFT improvement over full-history retention make streaming video interaction feasible for real-time applications with strict latency requirements.

5. Data synthesis as a reusable resource: The streaming data synthesis pipeline provides a template for converting offline video resources into evidence-aligned streaming training sequences, addressing the scarcity of high-quality streaming interaction data.

---

Conclusion

OneStreamer presents a unified framework for streaming video interaction where proactive generation serves as a shared learning mechanism for:

• Perception: interpreting observed video prefixes through causal streaming caption targets
• Memory formation: turning observed content into time-grounded, reusable factual records via PHCM
• Timely response: learning when available evidence warrants a response through PSTL

Key contributions:

1. PHCM complements a recent visual window with hierarchical caption memory, improving historical QA without degrading real-time perception
2. PSTL outperforms dense state supervision while supervising fewer state tokens, demonstrating that selective supervision of state changes and persistence is more effective
3. OneStreamer-1M provides a broad-coverage training corpus with evidence-aligned content and timing

The 4B model achieves state-of-the-art results across all eight streaming video understanding benchmarks, with an average relative improvement of 25.0% over the Qwen3-VL baseline.

Future directions may include scaling to larger models, extending to additional task families, exploring adaptive memory granularity, and investigating reinforcement learning approaches for response timing policies.