Omni Interaction Agent Technical Report

Summary (Overview)

• Gander is an end-to-end model that unifies omni perception, realtime interaction, and agentic capabilities within a single framework, moving beyond turn-based paradigms to enable natural full-duplex interaction.
• The architecture employs a Cerebellum-Brain collaborative framework: the Cerebellum handles realtime interaction and omni conversation, while the Brain (training-free, plug-and-play) manages complex reasoning and long-horizon agentic tasks.
• The Cerebellum uses a streaming Thinker-Talker architecture with chunk-level flattening, where all modalities (video, speech, text) are organized into a unified autoregressive token stream, enabling the model to dynamically predict whether to listen, speak, or interrupt.
• Gander achieves best-in-class turn-taking behavior on Full-Duplex-Bench v3 (100% appropriate turn-taking, 8.0% premature interruptions vs. 13.5% for GPT-Realtime), while maintaining competitive spoken dialogue and omni understanding capabilities.
• The model, code, and data are released to the community to facilitate further research in Omni Interaction Agents.

---

Introduction and Theoretical Foundation

Background and Motivation

The paper addresses two fundamental limitations of current AI systems:

1. Interaction paradigm gap: Human–AI interaction remains largely text-based and turn-by-turn, which is fundamentally different from the fluid, collaborative nature of human–human communication where participants continuously exchange information across modalities, listen and speak concurrently, and flexibly interrupt or redirect ongoing interactions.

2. Two-tier capability trade-off: A single monolithic model faces an inherent tension between low-latency responsiveness (required for realtime interaction) and long-horizon reasoning (required for complex agentic tasks).

Key Theoretical Questions

The authors pose two central questions:

1. Can truly interactive omni communication be achieved by composing conventional perception and interaction modules, such as VAD and ASR, or must interactivity itself be an intrinsic capability of the model?
2. Can a single model simultaneously provide the low-latency responsiveness required for realtime interaction and the long-horizon reasoning required for complex agentic tasks?

Core Design Principles

Principle 1 — Interactivity must be intrinsic: Robust interactivity cannot be treated as an external orchestration layer. Real-world interaction includes spontaneous interruptions, proactive agent-initiated engagement, noisy environments, multi-party conversations, and backchannel acknowledgments (e.g., "uh-huh," "right," "I see"). The model must learn to dynamically control its interaction state and coordinate perception with generation in realtime.

Principle 2 — Brain–Cerebellum decoupling: Following recent work (Huang et al., 2026; Lab, 2026; Wu et al., 2025), the architecture separates realtime interaction (Cerebellum) from asynchronous long-horizon reasoning (Brain). The Cerebellum selectively invokes the Brain through tool calls, and the Brain returns intermediate summaries, plans, or final results.

---

Methodology

3.1 Architecture Overview

Gander consists of three components:

1. Front Cerebellum: A realtime, full-duplex multimodal model (Thinker-Talker architecture) responsible for continuous multimodal perception and interactive communication.
2. Back Brain: A training-free, general-purpose task execution agent (e.g., Codex, Claude Code) for complex reasoning and long-horizon workflows.
3. Agent Orchestration Runtime: The coordination layer managing realtime multimodal inference and asynchronous background task orchestration.

3.2 Cerebellum-Brain Collaborative Framework

Tool Call Interface

The front cerebellum emits structured tool invocations:

<tool_call> {"name": "tool_state", "arguments": {"key": "value"}} </tool_call>

Three primary operations are defined:

Agent Orchestration Runtime

The runtime supports two control modes:

• Lean mode: Directly executes task actions classified by the front cerebellum; shorter control path, lower latency, greater determinism.
• Coordinator mode: An independent control plane model generates execution directives (reasoning intensity, permission policy, delivery strategy); decouples planning from realtime interaction.

The Gateway manages five persistent entities: Project, Task, Run, WorkerEvent, Delivery.

Training-Free Back Brain

The back brain can invoke three runtime-specific interfaces:

• context_fetch: Retrieves realtime task context and artifacts
• memory_search: Retrieves persistent multimodal memories across sessions
• share: Returns verified findings to the front cerebellum

3.3 Streaming Thinker-Talker Architecture

Omni Perception

Visual perception: Uses any-resolution partitioning (Guo et al., 2024) with a SigLIP vision transformer and query-based resampler, achieving ~16× compression (vs. 4× in prior models). Operates at up to 448×448 resolution.

Acoustic perception: Uses a streaming chunk-wise speech encoder with 5× temporal downsampling via an MLP projector, yielding ~10 tokens/second entering the backbone.

Streaming Chunk Flattening

The interaction is partitioned into fixed 1-second windows, each forming one chunk:

$$\text{Chunk} = [\text{Audio tokens} \oplus \text{Visual tokens} \oplus \text{Control token} \oplus N \text{ text tokens}]$$

Where $N$ is determined by the model (may be zero).

Control tokens take three values:

1. Listen: Remain silent, continue observing
2. Speak: Generate spoken content
3. Interrupt: Halt an ongoing utterance when context warrants

Context management: A sliding window of 128 chunks (~2 minutes of interaction) keeps per-step inference cost stable.

Speech Generation

The backbone stays in the text domain; speech is produced by two decoupled stages:

1. Speech token decoder: Produces discrete speech units from backbone hidden states
2. Streaming flow matching decoder: Renders waveform chunks incrementally, enabling zero-shot voice control via reference audio

4. Data Construction Pipeline

The training corpus (2.7M examples total) is organized into four families:

InteractionSpeech pipeline (four stages):

1. Dialogue sources: 11.2K topic seeds expanded via DeepSeek-V4-Pro; existing multi-turn dialogues filtered for spoken suitability
2. Interaction events: Competitive interruption (user barges in) and supportive backchannel (brief acknowledgment); backchannels validated by length (<8 chars Chinese, <6 words English) and lexicon matching (266 Chinese + 170 English expressions)
3. Rendering: User channel rendered via voice cloning TTS with explicit timing supervision
4. Quality control: Rule-based gate + LLM judge on naturalness, coherence, interruption plausibility, backchannel plausibility

---

Empirical Validation / Results

5.2 Full-Duplex Interaction (Full-Duplex-Bench v3)

Table 3: Full-Duplex-Bench v3 Results (100 scenarios)

Key findings:

• Gander achieves best turn-taking behavior: 100% appropriate turn-taking (matched only by cascaded), and lowest premature interruption rate (8.0% vs. 13.5% for GPT-Realtime)
• The back brain alone (text-driven) achieves the highest tool selection (0.934) and response quality (0.740), indicating the execution tier is not the limiting factor
• The accuracy gap in end-to-end mode is attributed to the front cerebellum's delegation decisions and ASR error on synthesized speech

5.3 Spoken Conversation (SpokenQA & VoiceBench)

Table 4: SpokenQA and VoiceBench Results

Key findings:

• Gander leads the full-duplex group on knowledge-oriented subsets (75.60 and 59.30 on SpokenQA)
• A frame-synchronous model matches turn-based models on spoken factual QA, showing the streaming formulation does not inherently sacrifice knowledge retention
• The back brain was never invoked on these 2,052 samples, confirming selective routing behavior

5.4 Omni Understanding (WorldSense & Daily-Omni)

Table 5: Omni Understanding Accuracy (%)

Table 6: Modality Ablation (Fusion Gain)

Key findings:

• Gander retains competitive omni understanding despite interaction-only training, staying within 1.67 points of its MiniCPM-o 4.5 initialization on Daily-Omni
• The fusion gain (+8.88 overall) confirms genuine multimodal integration rather than reliance on a dominant modality
• The asymmetry between benchmarks (Daily-Omni −1.67 vs. WorldSense −6.08) shows interaction training preserves temporal reasoning over audio-visual alignment while sacrificing fine-grained perceptual attribute inspection

---

Theoretical and Practical Implications

Theoretical Contributions

1. Interactivity as an intrinsic capability: Gander demonstrates that interaction decisions (listen/speak/interrupt) can be learned end-to-end within a unified autoregressive framework, challenging the pipeline-based approach that treats interactivity as an external orchestration layer.

2. Brain–Cerebellum decoupling: The architecture provides evidence that realtime interaction and long-horizon reasoning can be effectively separated and coordinated, with the training-free back brain offering a scalable pathway — improvements in reasoning models can be directly propagated without retraining the interaction model.

3. Chunk-level flattening: The streaming chunk paradigm shows that audio, visual, and text modalities can be unified on a shared temporal axis, enabling the model to maintain continuous temporal awareness and make proactive interaction decisions.

Practical Implications

1. Deployment readiness: The 9B-parameter Cerebellum achieves competitive or superior interaction behavior against much larger commercial systems, making realtime omni interaction feasible for practical deployment.

2. Industrial applicability: The architecture is being scaled in Tencent's industrial Hy-Realtime model, suggesting practical viability in production environments.

3. Open research: Release of model weights, code, and data enables community reproduction and further advancement in Omni Interaction Agents.

---

Conclusion

Main Takeaways

Gander provides an initial foundation for a unified Omni Interaction Agent, demonstrating that:

• End-to-end omni interaction with intrinsic interactivity is achievable
• Brain–Cerebellum decoupling effectively resolves the responsiveness-intelligence trade-off
• Interaction training preserves most omni understanding capabilities while adding full-duplex interaction skills

Future Directions

1. Data and Model Scaling: Agent invocation and conversational behavior remain sensitive to training data distribution; scaling is critical for robustness and generalization.

2. Stable Post-Training: On-policy distillation (OPD) and reinforcement learning (RL) for long-horizon Omni Interaction Agent scenarios remain open challenges, particularly reward design and credit assignment under joint multimodal interaction and agentic execution.

3. Brain–Cerebellum Architectural Exploration: Current communication relies primarily on ASR-derived signals; richer bidirectional communication (Brain→Cerebellum and Cerebellum→Brain) needs exploration.

4. Memory and Long Context Management: Efficient mechanisms for retaining and retrieving task-relevant information over extended interactions are needed.

5. Evaluation: Existing benchmarks evaluate omni understanding, duplex interaction, and agentic execution in isolation; a unified evaluation framework for omni interaction agents is largely missing.

Final Assessment

"Gander provides an initial foundation for a unified omni interaction agent architecture. Further progress toward reliable long-horizon deployment depends on advances in scaling, post-training, Brain–Cerebellum coordination, memory, and evaluation, especially for sustained realtime interaction and complex task execution."