Summary (Overview)

• Kwai Keye-VL-2.0-30B-A3B is an open-source Mixture-of-Experts (MoE) multimodal foundation model with 30B total parameters and only 3B active parameters, designed for long-video understanding and agentic intelligence.
• It is the first model to adapt DeepSeek Sparse Attention (DSA) to a GQA-based multimodal architecture, enabling lossless 256K context processing while capturing critical frames and long-range temporal dependencies.
• Introduces Cross-Modal Multi-Teacher On-Policy Distillation (MOPD) paired with Context-RL and Video-RL to overcome catastrophic forgetting during multi-task alignment, allowing the model to natively support agent collaboration across Code, Tool, and Search scenarios.
• Achieves state-of-the-art performance among models of similar scale, particularly excelling in fine-grained temporal localization on TimeLens benchmarks (ActivityNet, QVHighlights, Charades) and long-video comprehension on Video-MME-v2 and LongVideoBench.
• The model, checkpoints, and infrastructure are fully open-sourced via Hugging Face and GitHub to accelerate community progress.

Introduction and Theoretical Foundation

Background and Motivation

Recent large language models (e.g., GPT-5.5, Claude Opus 4.8, Gemini 3.5 Flash, Qwen3.7) have made substantial progress in multimodal reasoning and long-context understanding. However, scaling to hour-level video contexts incurs prohibitive computational and memory costs (the infrastructural bottleneck), and integrating complex agent tasks often induces catastrophic forgetting of foundational reasoning capabilities (the algorithmic dilemma).

Theoretical Basis

Keye-VL-2.0 addresses these challenges through two paradigm-shifting innovations:

1. Extreme Context Scaling via Multimodal DSA: Standard dense attention leads to catastrophic KV cache expansion. DSA compresses and sparsifies video feature aggregation, constraining linear KV cache growth and enabling 256K lossless context processing.
2. Resolving Modality Conflict via Cross-Modal MOPD: Direct injection of multiple capabilities often degrades reasoning. MOPD uses specialized teacher models to provide dense token-level feedback on student-generated trajectories, isolating task-specific expertise and preserving general-purpose reasoning baselines.

Methodology

Model Architecture

The model comprises four core components:

• Vision Encoder (ViT): Inherited from Keye-VL-1.5, based on SigLIP-400M-384-14, with native-resolution encoding, 2D RoPE, and adaptive position encoding.
• Language Decoder (LLM): Based on Qwen3-30B-A3B-Thinking-2507.
• MLP Projector: Randomly initialized and trained to align visual features.
• Sparse Attention Module: A GQA-compatible DSA design combining MQA-based indexing with grouped GQA aggregation.

DSA for Long-Context Multimodal Modeling

The DSA contains a Lightning Indexer and a fine-grained token selection mechanism. For throughput, the indexer follows an MQA key-sharing design:

$$I_{t,s} = \sum_{j=1}^{H_I} w_{t,j}^I \cdot \text{ReLU}(q_{t,j}^I \cdot k_s^I)$$

where $H_I$ is the number of indexer heads, $q_{t,j}^I$ and $w_{t,j}^I$ are derived from the current query token $h_t$, and $k_s^I$ is the shared key from the preceding token $h_s$. The Top-$k$ tokens form the sparse index set:

$$\Omega_t = \{ s \mid I_{t,s} \in \text{Top-}k(I_{t,:}) \}$$

In the GQA backbone, the sparse attention output for the $g$-th group is:

$$u_{t,g} = \text{Attn}(h_{t,g}, \{c_{s,g} \mid s \in \Omega_t\})$$

With $k = 2048$, core attention complexity reduces from $O(L^2)$ to $O(Lk)$ where $k \ll L$.

Two-Stage DSA Training

1. Dense Warm-up: Initializes the indexer using a KL divergence loss:

   $$\mathcal{L}_{\text{warmup}}^I = \sum_t \sum_{g=1}^G D_{\text{KL}}(p_{t,:,g} \parallel \text{Softmax}(I_{t,:}))$$

2. Sparse Adaptation: All parameters unfrozen, training switches to sparse mode:

   $$\mathcal{L}_{\text{sparse}}^I = \sum_t \sum_{g=1}^G D_{\text{KL}}(p_{t,S_t,g} \parallel \text{Softmax}(I_{t,S_t}))$$

   The total loss is:

   $$\mathcal{L}_{\text{total}} = \mathcal{L}_{\text{NTP}} + \lambda \mathcal{L}_{\text{sparse}}^I$$

Pre-Training Pipeline (Four-Stage Curriculum)

Post-Training

Reinforcement Learning

General RL uses Group Sequence Policy Optimization (GSPO):

$$J_{\text{GSPO}}(\theta) = \mathbb{E}_{x \sim \mathcal{D}, \{y_i\}_{i=1}^G \sim \pi_{\theta_{\text{old}}}(\cdot|x)} \left[ \frac{1}{G} \sum_{i=1}^G \min(s_i(\theta) \hat{A}_i, \text{clip}(s_i(\theta), 1-\epsilon, 1+\epsilon) \hat{A}_i) \right]$$

where $s_i(\theta)$ is an importance sampling ratio and $\hat{A}_i$ is the normalized advantage.

Specialized RL covers Grounding, Spatial, Math, Counting, and OCR experts using domain-specific rewards (e.g., IoU for grounding, symbolic equivalence for math).

Video RL uses temporal IoU (tIoU) rewards and LLM-as-Judge for dense captioning.

Agentic RL covers Coding RL (Online Judge, SWE-bench), Tool Use RL (150+ API domains), and Search RL with environment-grounded rewards.

Cross-Modal Multi-Teacher On-Policy Distillation (MOPD)

MOPD maintains 13 RL-trained domain teachers. For a student on-policy response $y_i = (y_{i,1}, \ldots, y_{i,T}) \sim \pi_\theta(\cdot|x_i)$, the token-level advantage is computed over a top-$k$ overlap set $\Omega_{i,t}$:

$$A_{i,t} = \sum_{v \in \Omega_{i,t}} \bar{\pi}_\theta(v|s_{i,t}) \left[ \log \pi_T^{(r(i))}(v|s_{i,t}) - \log \pi_\theta(v|s_{i,t}) \right]$$

The student is optimized with:

$$\mathcal{L}_{\text{MOPD}} = -\mathbb{E} \left[ \frac{1}{|\mathcal{M}_i|} \sum_{t \in \mathcal{M}_i} b\hat{A}_{i,t} \log \pi_\theta(y_{i,t}|x_i, y_{i,<t}) \right]$$

with token-category-aware advantage scaling and localized repetition penalties.

Efficient Infrastructure

• ViT–LM Heterogeneous Parallelism: Co-located on same GPUs with separate sharding strategies; recompute-or-offload reduces ViT activation memory to near zero.
• DSA Optimization: FlashInfer and TileLang achieve >2× speedup; top-$k$ memory optimization and short-sequence optimization for variable-length inputs.
• Inference Efficiency: Chunk ViT, sparse attention deduplication, and decode optimization reduce prefill cost by >3× and decode cost by >5× at 128K context.

Empirical Validation / Results

Video Understanding

Table 2: Video Understanding Evaluation

Keye-VL-2.0 achieves best on LongVideoBench (74.1), Video-MME-v2 ACC (42.4 at 512 frames), and all three TimeLens subsets (ActivityNet, QVHighlights, Charades), demonstrating superior long-video comprehension and fine-grained temporal localization.

Agentic Capability Evaluation

Table 3: Code Agent Evaluation

Table 4: Tool-Use Evaluation

Keye-VL-2.0 shows strong coding (64.2 on LiveCodeBench v6) and tool-use abilities (best on τ²-Bench and VitaBench), with only 3B active parameters.

Theoretical and Practical Implications

• Sparse Long-Context Modeling: Demonstrates that DSA can be effectively adapted to GQA-based multimodal architectures, enabling 256K context processing at controllable cost. This provides a practical path for scaling video understanding beyond frame-limited perception.
• Capability Consolidation via MOPD: MOPD resolves the multimodal alignment dilemma by isolating task-specific expertise through on-policy distillation, preventing catastrophic forgetting while integrating heterogeneous agent capabilities.
• Deployable Efficiency: With only 3B active parameters and optimized DSA kernels, the model achieves high inference efficiency, making hour-level video understanding feasible for real applications.
• Open-Source Impact: Full release of model checkpoints and infrastructure (ViT-LM parallelism, DSA kernels, MOPD system) accelerates community progress toward scalable multimodal agentic applications.

Conclusion

Kwai Keye-VL-2.0-30B-A3B is a 30B MoE multimodal foundation model (3B active) that pioneers DSA in GQA-based architectures for 256K context processing. Its post-training pipeline, featuring Cross-Modal Multi-Teacher On-Policy Distillation and diverse RL stages, enables leading performance in long-video understanding, temporal grounding, and agentic tasks without sacrificing general reasoning. Future work will focus on deeper integration into real business pipelines (generative recommendation, content governance), developing Video × Agent workflows for automated orchestration, and strengthening the underlying infrastructure toward native multimodal modeling. The open-source release aims to turn long-context multimodal intelligence into reliable, scalable infrastructure for real-world applications.