Kimi K3: Open Frontier Intelligence - Technical Report Summary

Summary (Overview)

• Model Scale: Kimi K3 is a 2.8T parameter Mixture-of-Experts (MoE) model with 104B activated parameters, native vision capabilities, and a 1-million-token context window, representing a significant leap in open-source frontier intelligence.

• Architectural Innovation: The model introduces three key architectural advances—Kimi Delta Attention (KDA) for efficient long-sequence mixing, Attention Residuals (AttnRes) for selective information flow across depth, and Stable LatentMoE with 896 routed experts (16 active per token)—yielding approximately 2.5× improvement in scaling efficiency over Kimi K2.

• Post-Training Excellence: The model employs reinforcement learning across general, agentic, and coding domains with multiple reasoning-effort levels, consolidated through Multi-Teacher On-Policy Distillation (MOPD), enabling compositional generalization and robust long-horizon execution.

• Frontier Performance: Kimi K3 achieves frontier-level performance across long-horizon coding, agentic, knowledge, reasoning, and vision tasks, consistently outperforming other open and proprietary models (except Claude Fable 5 and GPT-5.6 Sol) in the evaluated suite.

• Open Release: The full Kimi K3 model weights are released to the public, making frontier intelligence accessible for research and deployment.

---

Introduction and Theoretical Foundation

Background and Motivation

The paper addresses a critical gap in the evolution of large language models. While scaling has traditionally meant investing more computation in pre-training (the "first axis"), the rise of reasoning models has established test-time computation as a "second axis" of scaling. Key developments include:

• OpenAI's o-series: Scales reinforcement learning and test-time reasoning
• Anthropic's extended-thinking models: Allocate adaptive thinking budgets and interleave reasoning with tool use
• DeepSeek-R1 and Kimi K1.5: Demonstrate large-scale RL can elicit sophisticated reasoning behaviors
• Kimi K2.5 Agent Swarm: Extends test-time scaling from sequential reasoning to parallel agent coordination

However, the open-source ecosystem has progressed slowly on the first axis, with many recent models remaining within or slightly above the 1T-class parameter regime. Kimi K3 pursues both scaling axes together: scaling the pre-trained foundation to unprecedented 3T-class parameters while scaling RL, reasoning effort, and long-horizon interaction at 1M context length.

Theoretical Foundation

The architecture is founded on three complementary information-flow dimensions:

1. Sequence Length: Hybrid Attention combining KDA with Gated MLA
2. Network Depth: Attention Residuals for selective cross-layer retrieval
3. Model Width: Stable LatentMoE for sparse channel mixing

The theoretical motivation draws from the insight that standard residual connections compress all prior information into a single state—a bottleneck reminiscent of RNNs over time. AttnRes applies the attention mechanism to depth, allowing each layer to selectively access all preceding layers with data-dependent weights.

---

Methodology

2.1 Hybrid Attention Architecture

Each block contains 3 KDA layers followed by 1 Gated MLA layer (3:1 mixing ratio), with an additional Gated MLA at the backbone's end for final global attention.

Kimi Delta Attention (KDA)

KDA extends the delta-rule recurrence with a channel-wise forget gate:

$$S_t = \left(I - \beta_t k_t k_t^\top\right) \text{Diag}(\alpha_t) S_{t-1} + \beta_t k_t v_t^\top, \quad \tilde{o}_t = S_t^\top q_t$$

where $\alpha_t \in (0,1)^{d_k}$ is the channel-wise one-step retention factor, and $\beta_t \in (0,1)$ controls the delta-rule write strength.

Key innovations over Kimi Linear:

1. Lower-bounded decay: Uses a scaled sigmoid instead of negative-Softplus:

$$g_t^h = g_{\min} \text{Sigmoid}\left(e^{A^h} z_t^h\right) \in (g_{\min}, 0)^{d_k}, \quad \alpha_t^h = \exp(g_t^h) \in (e^{g_{\min}}, 1)^{d_k}$$

With $g_{\min} = -5$, all retention factors satisfy $\alpha > e^{-5} \approx 6.7 \times 10^{-3}$, keeping the cumulative log-decay within BF16 dynamic range. This allows all causal tiles to use dense Tensor Core matrix multiplications, eliminating the position-pair diagonal path bottleneck.

2. Full-rank output gate:

$$y_t = W_o [\text{Sigmoid}(W_g x_t) \odot \text{RMSNorm}(\tilde{o}_t)]$$

Gated MLA

Retains Multi-head Latent Attention with No Position Encoding (NoPE), augmented with input-dependent full-rank output gating:

$$y_t = W_o [\text{Sigmoid}(W_g x_t) \odot \tilde{o}_t]$$

2.2 Attention Residuals

AttnRes applies attention mechanisms across network depth. For each layer $l$, a learnable pseudo-query $q_l = w_l$ attends over preceding layer outputs:

$$\alpha_{i \rightarrow l} = \frac{\phi(q_l, k_i)}{\sum_{j=0}^{l-1} \phi(q_l, k_j)}, \quad h_l = \sum_{i=0}^{l-1} \alpha_{i \rightarrow l} \cdot v_i$$

where $\phi(q, k) = \exp(q^\top \text{RMSNorm}(k))$.

Block Attention Residuals: Partitions layers into blocks, reducing memory overhead from $O(Ld)$ to $O(Nd)$ where $N$ is the number of blocks. Kimi K3 uses 8 blocks of 12 layers each, with the embedding layer counted as an additional source.

2.3 Stable LatentMoE

The routed path separates full model width from compact latent space:

$$u = \sum_{i \in \mathcal{T}_k(x)} p_i E_i^{\text{routed}}(W_\downarrow x), \quad y = \sum_{j=1}^{N_s} E_j^{\text{shared}}(x) + W_\uparrow \text{RMSNorm}(u)$$

Three stabilizing components:

1. Normalized LatentMoE: RMSNorm between expert aggregation and up-projection
2. SiTU-GLU activation: Bounded activation with soft caps:

$$\text{SiTU-GLU}(x) = \left[\beta_1 \tanh\left(\frac{W_g x}{\beta_1}\right) \odot \text{Sigmoid}(W_g x)\right] \odot \left[\beta_2 \tanh\left(\frac{W_u x}{\beta_2}\right)\right]$$

with $\beta_1 = 4$, $\beta_2 = 25$—preserves SwiGLU's local response while bounding both factors.

3. Quantile Balancing: Sets expert biases from router-score quantiles:

$$\hat{b}_j^{(t+1)} \leftarrow -\text{quantile}_{1-k/n}\left(s_{:,j} - \alpha^{(t)}\right), \quad b^{(t+1)} \leftarrow \hat{b}^{(t+1)} - \text{mean}(\hat{b}^{(t+1)})\mathbf{1}$$

2.4 Native Vision (MoonViT-V2)

• 27-layer vision transformer (~0.4B parameters) trained from scratch with next-token prediction
• No contrastive pre-training (SigLIP) needed—achieves better training stability with lower gradient norms
• Fully shared parameters for images and videos with factorized attention
• Pixel-shuffle 2×2 downsampling reduces visual tokens by 4×
• Handles inputs up to 3584×3584 pixels

2.5 Per-Head Muon

Orthogonalizes momentum matrices per-head for attention projections, equalizing update scales across heads and improving training stability.

4.1 Post-Training Pipeline

Three-stage paradigm: SFT → Domain-Specialized RL → MOPD

Reinforcement Learning

• Three domains × three reasoning effort levels = 9 expert models
• Partial rollout scheme with fraction $\lambda \in (0,1)$ of trajectories completing before policy optimization
• Per-problem budget control with token budget $b_0(x)$ and multiplier $\tau$ for reasoning effort calibration
• Agentic Generative Reward Model with mandatory protocol (read → generate rubric → score → record)

Multi-Teacher On-Policy Distillation (MOPD)

Per-token OPD reward:

$$r_{opd}^d(y_t | e, x, y_{<t}) = \text{clip}\left(\text{sg}\left[\log \frac{\pi_{\text{teacher}}^{(d,e)}(y_t | x, y_{<t})}{\pi_\theta(y_t | e, x, y_{<t})}\right], -R_{\max}, R_{\max}\right)$$

Deployment-Aware Post-Training

• MXFP4 Quantization-Aware Training: Expert weights in MXFP4, activations in MXFP8
• Draft Model Fine-Tuning: MTP layer converted to EAGLE-3-style draft model, optimized with LK loss:

$$\mathcal{L}_{LK} = -\log \sum_{x \in \mathcal{V}} \min(p(x), q(x))$$

4.2 Task Synthesis Environments

• Unified White-Box RL Environment: Configurable agent harnesses (Kimi Code, Claude Code, Codex, etc.)
• Knowledge-Graph-Guided Task Synthesis: Self-evolving hierarchical knowledge graph for controlled data coverage
• Kernel Optimization Tasks: Multi-framework GPU programming (CUDA, Triton, CuTe, etc.)
• Personal Assistant Tasks: Mock implementations of Gmail, Notion, Slack with persistent evolving environments
• Autonomous Execution Tasks (AET): Verify-in-the-loop optimization with black-box system replication
• Web Development Tasks: Containerized sandboxes with deterministic and model-judged rewards

---

Empirical Validation / Results

Benchmark Performance (Figure 1)

Key Results

• 2.5× scaling efficiency improvement over Kimi K2 (Figure 7)
• KDA lower-bounded decay eliminates diagonal-tile bottleneck
• From-scratch MoonViT-V2 matches SigLIP-initialized baseline with better stability
• RL scaling improves capabilities across all domains with increased tool-call steps (Figure 8)

Architectural Comparison (Table 1)

---

Theoretical and Practical Implications

Infrastructure Innovations

1. KDA Systems Co-Design:

   • FlashKDA: CUTLASS-based chunkwise kernel overlapping intra-chunk computation with cross-chunk state propagation
   • KDA Context Parallelism (KCP): Decomposes segment effects into locally computable cumulative transitions and zero-initiated states, requiring only fixed-size all-gather for synchronization

2. MoonEP: Perfectly balanced expert-parallel training with provable guarantees (≤ E/R redundant experts per rank), zero-copy communication, and sync-free static-shape execution

3. Memory-Efficient Training: Unified activation manager with pluggable storage backends, mathematical transformation eliminating backward dependency on output for MoE gradients

4. Million-Token Agentic RL: Co-located system with partial rollouts, external KV-cache retention, and resumable microVM sandboxes

Significance

• Open Frontier: First open 3T-class model with 1M context, making frontier intelligence accessible
• Architectural Blueprint: KDA + AttnRes + Stable LatentMoE provides a template for scaling information flow across all dimensions
• Training Stability at Extreme Scale: SiTU-GLU, Quantile Balancing, and from-scratch vision training address previously unsolved stability challenges at 3T scale
• Efficient Deployment: QAT with MXFP4 and EAGLE-3 draft model integration reduce serving costs

---

Conclusion

Kimi K3 represents a major milestone in open frontier intelligence, demonstrating that both pre-training and test-time scaling axes can be pursued simultaneously to the frontier. Key contributions include architectural innovations (KDA, AttnRes, Stable LatentMoE), infrastructure advances (FlashKDA, KCP, MoonEP), and post-training methodologies (multi-domain RL with effort levels, MOPD).

Future Directions:

• Closing the remaining gap to the most powerful proprietary systems (Claude Fable 5, GPT-5.6 Sol)
• Further scaling of both axes
• Continued improvement in long-horizon agentic capabilities
• Broader adoption through the open release of full model weights

The paper demonstrates that open-source models can achieve frontier performance while maintaining competitive efficiency, setting a new standard for the open ecosystem.