# Kimi K3: Open Frontier Intelligence > Kimi K3, a 2.8T-parameter MoE model with 104B active parameters, achieves frontier performance across coding, reasoning, and vision tasks via architectural innovations yielding 2.5x scaling efficiency over its predecessor. - **Source:** [arXiv](https://arxiv.org/abs/2607.24653) - **Published:** 2026-07-29 - **Permalink:** https://picx.dev/p/048q6D ## Summary # Kimi K3: Open Frontier Intelligence - Technical Report Summary ## Summary (Overview) - **Model Scale**: Kimi K3 is a 2.8T parameter Mixture-of-Experts (MoE) model with 104B activated parameters, native vision capabilities, and a 1-million-token context window, representing a significant leap in open-source frontier intelligence. - **Architectural Innovation**: The model introduces three key architectural advances—Kimi Delta Attention (KDA) for efficient long-sequence mixing, Attention Residuals (AttnRes) for selective information flow across depth, and Stable LatentMoE with 896 routed experts (16 active per token)—yielding approximately **2.5× improvement in scaling efficiency** over Kimi K2. - **Post-Training Excellence**: The model employs reinforcement learning across general, agentic, and coding domains with multiple reasoning-effort levels, consolidated through Multi-Teacher On-Policy Distillation (MOPD), enabling compositional generalization and robust long-horizon execution. - **Frontier Performance**: Kimi K3 achieves frontier-level performance across long-horizon coding, agentic, knowledge, reasoning, and vision tasks, consistently outperforming other open and proprietary models (except Claude Fable 5 and GPT-5.6 Sol) in the evaluated suite. - **Open Release**: The full Kimi K3 model weights are released to the public, making frontier intelligence accessible for research and deployment. --- ## Introduction and Theoretical Foundation ### Background and Motivation The paper addresses a critical gap in the evolution of large language models. While scaling has traditionally meant investing more computation in pre-training (the "first axis"), the rise of reasoning models has established test-time computation as a "second axis" of scaling. Key developments include: - **OpenAI's o-series**: Scales reinforcement learning and test-time reasoning - **Anthropic's extended-thinking models**: Allocate adaptive thinking budgets and interleave reasoning with tool use - **DeepSeek-R1 and Kimi K1.5**: Demonstrate large-scale RL can elicit sophisticated reasoning behaviors - **Kimi K2.5 Agent Swarm**: Extends test-time scaling from sequential reasoning to parallel agent coordination However, the open-source ecosystem has progressed slowly on the first axis, with many recent models remaining within or slightly above the 1T-class parameter regime. Kimi K3 pursues **both scaling axes together**: scaling the pre-trained foundation to unprecedented 3T-class parameters while scaling RL, reasoning effort, and long-horizon interaction at 1M context length. ### Theoretical Foundation The architecture is founded on three complementary information-flow dimensions: 1. **Sequence Length**: Hybrid Attention combining KDA with Gated MLA 2. **Network Depth**: Attention Residuals for selective cross-layer retrieval 3. **Model Width**: Stable LatentMoE for sparse channel mixing The theoretical motivation draws from the insight that standard residual connections compress all prior information into a single state—a bottleneck reminiscent of RNNs over time. AttnRes applies the attention mechanism to depth, allowing each layer to selectively access all preceding layers with data-dependent weights. --- ## Methodology ### 2.1 Hybrid Attention Architecture Each block contains **3 KDA layers followed by 1 Gated MLA layer** (3:1 mixing ratio), with an additional Gated MLA at the backbone's end for final global attention. #### Kimi Delta Attention (KDA) KDA extends the delta-rule recurrence with a channel-wise forget gate: $$S_t = \left(I - \beta_t k_t k_t^\top\right) \text{Diag}(\alpha_t) S_{t-1} + \beta_t k_t v_t^\top, \quad \tilde{o}_t = S_t^\top q_t$$ where $\alpha_t \in (0,1)^{d_k}$ is the channel-wise one-step retention factor, and $\beta_t \in (0,1)$ controls the delta-rule write strength. **Key innovations over Kimi Linear:** 1. **Lower-bounded decay**: Uses a scaled sigmoid instead of negative-Softplus: $$g_t^h = g_{\min} \text{Sigmoid}\left(e^{A^h} z_t^h\right) \in (g_{\min}, 0)^{d_k}, \quad \alpha_t^h = \exp(g_t^h) \in (e^{g_{\min}}, 1)^{d_k}$$ With $g_{\min} = -5$, all retention factors satisfy $\alpha > e^{-5} \approx 6.7 \times 10^{-3}$, keeping the cumulative log-decay within BF16 dynamic range. This allows **all causal tiles to use dense Tensor Core matrix multiplications**, eliminating the position-pair diagonal path bottleneck. 2. **Full-rank output gate**: $$y_t = W_o [\text{Sigmoid}(W_g x_t) \odot \text{RMSNorm}(\tilde{o}_t)]$$ #### Gated MLA Retains Multi-head Latent Attention with No Position Encoding (NoPE), augmented with input-dependent full-rank output gating: $$y_t = W_o [\text{Sigmoid}(W_g x_t) \odot \tilde{o}_t]$$ ### 2.2 Attention Residuals AttnRes applies attention mechanisms across network depth. For each layer $l$, a learnable pseudo-query $q_l = w_l$ attends over preceding layer outputs: $$\alpha_{i \rightarrow l} = \frac{\phi(q_l, k_i)}{\sum_{j=0}^{l-1} \phi(q_l, k_j)}, \quad h_l = \sum_{i=0}^{l-1} \alpha_{i \rightarrow l} \cdot v_i$$ where $\phi(q, k) = \exp(q^\top \text{RMSNorm}(k))$. **Block Attention Residuals**: Partitions layers into blocks, reducing memory overhead from $O(Ld)$ to $O(Nd)$ where $N$ is the number of blocks. Kimi K3 uses **8 blocks of 12 layers each**, with the embedding layer counted as an additional source. ### 2.3 Stable LatentMoE The routed path separates full model width from compact latent space: $$u = \sum_{i \in \mathcal{T}_k(x)} p_i E_i^{\text{routed}}(W_\downarrow x), \quad y = \sum_{j=1}^{N_s} E_j^{\text{shared}}(x) + W_\uparrow \text{RMSNorm}(u)$$ **Three stabilizing components:** 1. **Normalized LatentMoE**: RMSNorm between expert aggregation and up-projection 2. **SiTU-GLU activation**: Bounded activation with soft caps: $$\text{SiTU-GLU}(x) = \left[\beta_1 \tanh\left(\frac{W_g x}{\beta_1}\right) \odot \text{Sigmoid}(W_g x)\right] \odot \left[\beta_2 \tanh\left(\frac{W_u x}{\beta_2}\right)\right]$$ with $\beta_1 = 4$, $\beta_2 = 25$—preserves SwiGLU's local response while bounding both factors. 3. **Quantile Balancing**: Sets expert biases from router-score quantiles: $$\hat{b}_j^{(t+1)} \leftarrow -\text{quantile}_{1-k/n}\left(s_{:,j} - \alpha^{(t)}\right), \quad b^{(t+1)} \leftarrow \hat{b}^{(t+1)} - \text{mean}(\hat{b}^{(t+1)})\mathbf{1}$$ ### 2.4 Native Vision (MoonViT-V2) - 27-layer vision transformer (~0.4B parameters) trained **from scratch with next-token prediction** - No contrastive pre-training (SigLIP) needed—achieves better training stability with lower gradient norms - Fully shared parameters for images and videos with factorized attention - Pixel-shuffle 2×2 downsampling reduces visual tokens by 4× - Handles inputs up to 3584×3584 pixels ### 2.5 Per-Head Muon Orthogonalizes momentum matrices per-head for attention projections, equalizing update scales across heads and improving training stability. ### 4.1 Post-Training Pipeline Three-stage paradigm: **SFT → Domain-Specialized RL → MOPD** #### Reinforcement Learning - Three domains × three reasoning effort levels = **9 expert models** - Partial rollout scheme with fraction $\lambda \in (0,1)$ of trajectories completing before policy optimization - Per-problem budget control with token budget $b_0(x)$ and multiplier $\tau$ for reasoning effort calibration - Agentic Generative Reward Model with mandatory protocol (read → generate rubric → score → record) #### Multi-Teacher On-Policy Distillation (MOPD) Per-token OPD reward: $$r_{opd}^d(y_t | e, x, y_{