# Motif 3: Technical Report > Motif 3, a 314B-parameter MoE model with 13.2B active parameters, matches frontier performance using grouped differential latent attention and multi-teacher distillation. - **Source:** [arXiv](https://arxiv.org/abs/2608.09119) - **Published:** 2026-08-12 - **Permalink:** https://picx.dev/p/ZLy6La - **Whiteboard:** https://picx.dev/p/ZLy6La/image ## Summary ## Summary (Overview) - **Motif 3** is a decoder-only Mixture-of-Experts (MoE) language model with **314 billion total parameters** and **13.2 billion activated parameters per token**, featuring fine-grained sparsity with 384 routed experts per layer (8 selected per token). - The architecture introduces **Grouped Differential Latent Attention (GDLA)** , combining grouped differential attention with Multi-head Latent Attention (MLA) for noise suppression and efficient inference. - Pretrained on **~12.5 trillion tokens** with a diverse corpus, using advanced stabilization techniques including sigmoid-based routing, decaying router noise, and layer-adaptive auxiliary losses. - Post-training employs **Multi-teacher On-Policy Distillation (MOPD)** , consolidating seven specialist teachers (agentic tool use, professional work, software engineering, long-context reasoning, mathematics, code/science, chat) into a unified model. - Achieves competitive performance on agentic tasks (τ³-Banking: 35.3, Terminal-Bench 2.1: 74.9), reasoning benchmarks (GPQA Diamond: 83.4), and demonstrates strong calibration with a non-hallucination score of 71.6 on AA-Omniscience. --- ## Introduction and Theoretical Foundation Large language models (LLMs) have evolved from general-purpose systems into increasingly capable problem solvers. Recent advances highlight that further progress depends not only on scaling data and computation but also on developing **more expressive and efficient architectures**. Key motivations for Motif 3: - **Fine-grained MoE sparsity**: A large pool of experts provides substantial capacity while limiting per-token computation. - **Attention efficiency**: Standard self-attention can allocate probability mass to irrelevant context; differential attention improves selectivity. - **KV-cache reduction**: MLA compresses key-value states into low-rank latent representations, critical for long-context inference. The theoretical foundation rests on three pillars: 1. **Differential Attention** [70]: Subtracts one attention distribution from another to cancel shared noise patterns, concentrating attention on relevant context. 2. **Multi-head Latent Attention (MLA)** [14, 15]: Compresses KV states into a low-rank latent representation, substantially reducing KV-cache requirements. 3. **Grouped Differential Attention (GDA)** [63]: Asymmetric allocation with more signal heads than noise heads, increasing signal-modeling capacity with limited additional computation. --- ## Methodology ### 2.1 Architecture Overview Motif 3 uses a hybrid attention schedule: one full causal-attention layer followed by three sliding-window attention layers. Key components: - **GDLA layers** with query-dependent output gating - **Sparse MoE feed-forward layers** with Expert-Specific PolyNorm activations - **Modified manifold-constrained hyper-connections (mHC)** - **Multi-token prediction (MTP)** for self-speculative decoding ### 2.2 Grouped Differential Latent Attention (GDLA) **Latent Query and Key-Value Representations:** The query is constructed through low-rank projections: $$\mathbf{c}_t^Q = \text{RMSNorm}\left(\mathbf{x}_t \mathbf{W}_a^Q\right)$$ $$\left[\mathbf{q}_t^C; \mathbf{q}_t^R\right] = \mathbf{c}_t^Q \mathbf{W}_b^Q$$ The KV down-projection jointly produces a compressed representation and decoupled rotary key: $$\left[\mathbf{c}_t^{KV}; \mathbf{k}_t^R\right] = \mathbf{x}_t \mathbf{W}_a^{KV}$$ The full KV latent is normalized and expanded: $$\overline{\mathbf{c}}_t^{KV} = \operatorname{RMSNorm}\left(\mathbf{c}_t^{KV}\right)$$ $$\left[\mathbf{K}_t^C; \mathbf{V}_t\right] = \mathrm{Reshape}_{\mathrm{heads}}\left(\overline{\mathbf{c}}_t^{KV} \mathbf{W}_b^{KV}\right)$$ **Grouped Differential Attention:** With $n_S$ signal heads and $n_N$ noise heads, the grouped ratio $g$ satisfies: $$n_N = \frac{n_H}{g+1}, \quad n_S = g n_N$$ A token-dependent coefficient is predicted for each signal head: $$\boldsymbol{\lambda}_t = \sigma\left(\mathbf{x}_t \mathbf{W}^\lambda\right), \quad \boldsymbol{\lambda}_t \in (0,1)^{n_S}$$ The grouped differential output: $$\mathbf{D}_t = \mathbf{H}_{S,t} - \lambda_t \odot \operatorname{Repeat}_g\left(\mathbf{H}_{N,t}\right)$$ **Output Gating:** $$\mathbf{G}_t = \operatorname{Reshape}_{\text{heads}}\left(\mathbf{c}_t^Q \mathbf{W}^G\right)$$ $$\operatorname{GDLA}(\mathbf{x}_t) = \operatorname{vec}\left(\sigma(\mathbf{G}_t) \odot \mathbf{D}_t\right) \mathbf{W}^O$$ ### 2.3 Modified Manifold-Constrained Hyper-Connections mHC generalizes a single residual stream into $n$ parallel streams (here $n=4$): $$\mathbf{X}_{\ell+1,t} = \mathbf{H}_{\text{res},\ell,t} \mathbf{X}_{\ell,t} + \mathbf{H}_{\text{post},\ell,t}^\top F_\ell\left(\mathbf{H}_{\text{pre},\ell,t} \mathbf{X}_{\ell,t}; \mathbf{W}_\ell\right)$$ The **key modification** is a time-dependent post-mapping scale annealed from 2 to 1 during pretraining: $$\mathbf{H}_{\text{post}}^{(t)} = s_t \sigma(\mathbf{Z}_{\text{post}}), \qquad s_t: 2 \longrightarrow 1$$ This prevents repeated residual amplification from producing activation outliers at large depth. ### 2.4 Expert-Specific PolyNorm Each expert learns its own polynomial activation: $$\mathrm{PolyNorm}_i(\mathbf{z}) = \sum_{n=1}^{3} a_{i,n} \frac{\mathbf{z}^n}{\mathrm{RMS}(\mathbf{z}^n)} + b_i$$ Coefficients are sigmoid-parameterized $a_{i,n} = \sigma(\tilde{a}_{i,n}) \in (0,1)$ and biases clipped to $[-0.5, 0.5]$. ### 2.5 Post-Training Pipeline 1. **General SFT**: Trained from pretrained checkpoint on consolidated corpus 2. **Specialist Teachers**: Six trained with GRPO across 13 verifier domains; one software-engineering teacher via SFT 3. **MOPD**: Distills complementary capabilities into a single student The MOPD loss: $$\mathcal{L}_{\mathrm{MOPD}} = -\mathbb{E}_t[\widetilde{w}_t d_t \log \pi_\theta(y_t | x, y_{