Summary (Overview)

• Motif 3 is a decoder-only Mixture-of-Experts (MoE) language model with 314 billion total parameters and 13.2 billion activated parameters per token, featuring fine-grained sparsity with 384 routed experts per layer (8 selected per token).
• The architecture introduces Grouped Differential Latent Attention (GDLA) , combining grouped differential attention with Multi-head Latent Attention (MLA) for noise suppression and efficient inference.
• Pretrained on ~12.5 trillion tokens with a diverse corpus, using advanced stabilization techniques including sigmoid-based routing, decaying router noise, and layer-adaptive auxiliary losses.
• Post-training employs Multi-teacher On-Policy Distillation (MOPD) , consolidating seven specialist teachers (agentic tool use, professional work, software engineering, long-context reasoning, mathematics, code/science, chat) into a unified model.
• Achieves competitive performance on agentic tasks (τ³-Banking: 35.3, Terminal-Bench 2.1: 74.9), reasoning benchmarks (GPQA Diamond: 83.4), and demonstrates strong calibration with a non-hallucination score of 71.6 on AA-Omniscience.

---

Introduction and Theoretical Foundation

Large language models (LLMs) have evolved from general-purpose systems into increasingly capable problem solvers. Recent advances highlight that further progress depends not only on scaling data and computation but also on developing more expressive and efficient architectures.

Key motivations for Motif 3:

• Fine-grained MoE sparsity: A large pool of experts provides substantial capacity while limiting per-token computation.
• Attention efficiency: Standard self-attention can allocate probability mass to irrelevant context; differential attention improves selectivity.
• KV-cache reduction: MLA compresses key-value states into low-rank latent representations, critical for long-context inference.

The theoretical foundation rests on three pillars:

1. Differential Attention [70]: Subtracts one attention distribution from another to cancel shared noise patterns, concentrating attention on relevant context.
2. Multi-head Latent Attention (MLA) [14, 15]: Compresses KV states into a low-rank latent representation, substantially reducing KV-cache requirements.
3. Grouped Differential Attention (GDA) [63]: Asymmetric allocation with more signal heads than noise heads, increasing signal-modeling capacity with limited additional computation.

---

Methodology

2.1 Architecture Overview

Motif 3 uses a hybrid attention schedule: one full causal-attention layer followed by three sliding-window attention layers. Key components:

• GDLA layers with query-dependent output gating
• Sparse MoE feed-forward layers with Expert-Specific PolyNorm activations
• Modified manifold-constrained hyper-connections (mHC)
• Multi-token prediction (MTP) for self-speculative decoding

2.2 Grouped Differential Latent Attention (GDLA)

Latent Query and Key-Value Representations:

The query is constructed through low-rank projections:

$$\mathbf{c}_t^Q = \text{RMSNorm}\left(\mathbf{x}_t \mathbf{W}_a^Q\right)$$ $$\left[\mathbf{q}_t^C; \mathbf{q}_t^R\right] = \mathbf{c}_t^Q \mathbf{W}_b^Q$$

The KV down-projection jointly produces a compressed representation and decoupled rotary key:

$$\left[\mathbf{c}_t^{KV}; \mathbf{k}_t^R\right] = \mathbf{x}_t \mathbf{W}_a^{KV}$$

The full KV latent is normalized and expanded:

$$\overline{\mathbf{c}}_t^{KV} = \operatorname{RMSNorm}\left(\mathbf{c}_t^{KV}\right)$$ $$\left[\mathbf{K}_t^C; \mathbf{V}_t\right] = \mathrm{Reshape}_{\mathrm{heads}}\left(\overline{\mathbf{c}}_t^{KV} \mathbf{W}_b^{KV}\right)$$

Grouped Differential Attention:

With $n_S$ signal heads and $n_N$ noise heads, the grouped ratio $g$ satisfies:

$$n_N = \frac{n_H}{g+1}, \quad n_S = g n_N$$

A token-dependent coefficient is predicted for each signal head:

$$\boldsymbol{\lambda}_t = \sigma\left(\mathbf{x}_t \mathbf{W}^\lambda\right), \quad \boldsymbol{\lambda}_t \in (0,1)^{n_S}$$

The grouped differential output:

$$\mathbf{D}_t = \mathbf{H}_{S,t} - \lambda_t \odot \operatorname{Repeat}_g\left(\mathbf{H}_{N,t}\right)$$

Output Gating:

$$\mathbf{G}_t = \operatorname{Reshape}_{\text{heads}}\left(\mathbf{c}_t^Q \mathbf{W}^G\right)$$ $$\operatorname{GDLA}(\mathbf{x}_t) = \operatorname{vec}\left(\sigma(\mathbf{G}_t) \odot \mathbf{D}_t\right) \mathbf{W}^O$$

2.3 Modified Manifold-Constrained Hyper-Connections

mHC generalizes a single residual stream into $n$ parallel streams (here $n=4$):

$$\mathbf{X}_{\ell+1,t} = \mathbf{H}_{\text{res},\ell,t} \mathbf{X}_{\ell,t} + \mathbf{H}_{\text{post},\ell,t}^\top F_\ell\left(\mathbf{H}_{\text{pre},\ell,t} \mathbf{X}_{\ell,t}; \mathbf{W}_\ell\right)$$

The key modification is a time-dependent post-mapping scale annealed from 2 to 1 during pretraining:

$$\mathbf{H}_{\text{post}}^{(t)} = s_t \sigma(\mathbf{Z}_{\text{post}}), \qquad s_t: 2 \longrightarrow 1$$

This prevents repeated residual amplification from producing activation outliers at large depth.

2.4 Expert-Specific PolyNorm

Each expert learns its own polynomial activation:

$$\mathrm{PolyNorm}_i(\mathbf{z}) = \sum_{n=1}^{3} a_{i,n} \frac{\mathbf{z}^n}{\mathrm{RMS}(\mathbf{z}^n)} + b_i$$

Coefficients are sigmoid-parameterized $a_{i,n} = \sigma(\tilde{a}_{i,n}) \in (0,1)$ and biases clipped to $[-0.5, 0.5]$.

2.5 Post-Training Pipeline

1. General SFT: Trained from pretrained checkpoint on consolidated corpus
2. Specialist Teachers: Six trained with GRPO across 13 verifier domains; one software-engineering teacher via SFT
3. MOPD: Distills complementary capabilities into a single student

The MOPD loss:

$$\mathcal{L}_{\mathrm{MOPD}} = -\mathbb{E}_t[\widetilde{w}_t d_t \log \pi_\theta(y_t | x, y_{<t})]$$

where $d_t$ is the detached OPD signal:

$$d_t = \mathrm{sg}\left[\log \frac{\pi_{T(x)}(y_t | x, y_{<t})}{\pi_{\mathrm{old}}(y_t | x, y_{<t})}\right]$$

---

Empirical Validation / Results

Pretraining Evaluation (Base Model)

Post-Training Evaluation

Controlled Experiments

• GDLA achieves lower loss than both GDA and MLA, reaching a loss of 3.2 with 9.2% fewer training tokens than MLA.
• Expert-Specific PolyNorm maintains higher effective rank in expert gate weights than SwiGLU, indicating more evenly distributed singular-value spectra.
• Decaying router noise reduces maximum expert load more rapidly and guides routing distribution toward the median-load regime early in training.

Model Configuration

---

Theoretical and Practical Implications

Theoretical Contributions

1. GDLA unifies two research directions: Differential attention's noise suppression with MLA's efficient KV compression, demonstrating that these approaches are complementary rather than competing.

2. Asymmetric clipping for shared key projections: The QK-Clip formulation with key-side clipping ratio $r = \frac{1}{1+\sqrt{G}}$ addresses the problem of shared keys degenerating under symmetric clipping, a subtle issue in grouped-query attention.

3. Activation outlier management without hard clipping: The annealed mHC post-mapping scale ($s_t: 2 \rightarrow 1$) and FFN magnitude regularization provide soft alternatives to hard activation clipping, preserving useful rescaling behavior.

Practical Implications

1. Fine-grained sparsity at scale: 384 experts with top-8 routing demonstrates that very large expert pools can be trained stably with proper balancing, achieving competitive performance with far fewer activated parameters than dense alternatives.

2. Efficient long-context training: Window-aware context parallelization reduces communication from $\Theta(L)$ to $\Theta(W)$, achieving a 1,024× analytic reduction at 128-token windows, enabling 256K context training.

3. Multi-teacher distillation: MOPD successfully consolidates specialized capabilities without environment rewards at distillation time, producing a single deployable model with broad expertise.

4. Low-precision training: Selective MXFP8 computation with FP32 critical states demonstrates that aggressive quantization can be applied to expert computation without degrading stability.

---

Conclusion

Motif 3 demonstrates that sparse scaling, careful stabilization, and targeted capability transfer can jointly produce a model that is both computationally efficient and broadly capable. Key takeaways:

• The architecture achieves strong performance with only 13.2B activated parameters, showing the value of fine-grained expert sparsity.
• GDLA provides measurable training efficiency gains over both GDA and MLA.
• The post-training pipeline successfully consolidates seven specialist capabilities into a single model.

Limitations

• Primarily a text model; lacks direct visual input understanding
• Performance varies on underrepresented tasks/domains
• Long-horizon agentic applications require more robust state tracking and planning

Future Directions

1. Scaling: Architectures that further reduce training/inference costs and scale beyond 314B parameters
2. Extended context: Native context lengths beyond one million tokens
3. Multimodality: Adding visual capabilities for image and video inputs
4. Agentic capabilities: Richer environments, longer interaction trajectories, improved planning and memory