# Mellum2 Technical Report > Mellum 2 is an efficient 12B MoE model specialized for software engineering, matching the inference cost of a 7B dense model while achieving competitive performance on coding and reasoning tasks. - **Source:** [arXiv](https://arxiv.org/abs/2605.31268) - **Published:** 2026-06-02 - **Permalink:** https://picx.dev/p/K4SKZk - **Whiteboard:** https://picx.dev/p/K4SKZk/image ## Summary # Mellum 2 Technical Report Summary ## Summary (Overview) * **Efficient MoE Architecture:** Mellum 2 is an open-weight 12B-parameter Mixture-of-Experts (MoE) language model with 2.5B active parameters per token (64 experts, 8 active). It is designed for efficient inference, matching the latency of a Qwen2.5-7B dense model while offering a larger parameter envelope. * **Specialized for Software Engineering:** As a successor to the dense Mellum-4B, Mellum 2 is a general-purpose language model specialized for software engineering tasks, including code generation/editing, debugging, reasoning, tool use, and conversational assistance. * **Innovative Training Pipeline:** The model was pre-trained on ~10.6 trillion tokens using a three-phase curriculum that progressively shifts from web data to curated code/math. It employs the Muon optimizer with FP8 hybrid precision, a Warmup-Hold-Decay schedule, and features a Multi-Token Prediction head for improved performance and speculative decoding. * **Extended Context & Post-Training:** The base model's context was extended to 128K using layer-selective YaRN. Two post-trained variants were released: an **Instruct** model for direct answers and a **Thinking** model that emits explicit reasoning traces, both refined via reinforcement learning with verifiable rewards (RLVR). * **Competitive Performance:** Evaluations show Mellum 2 is competitive with open-weight baselines in the 4B–14B parameter range on code, math, reasoning, and tool-use benchmarks, while operating at the compute cost of a 2.5B dense model. ## Introduction and Theoretical Foundation Large language models (LLMs) have evolved from simple code autocomplete to comprehensive coding assistants capable of code generation, editing, debugging, multi-step reasoning, tool use, and agentic workflows. The open-weights landscape features a trade-off: dense 4–14B models are cheap to serve but plateau on harder tasks, while very large MoE models offer frontier quality at high deployment costs. Mellum 2 aims to strike a balance, extending the line of small MoE coding models. It is designed as a practical, deployable coding assistant for JetBrains IDEs, succeeding the completion-focused Mellum-4B. The core motivation is to achieve a wide knowledge scope (via a large total parameter count) with low serving cost (via sparse activation), targeting the inference budget of a Qwen2.5-7B model. ## Methodology ### Model Architecture Mellum 2 is a decoder-only Transformer based on the Qwen3-MoE recipe, with key efficiency-oriented modifications: * **Backbone:** 28 layers, hidden dimension 2,304, with pre-RMSNorm ($\epsilon = 10^{-6}$) and SiLU-gated MLPs. * **Attention:** Grouped-Query Attention (GQA) with 32 query heads and **4 KV heads** (head dimension 128), QK-Norm, and RoPE with base $\theta = 500,000$. * **Sliding Window Attention (SWA):** Applied to **3 out of every 4 layers** with a window size of 1,024 tokens; the remaining layer uses full attention. * **Mixture-of-Experts:** **64 experts** per layer with **8 active** per token (top-8 routing), expert intermediate size 896, no shared expert. Uses dropless routing. * **Multi-Token Prediction (MTP):** A single additional transformer layer trained with loss weight $\alpha = 0.1$. It serves as an auxiliary pre-training objective and a built-in draft model for speculative decoding, and is removed during evaluation. * **Vocabulary:** 98,304 tokens, untied embeddings. Native context length is 8,192 (extended to 131,072). **Table 2 — Architecture configuration of Mellum 2.** | Scale | | | :--- | :--- | | Total parameters | ~12B | | Active parameters | ~2.5B | | Vocabulary size | 98,304 | | Context length | 8,192 / 131,072 ★ | | **Backbone** | | | Layers | 28 | | Hidden dimension | 2,304 | | Activation | SiLU (gated) | | Normalization | RMSNorm ($\epsilon = 10^{-6}$) | | Position encoding | RoPE ($\theta = 500,000$) | | **Attention** | | | Query heads | 32 | | KV heads (GQA) | 4 | | Head dimension | 128 | | QK-Norm | Yes (RMSNorm) | | Sliding window | 1,024 (3:1 SWA) | | **Mixture-of-Experts & MTP** | | | Experts (total) | 64 | | Experts (active) | 8 (top-8) | | Expert MLP size | 896 | | Shared expert | None | | MTP layers | 1 ($\alpha = 0.1$) | ### Pre-Training Data and Curriculum The pre-training corpus of ~10.6T tokens comprises web/general knowledge, source code, and mathematical content. Training follows a three-phase "web early, curated late" curriculum: **Table 3 — Three-phase pre-training curriculum.** | Phase | Tokens (T) | % Total | LR State | Web % | Code % | Math % | | :--- | :--- | :--- | :--- | :--- | :--- | :--- | | 1: Foundation | 6.18 | 58.0 | Warmup → Hold | 70 | 23 | 6 | | 2: Quality Uplift | 2.79 | 26.2 | Hold | 44 | 42 | 14 | | 3: Capability Sharpening | 1.69 | 15.9 | Decay | 23 | 59 | 18 | | **Total** | **10.65** | **100.0** | | | | | * High-quality data is repeated up to 4x across phases. * A Fill-in-the-Middle (FIM) objective is used, with its application rate varying by phase (50% in Phase 1, 10% in Phase 2, 50% on code-only in Phase 3). ### Training Setup * **Optimizer:** Distributed Muon (Moonlight configuration with extra scale factor 0.2), with Adam $\epsilon = 10^{-8}$. * **Schedule:** Warmup-Hold-Decay (WHD) with a peak LR of $3 \times 10^{-4}$, warming up over 2,000 steps, holding, then decaying linearly to zero over Phase 3. * **Precision:** BF16 base with FP8 hybrid mixed-precision training (tensorwise recipe). * **Batch Size:** Global batch size ramps from 2,048 to 4,096 sequences (8,192 tokens each). * **MoE Training:** Uses a global auxiliary load-balancing loss (coefficient $10^{-3}$) and a router z-loss ($10^{-3}$). ### Long Context Extension The pre-trained base model's context was extended from 8,192 to 131,072 tokens using a **layer-selective YaRN** method. YaRN frequency re-mapping is applied only to the global (full-attention) layers, leaving the SWA layers unchanged. This approach outperformed uniform scaling methods. ### Post-Training Two variants were created from the same long-context base checkpoint via supervised fine-tuning (SFT): 1. **Instruct:** Answers directly. Loss computed on all assistant turns. 2. **Thinking:** Emits an explicit reasoning trace before the final answer. Loss computed only on the final assistant turn (with its reasoning). Multi-turn conversations are unfolded for training. Both variants were further refined via **Reinforcement Learning with Verifiable Rewards (RLVR)**, using a variant of the GRPO algorithm. Key adaptations include: * Asynchronous rollouts with a decoupled verification stack. * Per-token **IcePop truncation** to handle train-vs-inference disparity in MoE router log-probabilities. * **Reward shaping** with a soft overlong penalty and a concision penalty to suppress unnecessary inline reasoning in the Instruct variant. The loss function incorporates these modifications: $$A_i = R_i - \frac{1}{G-1} \sum_{j \neq i} R_j,$$ $$r_{i,t} = \frac{\pi_{\text{train}}(y_{i,t} \mid y_{i,