# On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stability

> Qwen3.8-Flash-Next matches its 397B predecessor's quality using one-third activated parameters and one-ninth training FLOPs via GDN, QSA, GR, and n-gram embeddings.

- **Source:** [arXiv](https://arxiv.org/abs/2608.30320)
- **Published:** 2026-09-26
- **Permalink:** https://picx.dev/p/emCgDr
- **Whiteboard:** https://picx.dev/p/emCgDr/image

## Summary

## Summary (Overview)

- **Qwen3.8-Flash-Next** is a sparse mixture-of-experts (MoE) model with 125B total parameters, 6B activated per token, and 51B additional n-gram embedding parameters stored off-accelerator.
- The model matches or exceeds the quality of its 397B-A17B predecessor on 8 of 14 benchmarks while using ~1/3 the activated parameters, ~1/3 the training tokens, and ~1/9 the training FLOPs.
- Four key architectural innovations: (1) a **Gated DeltaNet (GDN) hybrid** token mixer, (2) **Qwen Sparse Attention (QSA)** for long-context efficiency, (3) a **Gated Residual (GR)** stream with four branches, and (4) an **n-gram embedding layer** for off-accelerator capacity scaling.
- The paper emphasizes a **joint design methodology** evaluating every change along three axes: loss/downstream benchmarks, computational cost, and training stability—revealing cases where these metrics disagree.
- The Muon optimizer combined with the new architecture shifts optimal hyperparameters (larger batch size and learning rate), eliminates the need for batch-size warmup, and substantially improves training stability under stress tests.

## Introduction and Theoretical Foundation

The paper addresses a fundamental tension in LLM architecture design: any architectural change simultaneously affects model capability, computational cost, and training stability. The authors argue these three dimensions form one coupled design problem that must be solved jointly.

**Background motivations:**

- **Token mixing bottleneck**: Full self-attention provides direct content-based access to all preceding tokens but has quadratic cost in sequence length and linear KV cache growth during generation. Sliding-window attention reduces cost but limits long-range information propagation to indirect paths through depth.

- **Residual stream limitations**: Pre-normalization attenuates signals each layer receives—every block reads the same stream, so early features must compete with everything written after them. This motivates widening the residual stream.

- **Capacity scaling**: Adding parameters through traditional dense layers increases per-token FLOPs. N-gram embeddings offer a complementary dimension: scaling capacity with negligible additional computation via deterministic, sparsely-accessed embedding tables.

**Theoretical foundations:**

The Gated DeltaNet recurrence is built on linear attention interpreted as a fast-weight memory (Schlag et al., 2021). For each head, with queries $\pmb q_t, \pmb k_t \in \mathbb{R}^{d_k}$ and values $\pmb v_t \in \mathbb{R}^{d_v}$, the state $S_t \in \mathbb{R}^{d_k \times d_v}$ follows the gated delta rule:

$$
\widetilde{\pmb S}_{t-1} = \alpha_t \pmb S_{t-1},\tag{1}
$$

$$
\pmb e_t = \pmb v_t - \widetilde{\pmb S}_{t-1}^{\top} \pmb k_t,\tag{2}
$$

$$
\pmb S_t = \widetilde{\pmb S}_{t-1} + \beta_t \pmb k_t \pmb e_t^{\top},\tag{3}
$$

$$
\pmb y_t = \pmb S_t^{\top} \pmb q_t,\tag{4}
$$

where $\alpha_t \in (0,1)$ is a data-dependent decay and $\beta_t \in (0,1)$ controls the delta update. The decay globally controls state lifetime, while the delta term writes only the residual error, distinguishing GDN from purely additive linear attention.

## Methodology

### 1. GDN Hybrid Token Mixing

The architecture interleaves three GDN layers with one full-attention layer per block of four. GDN layers use:

- **Short causal convolutions** before projection for local inductive bias
- **L2 normalization** on queries and keys to bound magnitudes
- **Sigmoid output gates** (instead of SiLU) with zero-centered RMSNorm

The write strength and decay are parameterized as:

$$
\beta_t = \sigma(\pmb W_\beta \pmb x_t),\tag{9}
$$

$$
\alpha_t = \exp\left[-\exp(\pmb A)\operatorname{softplus}(\pmb W_\alpha \pmb x_t + \pmb b_\alpha)\right].\tag{10}
$$

The output is modulated by an input-dependent gate:

$$
\pmb o_t = \pmb W_o \left[\sigma(\pmb W_z \pmb x_t) \odot \operatorname{RMSNorm}(\pmb y_t)\right].\tag{11}
$$

**Kernel efficiency**: FlashQLA, a TileLang-based fused linear-attention kernel library, achieves 2–3× forward and ~2× backward speedups over the Triton baseline.

### 2. Qwen Sparse Attention (QSA)

QSA uses a compressed lightweight indexer with MQA structure (4 query heads, 1 shared key head). Keys are partitioned into blocks of $r=4$ tokens and compressed via average pooling:

$$
\widehat{\pmb k}_b = \operatorname{RMSNorm}\left(\operatorname{AvgPool}(\pmb k_{p_b: p_b + r - 1})\right), \quad 0 \leq b < \left\lfloor\frac{n}{r}\right\rfloor.\tag{13}
$$

Block-level importance scores use ReLU-activated query–key similarities summed over indexer heads, with block-causal masking. With token budget $K=2048$, QSA selects up to 512 complete blocks per query.

**Training**: Two stages—(1) dense distillation of the full-sequence attention distribution into the indexer via KL divergence, and (2) sparse training where the backbone adapts to sparse patterns. The KL loss is:

$$
\mathcal{L}_{\mathrm{KL}} = \frac{1}{N}\sum_i D_{\mathrm{KL}}\left(\hat{\pmb a}_{i,:} \parallel \operatorname{Softmax}(\pmb I_{i,:})\right).\tag{18}
$$

### 3. Gated Residual (GR)

The residual stream is widened to $n_r=4$ branches. Each branch is normalized independently:

$$
\widehat{\pmb R}_i = \mathrm{RMSNorm}\left(\pmb R_i; \pmb\gamma_i\right), \quad i = 1, \ldots, n_r,\tag{30}
$$

Elementwise gating scores are predicted from all branches:

$$
\pmb G = \operatorname{unvec}\sigma\left(\pmb W_u \operatorname{SiLU}\left(\frac{1}{n_r}\pmb W_d \operatorname{vec}(\widehat{\pmb R})\right)\right) \in \mathbb{R}^{n_r \times d},\tag{31}
$$

$$
\pmb x = \frac{1}{n_r}\sum_{i=1}^{n_r}\pmb G_i \odot \widehat{\pmb R}_i,\tag{32}
$$

The write uses one data-dependent scalar per branch:

$$
\pmb s = 2\sigma\left(\frac{1}{n_r}\pmb W_w \operatorname{vec}(\widehat{\pmb R})\right) \in \mathbb{R}^{n_r},\tag{33}
$$

$$
\pmb R_i^{\prime} = \pmb R_i + s_i \pmb y.\tag{34}
$$

Key design choices: elementwise read gate (not per-branch scalar), no $H_{\mathrm{res}}$ mixing operator (saves memory traffic), and FP8 storage for residual state.

### 4. N-gram Embedding

A single n-gram embedding layer at Layer 2 uses multi-head hashing for lookup with 300 tokens per active parameter. Tables are prefetched from host memory, enabling 51B parameters off-accelerator.

### 5. Optimization

**Muon optimizer** is applied to 2D weights acting as linear maps (attention projections, expert weights). Input embeddings, output head, MoE router, and GR low-rank projections stay on AdamW. Key implementation details:

- **Newton–Schulz iterations**: 8 steps with Polar Express coefficient schedule
- **Fused parameter splitting**: qkv and GDN input projections split at per-head granularity before orthogonalization
- **Canzona**: decouples logical optimizer assignment from physical parameter layout with α-balanced partitioning
- **CUDA graphs**: capture the entire optimizer step to avoid kernel launch overhead

## Empirical Validation / Results

### Architecture Ablations

**Table 1: Architecture comparison** (25B-A3B MoE, 400B tokens + 80B at 32K context):

| Architecture | MMLU | MMLU-Pro | SuperGPQA | MATH | GSM8K | BBH | MMMLU | EvalPlus | MultiPL-E | Avg. |
|---|---|---|---|---|---|---|---|---|---|---|
| Full attention | 62.65 | 37.59 | 21.76 | 49.40 | 75.13 | 63.78 | 47.74 | 51.01 | 39.73 | 49.87 |
| SWA hybrid | 66.30 | 40.67 | 22.45 | 45.48 | 74.22 | 65.88 | 51.33 | 52.12 | 41.93 | 51.15 |
| **GDN hybrid** | **66.26** | **42.82** | **23.45** | **53.98** | **77.07** | **68.72** | **54.83** | 49.71 | **47.48** | **53.81** |

### QSA Evaluation

**Table 2: Full attention vs. QSA** (short-context benchmarks):

| Method | MMLU-Pro | SuperGPQA | MATH | GSM8K | BBH | MMMLU | EvalPlus | MultiPL-E | Avg. |
|---|---|---|---|---|---|---|---|---|---|
| Full Attn | 72.9 | 51.7 | 69.8 | 91.0 | 90.4 | 81.8 | 70.8 | 78.4 | 75.9 |
| **w/ QSA** | **73.7** | **52.1** | **71.6** | **92.2** | **91.6** | 81.1 | **72.3** | **79.8** | **76.8** |

**Long-context retrieval** (Table 3): QSA improves RULER from 90.08 to 93.00 beyond 512K and MRCR from 30.66 to 40.53 at 512K, achieving 7.6× prefill and 4.9× decode speedups at 1M context.

### Residual Design Ablations

**Table 5: Residual read/write ablation** (25B-A3B MoE, 560B tokens):

| Residual | Loss | Avg. Benchmarks |
|---|---|---|
| Pre-norm | 1.617 | 50.91 |
| mHC (static) | 1.596 | 52.49 |
| mHC (dynamic) | 1.594 | 54.47 |
| **GR** | **1.590** | **54.66** |

### Hyperparameter Scaling

- **Batch size**: Moving from $B=12.6$M to predicted optimum $B=25.2$M improves loss by $7.2\times10^{-3}$; batch-size warmup provides no benefit and costs 18.8% more optimizer steps.
- **Learning rate**: Predicted optimum $\eta=1.76\times10^{-3}$ at $B=8.4$M outperforms the previous recipe by $7.8\times10^{-3}$ in loss and 4.14 points in average benchmark accuracy.

### Stability Stress Tests

At 4× optimal learning rate: AdamW baseline spikes 183 times per 10k steps and crosses the clipping threshold 213 times; Muon runs never cross the threshold, and GR configuration records **zero loss spikes**. The gate in GR is identified as a key stability contributor—reducing spike rate from 32.0 to 3.2 per 10k steps at 3× optimal learning rate.

### Final Model Comparison

**Table 11: Qwen3.8-Flash-Next-Base vs. baselines** (excerpt):

| Benchmark | Flash-Next-Base | Qwen3.8-27B-Base | Qwen3.7-Plus-Base |
|---|---|---|---|
| MMLU | **90.36** | 87.51 | 90.43 |
| MMLU-Pro | **73.23** | 68.60 | 70.90 |
| SuperGPQA | **51.36** | 44.86 | 48.42 |
| MATH | 72.78 | 60.54 | **74.38** |
| EvalPlus | **78.76** | 76.05 | 78.06 |
| MultiPL-E | 79.09 | 74.50 | **81.68** |
| SWEBench-Pretrain | **50.99** | 41.66 | 49.24 |
| MGSM | **89.33** | 86.37 | 85.42 |
| MMMLU | **84.86** | 79.74 | 84.53 |

Flash-Next-Base outperforms Qwen3.7-Plus-Base on 8 of 14 benchmarks while using ~1/3 activated parameters and ~1/9 training FLOPs.

## Theoretical and Practical Implications

**Loss vs. downstream accuracy divergence**: The paper documents multiple cases where these metrics disagree:
- Enlarging n-gram vocabulary lowers loss monotonically but downstream accuracy saturates
- Data-dependent residual operators give marginal loss gains (0.002) but substantial benchmark gains (1.98 points)
- Sparse residual writes appear free in pre-training loss but degrade after post-training

**Stability mechanism**: The multiplicative gate in GR supplies the rescaling that would otherwise emerge as fragile activation outliers at high learning rates. This enables stable training without explicit clipping methods like qk-clip or SwiGLU-clip.

**Optimizer-architecture coupling**: Muon's properties (data efficiency at large batch sizes) combined with GR's stability margin shift the optimal operating point toward larger batch sizes and learning rates, improving both throughput and convergence.

**Practical implications**:
- The full-scale training proceeded without a single loss spike or anomalous gradient fluctuation
- Inference efficiency: QSA provides 7.6× prefill and 4.9× decode speedups at 1M context; GR's FP8 residual state halves memory traffic
- The design methodology (evaluating along three axes jointly) catches failures that single-metric evaluation would miss

## Conclusion

Qwen3.8-Flash-Next demonstrates that architecture, efficiency, and optimization form one coupled system. The key takeaways:

1. **GDN hybrid token mixing** with periodic full attention balances efficiency and quality, with FlashQLA providing 2–3× kernel speedups.
2. **QSA** reduces indexer complexity from $O(n^2)$ to $O(n^2/r)$ while improving long-context retrieval performance.
3. **GR** widens the residual stream with elementwise gating, improving both capability and stability—the stability margin enables more aggressive hyperparameters.
4. **N-gram embeddings** scale capacity off-accelerator with negligible per-token cost.
5. **Joint evaluation** across loss, benchmarks, efficiency, and stability is essential—removing any axis admits seemingly harmless shortcuts that fail at scale.

**Future directions**: The authors identify evaluation throughput as the tightest bottleneck. A cheaper mid-scale probe that reliably predicts post-training ordering would make the design space far more searchable.

---

_Markdown view of https://picx.dev/p/emCgDr, served by PicX — AI-generated visual whiteboard summaries of research papers._