Summary (Overview)

• Core Question: How does replacing softmax attention with linear recurrent sequence mixers change the quantization problem for LLMs, and how should quantization methods be adapted accordingly?

• Two Quantization Targets: The paper distinguishes between (1) extra-state quantization (weights/activations outside the recurrent state, shaped indirectly by the mixer during training) and (2) intra-state quantization (the recurrent state itself, which is repeatedly updated and requantized in place — fundamentally different from KV cache quantization).

• Key Finding on Activations: Decayed linear recurrences (e.g., Mamba-2, GLA, Gated DeltaNet, KDA) develop substantially weaker residual outlier channels than softmax attention, making them easier to quantize without incoherence processing. Incoherence processing (rotations) largely closes this gap.

• LoReS Recipe: The authors instantiate their findings as LoReS (Low-bit Recurrent State), combining value-axis scale grouping, decay-based mixed precision, and stochastic rounding for robust integer state quantization. It supports 8-bit states on Qwen-3.5 9B, Nemotron-H 8B, and Kimi-Linear 48B-A3B, with lower effective precision on shorter decode horizons.

• Deployment: LoReS is implemented in vLLM, demonstrating end-to-end decode speedups that combine complementarily with weight quantization.

---

Introduction and Theoretical Foundation

Softmax attention has shaped the architecture of LLMs and the deployment stack built around them, particularly in post-training quantization (PTQ). However, the space of sequence mixers is expanding: linear attention and state-space models replace softmax attention with linear recurrences, and recent large-scale hybrid models interleave recurrent and softmax layers.

The Fundamental Difference: The state-update rule distinguishes recurrent mixers:

General Recurrence Form: All recurrent variants follow:

$$\mathbf{S}_t = \mathbf{S}_{t-1} \operatorname{diag}(\boldsymbol{\alpha}_t)(\mathbf{I} - \delta \beta_t \mathbf{k}_t \mathbf{k}_t^{\top}) + \beta_t \mathbf{v}_t \mathbf{k}_t^{\top}, \qquad \mathbf{o}_t = \mathbf{S}_t \mathbf{q}_t$$

where $\|\mathbf{k}_t\|_2 = \|\mathbf{q}_t\|_2 = 1$, $\beta_t \in (0, 2)$, decay $\alpha_{t,i} \in (0, 1]$, and $\delta \in \{0, 1\}$ controls delta correction.

Three Key Mechanisms:

• Decay: Multiplicative forget gate (scalar per head or diagonal per channel)
• Write gating: Controls outer-product update strength
• Delta correction: Rank-one correction along the current key: $(\mathbf{I} - \beta_t \mathbf{k}_t \mathbf{k}_t^{\top})$

A KV cache grows linearly with sequence length, while a recurrent state $\mathbf{S}_t \in \mathbb{R}^{d_v \times d_k}$ has constant size and is updated in place — creating distinct numerical problems. The state error propagates temporally:

$$\mathbf{E}_t = \mathbf{E}_{t-1} \operatorname{diag}(\boldsymbol{\alpha}_t)(\mathbf{I} - \delta \beta_t \mathbf{k}_t \mathbf{k}_t^{\top}) + \boldsymbol{\varepsilon}_t$$

---

Methodology

Experimental Setup

The paper constructs a 1.3B-parameter "mixer zoo" where the sequence mixer varies while surrounding layer design and training recipe are fixed. Models share the same backbone (QKV projections, causal convolutions, normalization, RoPE, output projection), with only query/key normalization varied explicitly.

Extra-State Quantization Approach

• Quantization stack: GPTQ with per-output-channel weight scales and dynamic activation scales
• Rotations: Three orthogonal rotations evaluated:
  • $\mathbf{R}_1$: rotates the residual stream
  • $\mathbf{R}_4$: rotates input to the MLP down projection
  • $\mathbf{R}_o$: rotates input to the mixer's output projection
• Metric: Relative WikiText-2 perplexity changes

Intra-State Quantization Approach

For state quantization, the paper uses teacher-forced decode over eight sequences of 2048 WikiText-2 tokens, with the core metric $\Delta\ell$ being the median token-level increase in cross-entropy relative to a full-precision 32-bit trajectory.

Key properties distinguishing recurrent state from KV cache quantization:

1. State has two channel axes (vs. one channel axis + time axis in KV cache)
2. Decay suppresses error accumulation over time
3. State is repeatedly requantized (dequantize → update → requantize each step)

---

Empirical Validation / Results

Extra-State Quantization (Section 3)

Residual Stream Outliers: Softmax attention develops a dominant residual channel with magnitude approximately constant across tokens and layers. Decayed recurrences exhibit weaker outliers. Residual stream kurtosis closely tracks unrotated W4A4 degradation.

Table 2: Relative Perplexity Change (%) on WikiText-2 (1.3B mixer zoo)

Decayed recurrences (bold) show substantially lower W4A4 degradation than softmax models.

Channel Removal Experiment (Table 3): Removing the dominant residual channel before QK projections increases perplexity by 1691.3% in softmax models vs. only 2.2% in Diag models. The authors hypothesize that decayed recurrences need less "attention sink" structure because decay provides a token-focusing mechanism.

Intra-State Quantization (Section 4)

Scale Grouping: Value-axis grouping (one scale per value row) vs. key-axis grouping (one scale per key column).

Proposition 1 (Per-channel readout error): With row ranges $M_i := \max_j |S_{ij}|$ and column ranges $C_j := \max_i |S_{ij}|$, and scale $s_G := \|\check{S}_G\|_\infty / r$ where $r := 2^{b-1} - 1$:

$$\mathbb{E}_\mathbf{E}[(e_i^{\text{val}})^2] = \frac{M_i^2}{12r^2}, \qquad \mathbb{E}_\mathbf{E}[(e_i^{\text{key}})^2] = \frac{\sum_j q_j^2 C_j^2}{12r^2}$$

Value-axis grouping is preferred at INT8 for all models and at INT6 for all except Linear attention.

Decay-Based Mixed Precision:

Proposition 2 (Decaying error): If decay is bounded by $\|\bar{\boldsymbol{\alpha}}_t\|_\infty \leq \alpha < 1$, then:

$$\|\mathbf{E}_t\|_F \leq \sum_{\tau=1}^{t} \alpha^{t-\tau} \|\boldsymbol{\varepsilon}_\tau\|_F$$

Promoting the slowest-decaying half of heads to higher precision recovers most of the INT4-to-INT6 gap (Table 5): e.g., Qwen-3.5 from 0.825 (INT4) to 0.006 (INT5 effective).

Rounding:

Proposition 3 (Repeated quantization): For scalar decay $\alpha_t$, $\delta = 0$, with $\gamma_{\tau,t} = \prod_{j=\tau+1}^{t} \alpha_j$:

• RTN: $\|\mathbf{E}_t^{\text{RTN}}\|_F^2 = \sum_{\tau=1}^{t} \gamma_{\tau,t}^2 \|\boldsymbol{\varepsilon}_\tau^{\text{RTN}}\|_F^2 + 2\sum_{1 \leq \tau < s \leq t} \gamma_{\tau,t} \gamma_{s,t} \langle \boldsymbol{\varepsilon}_\tau^{\text{RTN}}, \boldsymbol{\varepsilon}_s^{\text{RTN}} \rangle_F$
• SR: $\mathbb{E}[\|\mathbf{E}_t^{\text{SR}}\|_F^2] = \sum_{\tau=1}^{t} \gamma_{\tau,t}^2 \mathbb{E}[\|\boldsymbol{\varepsilon}_\tau^{\text{SR}}\|_F^2]$

Stochastic rounding eliminates cross-term error reinforcement in expectation.

Stochastic rounding (SR) consistently lowers $\Delta\ell$ for all three hybrids at INT8 and INT6 (Table 6), e.g., Nemotron-H: 0.0105 → 0.0026 at INT6.

Rotations: Random rotations can increase error under repeated state quantization due to:

1. Scale feedback loops: Noisy states produce coarser quantization grids, compounding over time
2. Query alignment: Rotations leak error into query dimensions that were previously suppressed

Format: Value-grouped INT8 generally outperforms E4M3 FP8 for state quantization.

End-to-End Evaluation (Section 5)

Table 8: End-to-end accuracy (%) — GPQA and AIME are avg@32

INT8 states preserve most accuracy; losses grow mainly from INT4 weights.

Speedups: INT8 state quantization accelerates Mamba-2, GDN, and KDA kernels (≥2× throughput at batch 8). End-to-end vLLM decode shows speedups grow with batch size, and weight + state quantization are complementary.

---

Theoretical and Practical Implications

Theoretical Contributions

1. Characterization of how sequence mixer choice shapes quantization dynamics: Decayed recurrences develop different learned representations (weaker residual outliers) that make them inherently more quantization-robust. This connects architectural inductive biases to numerical properties.

2. Formal analysis of state quantization error: The paper provides the first systematic treatment of errors specific to recurrent state quantization — including the effect of decay on error suppression (Proposition 2) and why stochastic rounding eliminates cross-term error correlation under repeated requantization (Proposition 3).

3. Reveals failure modes of established techniques: Random rotations, effective for softmax KV cache quantization, can fail under repeated state requantization due to scale feedback loops and query alignment effects.

Practical Implications

1. LoReS is a complete recipe: Value-axis grouping + decay-based mixed precision + stochastic rounding enables robust 8-bit recurrent states (and 5-5.5 effective bits on shorter horizons) with minimal accuracy loss — directly usable in production.

2. Complementary speedups: Weight quantization helps at small batch sizes; state quantization helps at large batch sizes. Together they provide multiplicative decode speedups.

3. No need for incoherence processing: Decayed recurrent models can be quantized without rotation-based outlier processing, simplifying deployment.

---

Conclusion

This work provides the first systematic study of PTQ for linear recurrent and hybrid language models, separating weight/activation quantization from state quantization during decode. Key findings:

1. Decayed recurrences develop weaker residual outliers than softmax attention, making their activations easier to quantize
2. Value-axis scale grouping, decay-based precision allocation, and stochastic rounding form the basis of robust state quantization
3. Random rotations can fail under repeated state requantization
4. INT8 state quantization preserves accuracy across three large hybrid models

LoReS combines these insights into a production-ready pipeline implemented in vLLM, demonstrating that "quantizing recurrent language models is easier than you think" — both because the models are inherently more robust and because the required methods are now well understood.

Future directions: Quantization-aware training or low-precision training; GPU-native microscaling formats; deeper investigation of rotation effects on state quantization; explicit training-time integration of state quantization.