Summary (Overview)

• DASC (Decay-Aware State Compression) is a novel serving-time technique that compresses recurrent state checkpoints in hybrid linear-attention models (Gated DeltaNet/GDN and Kimi Delta Attention/KDA) by exploiting the fact that different heads/channels retain prefix information over markedly different timescales ("retention horizons").
• The paper introduces a weight-derived, input-independent metric $H_s = \ln\epsilon/g$ for estimating retention horizons from model decay parameters, eliminating the need for calibration prompts or online profiling.
• DASC achieves 2.63× compression of KDA recurrent state checkpoints at conservative settings while remaining close to full-cache quality, and reduces mean Time to First Token (TTFT) by 42.6% with 68.4% higher input throughput under matched HBM budgets.
• Two load policies are proposed: DASC-NR (zero-fills omitted units, no extra compute) and DASC-WR (refreshes omitted units from a bounded suffix, trading compute for accuracy recovery at aggressive compression).
• The method is implemented in SGLang and validated on Qwen3-Next-80B (head-wise GDN) and Kimi-Linear-48B (channel-wise KDA), demonstrating generalizability across both decay granularities.

---

Introduction and Theoretical Foundation

Background and Motivation

Hybrid linear-attention architectures now underpin trillion-parameter open-weight models (e.g., Qwen3.8-2.4T-A95B, Kimi K3 2.8T). These models interleave full-attention layers with linear-attention layers that summarize the prefix in fixed-size recurrent states. While this design slows KV cache growth, it introduces a new challenge: recurrent states are overwritten as tokens arrive and do not preserve earlier prefix boundaries.

Prefix caching therefore requires materializing full state checkpoints at regular token intervals alongside KV blocks. This creates a tension:

• Frequent state checkpointing → exhausts HBM quickly
• Sparse state checkpointing → increases replay or repeated prefill cost

Theoretical Foundation: Retention Horizons

The paper analyzes the decay structure of GDN and KDA recurrent states. A gated delta-rule layer summarizes the processed sequence in a recurrent state $S_t \in \mathbb{R}^{d_v \times d_k}$, which a query reads as $o_t = S_t q_t$. KDA updates this state as:

$$S_t = S_{t-1} \mathrm{Diag}(\alpha_t) \left(I - \beta_t k_t k_t^{\top}\right) + \beta_t v_t k_t^{\top}, \quad \alpha_t \in (0, 1)^{d_k} \tag{1}$$

where $\alpha_t$ controls input-dependent forgetting and $\beta_t$ the update strength. The three terms attenuate the old state, remove content aligned with $k_t$, and write the new key–value association.

The GDN parameterization of the log-decay $g = \log \alpha_t$ is:

$$g = -\exp(A_{\mathrm{log}}) \, \mathrm{softplus}(a + d t_{\mathrm{bias}}) \tag{2}$$

where $A_{\mathrm{log}}$ and $dt_{\mathrm{bias}}$ are learned decay parameters and $a$ is the input-dependent gate logit. The static retention horizon is defined as the number of tokens required for the decay factor to fall to $\epsilon$:

$$H_s = \frac{\ln \epsilon}{g} \tag{3}$$

with $\epsilon = 10^{-3}$ empirically. The metric yields one $H_s$ per head for GDN and per key channel for KDA.

Key finding: KDA exhibits broad channel decay (median $\alpha = 0.471$, IQR [0.170, 0.774]) but predominantly slow GDN head decay (0.969, [0.894, 0.997]). At a 16-token cutoff, 90.6% of KDA heads contain channels on both sides, motivating channel-wise selection for KDA and head-wise selection for GDN.

---

Methodology

Decay-Aware Selection and Storage

DASC constructs a fixed selection mask by comparing each unit's static horizon $H_s(u)$ with a threshold $W_{\max}$:

• Unit is global (retained) when $H_s(u) > W_{\max}$
• Unit is local (omitted) otherwise

Selection follows the architecture's decay granularity: one complete head state for GDN and one key channel $(h,k)$ for KDA.

The compression ratios are defined as:

$$C_{\mathrm{temp}}^{\mathrm{rag}} = \frac{\sum_{l=1}^{L} n_l}{\sum_{l=1}^{L} r_l}, \qquad C_{\mathrm{ckpt}}^{\mathrm{rag}} = \frac{B_{\mathrm{temp}}^{\mathrm{dense}} + B_{\mathrm{conv}}}{B_{\mathrm{temp}}^{\mathrm{rag}} + B_{\mathrm{conv}}} \tag{6}$$

where $n_l$ and $r_l$ are total and retained state-unit counts in layer $l$, and $B_{\mathrm{temp}}$, $B_{\mathrm{conv}}$ denote temporal and convolution state bytes.

Load Policies

DASC-NR (default): Loads retained global units exactly; leaves omitted local units at zero. No additional model computation.

DASC-WR: Replays a bounded suffix from a zero-initialized scratch recurrent state, refreshing only the local units. The replay window is:

$$W_u = \mathrm{clamp}(\mathrm{next\_pow2}(H_s(u)), 4, W_{\max}), \quad W_{\mathrm{replay}} = \max_{u \in \mathcal{L}_{\mathrm{local}}} W_u \tag{7}$$

TP-Balanced Placement

Two-stage balancing to avoid uneven state checkpoint payloads across tensor-parallel ranks:

1. Model-load-time permutation redistributes linear-attention head bundles across TP ranks (exact reindexing, no per-token communication).
2. Storage-time redistribution treats the TP group as a shared storage pool, using a variable-split all-to-all during STORE and inverse transfer during LOAD.

---

Empirical Validation / Results

Validation of Selection Signal

The weight-derived $H_s$ was validated against observed signals using Spearman correlations (Table 1(b)):

Quality Under Compression

RULER benchmarks: At $W_{\max} = 16$, Kimi-KDA scores 0.95/0.95/0.95 at 4k/8k/16k vs. dense 0.95/0.96/0.95; Qwen-GDN scores 0.84/0.83/0.81 vs. 0.84/0.83/0.80. Every aggregate is within 0.01 of dense. At $W_{\max} = 1024$, Kimi-KDA drops by 0.04–0.07 points.

End-task quality (Table 3, key results):

† marks Holm-adjusted $p < 0.05$ for DASC vs. Random. At $W_{\max} = 16$, DASC remains within 1.3 percentage points of dense on every row and exceeds random selection in all 15 reasoning comparisons.

Matched-HBM Serving Performance

• State checkpoint capacity: Channel-wise KDA achieves 2.63–28.04× compression across $W_{\max} = 16$–512; head-wise GDN achieves 1.10–2.48×.
• Serving gains (KDA): TTFT reduced by 25.4–42.6%, input throughput improved by 39.3–68.4%. At $W_{\max}=16$: TTFT from 567.6→326.0 ms; throughput from 33.25→55.98 k tokens/s.
• Serving gains (GDN): At $W_{\max}=512$: TTFT from 614.4→374.5 ms; throughput from 30.56→50.96 k tokens/s.

DASC-WR Accuracy Recovery

At $W_{\max} = 128$, DASC-WR improves all five end-task estimates over DASC-NR by 0.8–7.4 percentage points, matching or exceeding Dense on HMMT and GPQA. At $W_{\max} = 512$, it recovers 7.02 points and remains within 0.31 points of Dense while retaining 25.4% lower TTFT.

Composition with Quantization

Combining $W_{\max} = 16$ channel selection with INT8 state checkpoint storage yields 8.11× capacity relative to dense mixed-precision reference, with negligible quality loss (Table 4).

---

Theoretical and Practical Implications

Theoretical Contributions

1. Retention heterogeneity characterization: The paper provides the first systematic analysis showing that recurrent-state units in hybrid linear-attention models have widely different retention horizons that follow architectural structure (head-wise for GDN, channel-wise for KDA).

2. Input-independent selection signal: The weight-derived horizon $H_s$ provides a principled, calibration-free metric for unit selection, validated against empirical horizons (Spearman $\rho \approx 0.8$–0.86) and observed activity.

3. Causal ablation insights: Removing recurrent state checkpoints changes accuracy by at most one point on single-key retrieval, revealing task-specific redundancy—but the paper shows selection must be conservative because states matter for multi-key, aggregation, and reasoning tasks.

Practical Implications

1. Serving efficiency: DASC enables more state checkpoints to fit in the same HBM budget, directly reducing TTFT and improving throughput for prefix-reuse workloads.

2. Architecture-dependent gains: Channel-wise decay (KDA) yields dramatically better compression than head-wise decay (GDN), motivating finer-grained decay parameterization in future model designs.

3. Complementarity: DASC is orthogonal to existing techniques (KV quantization, eviction policies, replay systems) and can be composed with them for compounding benefits.

---

Conclusion

DASC turns retention heterogeneity into an input-independent state checkpoint policy for hybrid serving. On Kimi-KDA, it remains near dense quality while providing 2.63× state checkpoint capacity, 42.6% lower mean TTFT, and 68.4% higher input-token throughput under matched HBM. Suffix replay recovers accuracy at aggressive compression; Qwen-GDN extends DASC to head-wise decay.

Future directions include:

• Transfer to purely linear-attention or state-space models
• Scaling to larger TP degrees
• Finer-grained compression for head-decayed architectures (e.g., per-channel within GDN)

Limitations: All serving experiments use TP8; benefits are architecture-dependent, with GDN yielding smaller gains than KDA due to coarser decay granularity.