# DASC: Decay-Aware State Compression for Hybrid Linear-Attention Serving

> DASC compresses recurrent state checkpoints by retaining only units with long retention horizons, achieving 2.63x compression, 42.6% lower TTFT, and 68.4% higher throughput with negligible quality loss.

- **Source:** [arXiv](https://arxiv.org/abs/2608.30386)
- **Published:** 2026-09-19
- **Permalink:** https://picx.dev/p/rIhP0n
- **Whiteboard:** https://picx.dev/p/rIhP0n/image

## Summary

## Summary (Overview)

- **DASC (Decay-Aware State Compression)** is a novel serving-time technique that compresses recurrent state checkpoints in hybrid linear-attention models (Gated DeltaNet/GDN and Kimi Delta Attention/KDA) by exploiting the fact that different heads/channels retain prefix information over markedly different timescales ("retention horizons").
- The paper introduces a **weight-derived, input-independent metric** $H_s = \ln\epsilon/g$ for estimating retention horizons from model decay parameters, eliminating the need for calibration prompts or online profiling.
- DASC achieves **2.63× compression** of KDA recurrent state checkpoints at conservative settings while remaining close to full-cache quality, and reduces mean **Time to First Token (TTFT) by 42.6%** with **68.4% higher input throughput** under matched HBM budgets.
- Two load policies are proposed: **DASC-NR** (zero-fills omitted units, no extra compute) and **DASC-WR** (refreshes omitted units from a bounded suffix, trading compute for accuracy recovery at aggressive compression).
- The method is implemented in SGLang and validated on **Qwen3-Next-80B** (head-wise GDN) and **Kimi-Linear-48B** (channel-wise KDA), demonstrating generalizability across both decay granularities.

---

## Introduction and Theoretical Foundation

### Background and Motivation

Hybrid linear-attention architectures now underpin trillion-parameter open-weight models (e.g., Qwen3.8-2.4T-A95B, Kimi K3 2.8T). These models interleave full-attention layers with linear-attention layers that summarize the prefix in fixed-size recurrent states. While this design slows KV cache growth, it introduces a new challenge: **recurrent states are overwritten as tokens arrive** and do not preserve earlier prefix boundaries.

Prefix caching therefore requires materializing full state checkpoints at regular token intervals alongside KV blocks. This creates a tension:
- **Frequent state checkpointing** → exhausts HBM quickly
- **Sparse state checkpointing** → increases replay or repeated prefill cost

### Theoretical Foundation: Retention Horizons

The paper analyzes the decay structure of GDN and KDA recurrent states. A gated delta-rule layer summarizes the processed sequence in a recurrent state $S_t \in \mathbb{R}^{d_v \times d_k}$, which a query reads as $o_t = S_t q_t$. KDA updates this state as:

$$
S_t = S_{t-1} \mathrm{Diag}(\alpha_t) \left(I - \beta_t k_t k_t^{\top}\right) + \beta_t v_t k_t^{\top}, \quad \alpha_t \in (0, 1)^{d_k} \tag{1}
$$

where $\alpha_t$ controls input-dependent forgetting and $\beta_t$ the update strength. The three terms attenuate the old state, remove content aligned with $k_t$, and write the new key–value association.

The GDN parameterization of the log-decay $g = \log \alpha_t$ is:

$$
g = -\exp(A_{\mathrm{log}}) \, \mathrm{softplus}(a + d t_{\mathrm{bias}}) \tag{2}
$$

where $A_{\mathrm{log}}$ and $dt_{\mathrm{bias}}$ are learned decay parameters and $a$ is the input-dependent gate logit. The **static retention horizon** is defined as the number of tokens required for the decay factor to fall to $\epsilon$:

$$
H_s = \frac{\ln \epsilon}{g} \tag{3}
$$

with $\epsilon = 10^{-3}$ empirically. The metric yields one $H_s$ per head for GDN and per key channel for KDA.

**Key finding:** KDA exhibits broad channel decay (median $\alpha = 0.471$, IQR [0.170, 0.774]) but predominantly slow GDN head decay (0.969, [0.894, 0.997]). At a 16-token cutoff, 90.6% of KDA heads contain channels on both sides, motivating **channel-wise selection for KDA** and **head-wise selection for GDN**.

---

## Methodology

### Decay-Aware Selection and Storage

DASC constructs a fixed selection mask by comparing each unit's static horizon $H_s(u)$ with a threshold $W_{\max}$:

- Unit is **global** (retained) when $H_s(u) > W_{\max}$
- Unit is **local** (omitted) otherwise

Selection follows the architecture's decay granularity: one complete head state for GDN and one key channel $(h,k)$ for KDA.

The compression ratios are defined as:

$$
C_{\mathrm{temp}}^{\mathrm{rag}} = \frac{\sum_{l=1}^{L} n_l}{\sum_{l=1}^{L} r_l}, \qquad C_{\mathrm{ckpt}}^{\mathrm{rag}} = \frac{B_{\mathrm{temp}}^{\mathrm{dense}} + B_{\mathrm{conv}}}{B_{\mathrm{temp}}^{\mathrm{rag}} + B_{\mathrm{conv}}} \tag{6}
$$

where $n_l$ and $r_l$ are total and retained state-unit counts in layer $l$, and $B_{\mathrm{temp}}$, $B_{\mathrm{conv}}$ denote temporal and convolution state bytes.

### Load Policies

**DASC-NR (default):** Loads retained global units exactly; leaves omitted local units at zero. No additional model computation.

**DASC-WR:** Replays a bounded suffix from a zero-initialized scratch recurrent state, refreshing only the local units. The replay window is:

$$
W_u = \mathrm{clamp}(\text{next\_pow2}(H_s(u)), 4, W_{\max}), \quad W_{\mathrm{replay}} = \max_{u \in \mathcal{L}_{\mathrm{local}}} W_u \tag{7}
$$

### TP-Balanced Placement

Two-stage balancing to avoid uneven state checkpoint payloads across tensor-parallel ranks:
1. **Model-load-time permutation** redistributes linear-attention head bundles across TP ranks (exact reindexing, no per-token communication).
2. **Storage-time redistribution** treats the TP group as a shared storage pool, using a variable-split all-to-all during STORE and inverse transfer during LOAD.

---

## Empirical Validation / Results

### Validation of Selection Signal

The weight-derived $H_s$ was validated against observed signals using Spearman correlations (Table 1(b)):

| Model | Pair | Median | IQR |
|-------|------|--------|-----|
| GDN | $H_s, H_e$ | .801 | [.699, .877] |
| GDN | $H_s, \|S\|_F$ | .632 | [.541, .706] |
| GDN | $H_s, \|o\|_2$ | .537 | [.422, .630] |
| KDA | $H_s, H_e$ | .856 | [.843, .878] |
| KDA | $H_s, \|S\|_F$ | .533 | [.462, .602] |
| KDA | $H_s, \|o\|_2$ | .402 | [.329, .477] |

### Quality Under Compression

**RULER benchmarks:** At $W_{\max} = 16$, Kimi-KDA scores 0.95/0.95/0.95 at 4k/8k/16k vs. dense 0.95/0.96/0.95; Qwen-GDN scores 0.84/0.83/0.81 vs. 0.84/0.83/0.80. Every aggregate is within 0.01 of dense. At $W_{\max} = 1024$, Kimi-KDA drops by 0.04–0.07 points.

**End-task quality (Table 3, key results):**

| Model | Benchmark | Dense | DASC ($W_{\max}=16$) | Random ($W_{\max}=16$) |
|-------|-----------|-------|---------------------|----------------------|
| Kimi-KDA | AIME 2026 | 0.6073 | 0.6042 | $0.5635 \pm 0.0356$ |
| Kimi-KDA | GPQA-Diamond | 0.6187 | $0.6133^{\dagger}$ | $0.5749 \pm 0.0189$ |
| Kimi-KDA | MMLU-Pro | 0.6750 | $0.6732^{\dagger}$ | $0.6550 \pm 0.0046$ |

† marks Holm-adjusted $p < 0.05$ for DASC vs. Random. At $W_{\max} = 16$, DASC remains within 1.3 percentage points of dense on every row and exceeds random selection in all 15 reasoning comparisons.

### Matched-HBM Serving Performance

- **State checkpoint capacity:** Channel-wise KDA achieves 2.63–28.04× compression across $W_{\max} = 16$–512; head-wise GDN achieves 1.10–2.48×.
- **Serving gains (KDA):** TTFT reduced by 25.4–42.6%, input throughput improved by 39.3–68.4%. At $W_{\max}=16$: TTFT from 567.6→326.0 ms; throughput from 33.25→55.98 k tokens/s.
- **Serving gains (GDN):** At $W_{\max}=512$: TTFT from 614.4→374.5 ms; throughput from 30.56→50.96 k tokens/s.

### DASC-WR Accuracy Recovery

At $W_{\max} = 128$, DASC-WR improves all five end-task estimates over DASC-NR by 0.8–7.4 percentage points, matching or exceeding Dense on HMMT and GPQA. At $W_{\max} = 512$, it recovers 7.02 points and remains within 0.31 points of Dense while retaining 25.4% lower TTFT.

### Composition with Quantization

Combining $W_{\max} = 16$ channel selection with INT8 state checkpoint storage yields **8.11× capacity** relative to dense mixed-precision reference, with negligible quality loss (Table 4).

---

## Theoretical and Practical Implications

### Theoretical Contributions

1. **Retention heterogeneity characterization:** The paper provides the first systematic analysis showing that recurrent-state units in hybrid linear-attention models have widely different retention horizons that follow architectural structure (head-wise for GDN, channel-wise for KDA).

2. **Input-independent selection signal:** The weight-derived horizon $H_s$ provides a principled, calibration-free metric for unit selection, validated against empirical horizons (Spearman $\rho \approx 0.8$–0.86) and observed activity.

3. **Causal ablation insights:** Removing recurrent state checkpoints changes accuracy by at most one point on single-key retrieval, revealing task-specific redundancy—but the paper shows selection must be conservative because states matter for multi-key, aggregation, and reasoning tasks.

### Practical Implications

1. **Serving efficiency:** DASC enables more state checkpoints to fit in the same HBM budget, directly reducing TTFT and improving throughput for prefix-reuse workloads.

2. **Architecture-dependent gains:** Channel-wise decay (KDA) yields dramatically better compression than head-wise decay (GDN), motivating finer-grained decay parameterization in future model designs.

3. **Complementarity:** DASC is orthogonal to existing techniques (KV quantization, eviction policies, replay systems) and can be composed with them for compounding benefits.

---

## Conclusion

DASC turns retention heterogeneity into an input-independent state checkpoint policy for hybrid serving. On Kimi-KDA, it remains near dense quality while providing 2.63× state checkpoint capacity, 42.6% lower mean TTFT, and 68.4% higher input-token throughput under matched HBM. Suffix replay recovers accuracy at aggressive compression; Qwen-GDN extends DASC to head-wise decay.

**Future directions** include:
- Transfer to purely linear-attention or state-space models
- Scaling to larger TP degrees
- Finer-grained compression for head-decayed architectures (e.g., per-channel within GDN)

**Limitations:** All serving experiments use TP8; benefits are architecture-dependent, with GDN yielding smaller gains than KDA due to coarser decay granularity.

---

_Markdown view of https://picx.dev/p/rIhP0n, served by PicX — AI-generated visual whiteboard summaries of research papers._