Summary (Overview)

• Hybrid Linear Attention (HLA) is a novel query-dependent chunk-level attention mechanism for Gated DeltaNet (GDN) that enables selective access to sparse, distant information in long-context linear attention models.
• HLA represents each completed chunk as an exact affine state transition and computes content-dependent routing gates from compact, self-attentively pooled representatives, interpolating each historical transition with the identity map.
• The method improves over native GDN and fixed chunk mixing (MHLA) across Qwen3.5 models (0.8B–9B), with gains up to 5.57 percentage points on LongBench-V2 and 3.97 points on RULER.
• In a controlled from-scratch 1.3B setting trained on 100B tokens with 4K context, HLA improves RULER performance from 4K to 32K, with gains increasing from 0.83 points at 4K to 4.22 points at 32K, demonstrating generalization beyond training context.
• HLA introduces effective-support regularization to encourage concentrated routing for sparse inference, with only modest latency overhead (9.8–14.9% prefill, 13.9–18.8% decode).

---

Introduction and Theoretical Foundation

Background and Motivation

Softmax attention remains the standard for long-context modeling due to its content-dependent retrieval capabilities, but its KV cache grows linearly with context length, making long-context inference memory-intensive. Linear attention offers an alternative by compressing history into fixed-size recurrent states, but this compression limits expressivity—particularly the ability to represent sharp, highly selective attention patterns when relevant evidence is sparse.

The paper identifies two key limitations of existing recurrent linear attention:

1. Progressive attenuation of early information: In GLA, historical state writes are repeatedly modulated by subsequent recurrent gates, causing early source positions to have substantially lower cumulative retention. In GDN, early information must propagate through long chains of affine transitions, leading to progressive attenuation or overwriting.

2. Input-independent chunk mixing: MHLA applies fixed learned weights to combine chunk states, which cannot adapt to input- and query-dependent relevance variations.

Theoretical Foundation

The paper builds on Gated DeltaNet (GDN), which uses data-dependent decay and delta-rule updates for selective state modification. The recurrence is expressed as:

$$S_t = A_t S_{t-1} + B_t, \tag{1}$$

where

$$A_t = \exp(g_t) \left(I - \beta_t \bar{k}_t^{\top} \bar{k}_t\right), \qquad B_t = \beta_t \bar{k}_t^{\top} v_t. \tag{2}$$

Since affine maps are closed under composition, token-level updates within a chunk can be summarized exactly by a chunk-level pair $(A_j, B_j)$:

$$S_j = A_j S_{j-1} + B_j. \tag{3}$$

---

Methodology

Chunk-Wise Affine Transitions

The sequence is partitioned into $N = \lceil T / L \rceil$ contiguous chunks of at most $L$ tokens. Each chunk is summarized as an exact affine state transition $(A_j, B_j)$, where $B_j$ represents the chunk's additive contribution and $A_j$ determines how it transforms earlier memory.

Pooled Chunk Representations

Each completed chunk is divided into $P = L / p$ non-overlapping pooling windows of $p$ tokens. For window $b$ of chunk $j$, representative keys are computed via self-attention within the window:

$$q_i^r = q_i W_Q^r, \qquad K_{j,b}^p = X_{j,b} W_K^p, \qquad V_{j,b}^p = X_{j,b} W_V^p. \tag{4}$$ $$\bar{k}_{j,b}^r = \mathrm{Mean}_{\mathrm{rows}} \left[ \mathrm{softmax} \left(\frac{K_{j,b}^p (K_{j,b}^p)^{\top}}{\sqrt{d_k}}\right) V_{j,b}^p \right] W_O^p. \tag{5}$$

Chunk-Level Attention Weights

Query–representative similarities are aggregated using log-mean-exp:

$$s_{i,j} = \log \left[ \frac{1}{P} \sum_{b=1}^{P} \exp \left(\frac{q_i^r (\bar{k}_{j,b}^r)^{\top}}{\sqrt{d_k}}\right) \right]. \tag{6}$$

The chunk-level attention weight is computed via sigmoid:

$$w_{i,j} = \sigma(s_{i,j}), \quad \sigma(x) = \frac{1}{1 + \exp(-x)}. \tag{7}$$

The current chunk has weight $w_{i,c(i)} = 1$, and future chunks are masked with $w_{i,j} = 0$. At inference, weights below threshold $\theta = 0.1$ are set to zero.

Query-Conditioned Historical States

For query token $i$, historical chunks are applied chronologically with query-dependent interpolation:

$$S_{i,j} = \left[ (1 - w_{i,j}) I + w_{i,j} A_j \right] S_{i,j-1} + w_{i,j} B_j. \tag{8}$$

The final query-conditioned state is:

$$S_i^{HLA} = A_{c(i),\leq i} S_{i,c(i)-1} + B_{c(i),\leq i}. \tag{9}$$

Setting all visible attention weights to one recovers the original GDN recurrence.

Training Objective

For pretrained adaptation, the backbone is frozen and only routing/pooling modules are optimized:

$$\mathcal{L} = \mathcal{L}_{\mathrm{LM}} + \lambda_b \mathcal{L}_{\mathrm{budget}} + \lambda_s \mathcal{L}_{\mathrm{support}}, \tag{10}$$

where $\mathcal{L}_{\mathrm{LM}}$ is next-token cross-entropy. The effective support is measured as:

$$K_i^{\mathrm{eff}} = \left(\sum_{j < c(i)} \pi_{i,j}^2\right)^{-1}, \tag{12}$$

with normalized weights $\pi_{i,j}$ as in Eq. (11). The regularizers are:

$$\mathcal{L}_{\mathrm{budget}} = \mathbb{E}\left[ \max\left(0, \frac{G_i - K_b}{K_b}\right)^2 \right], \quad \mathcal{L}_{\mathrm{support}} = \mathbb{E}\left[ \max\left(0, K_i^{\mathrm{eff}} - K_s\right)^2 \right]. \tag{13}$$

---

Empirical Validation / Results

From-Scratch 1.3B Results on RULER

Table 1 shows HLA and GDN trained from scratch with 100B tokens and 4K context, evaluated from 4K to 32K:

HLA improves GDN by 0.83, 2.67, 3.07, and 4.22 points at 4K, 8K, 16K, and 32K respectively, with the gap widening beyond training context.

Qwen3.5 Long-Context Evaluation

Table 2 (RULER) shows HLA achieves the highest average score at every model scale:

Table 3 (LongBench-V2) shows consistent improvements:

Ablation Studies

• Chunk size: Best at $L=256$ (29.62% on LongBench-V2), with performance dropping at 512 and 1024 due to trade-off between retrieval granularity and contextual completeness.
• Pooling window: Best at $p=32$ (29.62%), with smaller windows retaining excessive redundancy and larger windows discarding fine-grained information.
• Memory scaling: Affine-state memory decreases approximately inversely with chunk size—at $L=256$, historical-state storage is reduced 2× compared to matched KV cache while achieving the highest accuracy.

Inference Efficiency

Table 4 shows latency overhead:

The relative overhead decreases with model scale.

---

Theoretical and Practical Implications

Theoretical Significance

• Query-dependent composition of recurrent memory: HLA demonstrates that making the composition of historical state transitions explicitly query-dependent substantially improves long-context retrieval in linear attention models, addressing a fundamental limitation of fixed-weight approaches.
• Exact affine summarization: The use of exact chunk-wise affine transitions (rather than approximations) preserves the full effect of each chunk while enabling selective access.
• Effective-support regularization: The use of order-2 effective number to encourage concentrated routing provides a principled way to balance expressivity and inference efficiency.

Practical Implications

• Lightweight adaptation: HLA requires only training the routing and pooling modules while freezing the backbone, making it practical for adapting existing pretrained GDN-based models.
• Beyond-training-context generalization: HLA improves performance at context lengths far exceeding the training context (4K→32K), suggesting it enhances the intrinsic long-context capability of recurrent models.
• Favorable memory-accuracy trade-off: The chunk-wise representation reduces historical-state storage by 2× at the default chunk size while improving accuracy.

---

Conclusion

HLA presents a query-dependent chunk-level attention framework for GDN that represents completed chunks as affine state transitions and dynamically modulates their composition based on the current query. The method controls both new memory and the transformation of earlier history, achieving consistent improvements over native GDN and fixed chunk mixing across pretrained Qwen3.5 models (0.8B–9B) and in controlled from-scratch experiments. The performance gains become more pronounced at longer contexts beyond the training distribution, demonstrating the value of query-dependent composition of compact recurrent-memory summaries for long-context modeling.

Future directions include evaluating HLA in demanding interactive settings such as long-running agentic systems, where long-range memory must be continuously accumulated and selectively reused over extended trajectories (multi-turn tool use, persistent memory, and goal tracking).