# Who Asks, Who Answers: Linear Attention Builds the Query in Hybrid Language Models

> Hybrid language models divide retrieval labor: linear attention builds the softmax reader's query, not just compressing context.

- **Source:** [arXiv](https://arxiv.org/abs/2611.01909)
- **Published:** 2026-11-07
- **Permalink:** https://picx.dev/p/6V58uA
- **Whiteboard:** https://picx.dev/p/6V58uA/image

## Summary

## Summary (Overview)

- **Core finding**: This paper identifies a division of labor in hybrid language models (HLMs) for in-context retrieval: **linear attention constructs the query**, while **softmax attention performs the lookup**. This challenges the prior assumption that linear attention merely compresses context while softmax attention handles all retrieval.
- **Encode–Address circuit**: The authors discover a two-part circuit: an early **Encode block** preserves token identity (replaceable by a simple linear projection), and an **Address block** immediately preceding the softmax reader constructs the final-position representation that determines which dictionary entry the reader retrieves.
- **Causal interventions**: Transplanting the Address block's output between prompts with different queried keys redirects retrieval in 0.97–0.99 of cases, even when removing the block leaves accuracy intact—showing the representation is causally sufficient for determining retrieval.
- **Cross-architecture transfer**: The Address block's computation transfers between **Gated DeltaNet** (Qwen3.5) and **Mamba-2** (Granite-4.0-H) without updating weights, achieving 0.92–0.94 retrieval accuracy via linear alignment maps.
- **Scaling insight**: Address computation becomes distributed in larger models (≥27B), where a chain of blocks guides successive softmax readers rather than a single block guiding all readers.

---

## Introduction and Theoretical Foundation

### Background: Hybrid Language Models

Hybrid language models interleave linear-attention layers (e.g., Gated DeltaNet, Mamba-2) with a small number of softmax-attention layers. Linear attention compresses past context into a fixed-size state with linear scaling in sequence length; softmax attention retains an explicit cache for exact retrieval. Recent models like Qwen3.5 and Granite-4.0-H use a 3:1 ratio of linear to softmax layers.

### Research Question

The paper asks: **What do linear-attention layers compute to support retrieval by softmax attention?** Prior work attributes retrieval to softmax attention (e.g., Gather-and-Aggregate heads in Bick et al., 2025) and treats linear attention as context compression. However, removing the earliest linear-attention layer reduces retrieval accuracy to at most 0.05 across models, even with all softmax layers intact—implying linear attention is actively involved.

### Theoretical Basis

- Recurrent models struggle with associative recall due to fixed-size state compression (Jelassi et al., 2024; Arora et al., 2024a)
- Hybrid models restore retrieval via softmax layers (Waleffe et al., 2024)
- The paper extends mechanistic interpretability of Transformers (induction heads, Gather-and-Aggregate) to hybrid architectures, asking **how the softmax reader's query is constructed** rather than only where retrieval occurs

The key theoretical contribution is showing that the Address block provides the **query** (not the keys) that the softmax reader uses for retrieval—a function not previously attributed to linear attention.

---

## Methodology

### Task Design

**Controlled in-context key–value retrieval task**: Each prompt contains a dictionary of eight key–value pairs (e.g., "cherry=steel, mango=wood") followed by a question about one key. The assistant response is prefilled so the value is predicted as the next token.

Three prompt formats are used:
- **chat8**: standard format with "Question: What is the value of mango?"
- **list8**: bulleted lookup table format
- **rev8**: value-first format ("iron is the value of elder")

All four models reach ≥0.99 accuracy on chat8. This synthetic task isolates retrieval from parametric knowledge and reasoning.

### Models Studied

| Model | Parameters | Layers (linear/softmax) | Recurrent Mechanism |
|-------|-----------|------------------------|---------------------|
| Qwen3.5-0.8B | 0.8B | 18/6 | Gated DeltaNet |
| Qwen3.5-9B | 9B | 24/8 | Gated DeltaNet |
| Granite4.0-1B | 1B | ~36/4 | Mamba-2 |
| Granite4.0-7B | 7B (1B active) | 36/4 (MoE) | Mamba-2 |
| Qwen3.5-27B | 27B | 48/16 | Gated DeltaNet |
| Qwen3.5-35B-A3B | 35B (3B active) | 30/10 (MoE) | Gated DeltaNet |

### Key Analysis Techniques

**1. Progressive layer pruning**: Truncate network after layer $L$ and measure retrieval accuracy (normalized as $(\mathrm{acc} - \mathrm{chance})/(\mathrm{acc_{full}} - \mathrm{chance})$, where chance $= 1/8$).

**2. Block-level interventions** (blocks = maximal sequences of consecutive linear-attention layers):
- **Removal**: Set a block's token-mixer output to zero at every position, leaving MLPs and softmax layers intact
- **Transplant**: Replace a block's mixer output at the final position in host prompt B with the corresponding output from donor prompt A (same dictionary, different queried key)

**3. Metrics**:
- **Accuracy**: fraction of correct next-token predictions
- **Logit gap**: correct-minus-alternative logit differences (swapping values across paired prompts)
- **Reader attention**: attention mass from final position to queried dictionary entry
- **Selectivity**: difference between log attention on queried entry and mean log attention on other entries
- **Top-1 rate**: fraction where queried entry gets most attention

**4. Query/Key intervention**: Recompute attention with query or keys computed with/without the Address block to determine which component it controls.

**5. Cross-architecture transfer**: Fit linear alignment maps (ridge regression) between models' paired activations without updating weights.

---

## Empirical Validation / Results

### Retrieval Emerges Abruptly at a Softmax Layer

Progressive pruning of Qwen3.5-9B reveals:
- Knowledge-focused performance (6 benchmarks) develops **gradually** with depth
- Retrieval stays at chance level until **layer 19** (softmax layer $L_{19}$), then jumps to 0.67 normalized accuracy
- Retaining 24/32 layers preserves 0.94 of retrieval accuracy but only 0.47 of knowledge performance

### The Encode–Address Circuit

**Removal results (Qwen3.5-9B)**:

| Intervention | Accuracy | Logit Gap | Reader Attention |
|-------------|----------|-----------|------------------|
| Intact | 1.00 | +23.8 | 0.83 |
| Remove $G_0$ (Encode) | 0.01 | 0.01 | 0.06 |
| Remove $G_4$ (Address) | 1.00 | +11.6 | 0.06 |
| Remove $G_1$–$G_3$ | ~1.00 | — | 0.53–0.73 |

**Transplant results**: Only transplanting $G_4$ redirects retrieval (0.99 success); transplanting any other block has no effect (0.00). The reader's attention shifts to the donor's entry. Results hold across formats (0.97 on list8, 0.98 on rev8).

### The Role of Encode

| Patched at swapped tokens | Value swap | Key swap |
|---------------------------|-----------|----------|
| Encode block output | 0.98 | 0.98 |
| Input token embedding | 1.00 | 1.00 |
| Any other block's output | 0.00 | 0.00 |

Replacing $G_0$ with a token-wise linear projection (ridge regression) preserves retrieval accuracy (1.00) but increases WikiText-2 perplexity from 10.6 to 44.7—Encode's retrieval role is a simple linear re-encoding, though the original block handles other language-modeling tasks.

### The Address Mechanism: Query vs. Keys

Query–key interventions show the Address block primarily controls the **query**:

| Condition | Selectivity | Top-1 rate |
|-----------|-------------|------------|
| Original (with $G_4$) | 5.45 | 1.00 |
| Query without $G_4$ | 1.36 | ~0.50 |
| Keys without $G_4$ | 4.80 | 1.00 |

### Recurrent State Carries the Retrieval Target

- Transplanting $G_4$'s **recurrent state** (from the penultimate token) redirects retrieval in 0.99 of cases
- Transplanting $G_0$ or $G_3$ recurrent states has no effect (0.00)
- Replacing the host's input to $G_4$ at the final position (keeping host state) fails (0.00)

### Cross-Architecture Transfer

**Output transplant success rates** (donor → receiver, chat8):

| Donor → Receiver | Cross-model | Same-model |
|------------------|-------------|------------|
| Qwen3.5-0.8B → Granite4.0-1B | 0.98 | 1.00 |
| Granite4.0-1B → Qwen3.5-0.8B | 0.91 | 0.97 |
| Qwen3.5-9B → Granite4.0-7B$^a$ | 0.99 | 1.00 |
| Granite4.0-7B → Qwen3.5-9B$^b$ | 0.83 | 0.92 |

**Block replacement accuracy**:

| Donor → Receiver | Removed | Linear proj. | Zeroed mixer | Donor Address |
|------------------|---------|--------------|--------------|---------------|
| Qwen3.5-0.8B → Granite4.0-1B | 0.01 | 0.24 | 0.28 | **0.92** |
| Granite4.0-1B → Qwen3.5-0.8B | 0.30 | 0.28 | 0.28 | **0.95** |
| Qwen3.5-9B → Granite4.0-7B$^a$ | 0.00 | 0.14 | 0.18 | **0.94** |
| Granite4.0-7B → Qwen3.5-9B$^b$ | 0.35 | 0.30 | 0.21 | 0.46 |

$a$: Both Address blocks of Granite4.0-7B replaced.
$b$: Qwen3.5-9B evaluated after L19 (intact accuracy: 0.75).

Key finding: Successful transfer requires the donor's **recurrent computation**—not a linear transformation (max 0.28) or the block's MLPs alone (max 0.18). The reverse transfer (7B → 9B) is harder because distributed computation across multiple blocks plus a softmax layer doesn't compose well.

### Scaling Results

| Model | Address block → guided readers | Removal | Transplant |
|-------|-------------------------------|---------|------------|
| Qwen3.5-0.8B | $G_3$ → L15, L19, L23 | 0.06 | 0.99 |
| Qwen3.5-2B | $G_3$ → L15, L19, L23 | 0.41 | 1.00 |
| Qwen3.5-4B | $G_4$ → L19, L23, L27 | 0.94 | 0.70 |
| Qwen3.5-9B | $G_4$ → L19, L23, L27 | 0.92 | 0.98 |
| Qwen3.5-27B | $G_8$–$G_{10}$ → L39, L43; $G_{11}, G_{12}$ → L51, L59 | 1.00 | 0.98 |
| Qwen3.5-35B-A3B | $G_6$ → L27; $G_7$ → L31; $G_8$ → L35 | 0.34 | 0.75 |

Key scaling insight: In larger models (≥27B), the query **no longer passes from one reader to the next through attention**. Each Address block guides a successive reader independently, making computation distributed and more redundant (removal accuracy stays high in 27B).

---

## Theoretical and Practical Implications

### Theoretical Contributions

1. **Reinterpreting linear attention's role**: Linear attention is not merely a memory-compression mechanism; it actively constructs the retrieval query. This refines the understanding of division of labor in hybrid architectures.

2. **The Address mechanism**: The recurrent state of the Address block carries the retrieval target, using it to build the representation from which softmax attention forms its query. This is a novel function not previously attributed to recurrent layers.

3. **Cross-architecture universality**: The finding that Address blocks transfer between Gated DeltaNet and Mamba-2 suggests a **shared retrieval mechanism** across hybrid architectures, potentially reflecting a general principle rather than an implementation artifact.

### Practical Implications

1. **Model design**: When deciding which linear-attention layers to remove, distill, or redesign, the Encode and Address blocks are the critical ones for retrieval. Intermediate blocks are more expendable.

2. **Interpretability tools**: The transplantation methodology provides a causal framework for testing whether a representation determines retrieval, beyond correlational pruning analyses.

3. **Scaling-aware design**: Larger models' distributed Address computation suggests that capacity naturally leads to specialized per-reader address blocks, which may inform how to structure future hybrid models.

---

## Conclusion

This paper reveals a clear division of labor in hybrid language models: **linear attention constructs the query, while softmax attention performs the lookup**. The Encode–Address circuit shows that:

1. The early Encode block re-encodes token identity in a way replaceable by a linear projection
2. The Address block determines which entry the softmax reader retrieves by carrying the retrieval target in its recurrent state and building the query representation
3. This computation transfers across architectures (Gated DeltaNet ↔ Mamba-2) and becomes distributed in larger models

**Future directions** include:
- Systematic comparison of dense vs. MoE architectures regarding Address robustness
- Understanding why Granite4.0-7B→Qwen3.5-9B transfer underperforms the reverse direction
- Extending the analysis beyond synthetic key–value retrieval to real-world benchmarks
- Exploring whether other hybrid components (e.g., MLPs) have similarly underappreciated retrieval roles

The central message is that query construction—not just memory compression—is a key function of linear attention in hybrid architectures, offering a new lens for designing and interpreting these models.

---

_Markdown view of https://picx.dev/p/6V58uA, served by PicX — AI-generated visual whiteboard summaries of research papers._