# What Attention Recalls and Recurrence Controls in Hybrid Language Models

> In hybrid language models, attention's KV cache enables exact item retrieval while the recurrent state controls output language and persona, revealing a functional dissociation.

- **Source:** [arXiv](https://arxiv.org/abs/2609.04434)
- **Published:** 2026-09-12
- **Permalink:** https://picx.dev/p/TuhAnW
- **Whiteboard:** https://picx.dev/p/TuhAnW/image

## Summary

## Summary (Overview)

- This paper investigates the functional roles of attention (KV cache) and recurrent state in hybrid language models that combine both mechanisms, using two novel cache-level interventions: **split-prefill** (keeping only one channel from a prefilled context) and **state-swap** (pairing the KV cache from one context with the recurrent state from another).
- The authors demonstrate a sharp **functional dissociation**: exact retrieval (key-value lookup, list indexing) survives only through the attention KV cache (64–98% of full accuracy) and collapses to zero through the recurrent state, while output language and persona transfer survive only through the recurrent state (70–80% and 3–5× higher than KV-only).
- State-swap experiments confirm this causally: the answer takes its **value** from the KV side and its **language** from the recurrent side, with both properties jointly correct on ≥99% of items.
- The recurrent state is shown to preserve **semantic fields** rather than exact item identities, producing false-yes responses to associated-but-absent words (DRM-style) and memory-conjunction errors (feature recombination).
- The authors frame the distinction as parallel to the **explicit/implicit memory** distinction in cognitive psychology: attention provides lookup over what was said; the recurrent state shapes how the model says it next.

## Introduction and Theoretical Foundation

Hybrid language models combine softmax attention with fixed-size recurrent blocks (e.g., Gated DeltaNet, Mamba) to achieve efficient long-context modeling. Modern open-weight families include:
- **Qwen3.5** (Yang et al., 2025a) — 3:1 Gated DeltaNet to Gated Attention pattern
- **Falcon-H1** (Zuo et al., 2025) — parallel-hybrid block running attention and SSMs alongside each other
- **Jamba** (Lieber et al., 2024) — interleaved block types

The central research question: *What does each component actually carry at inference?*

A natural first guess is **temporal**: attention is precise but expensive long-term memory; the recurrent state is a cheap short-term buffer. The paper challenges this view by demonstrating that the dissociation is **functional** — it concerns not the age of information each channel retains, but the *type of access* each channel provides.

Prior work established that fixed-state models struggle with exact copying, key–value retrieval, and needle-in-a-haystack lookup compared to attention (Arora et al., 2024; Park et al., 2024). This paper makes the stronger claim that the recurrent state does not function as an addressable memory for retrieval at inference, and examines what it *does* carry.

## Methodology

### Split-prefill
A prompt is split into a context $c$ and a query $q$. A forward pass over $c$ populates:
- The attention KV cache $\mathcal{K}(c)$
- The recurrent state $\mathcal{R}(c)$ (Gated DeltaNet for Qwen, Mamba for Falcon)

The query $q$ is then processed in three conditions:
1. **FULL**: keep both channels
2. **KV-ONLY**: keep $\mathcal{K}(c)$, drop $\mathcal{R}(c)$
3. **REC-ONLY**: keep $\mathcal{R}(c)$, drop $\mathcal{K}(c)$

Since $q$ is always processed normally, the intervention isolates the carry-over from $c$ in a specific channel.

### State-swap
Two contexts $c_A$ and $c_B$ of the same format but different content are prefilled. A shared query $q$ is then run on the hybrid cache $(\mathcal{K}(c_A), \mathcal{R}(c_B))$ and its mirror $(\mathcal{K}(c_B), \mathcal{R}(c_A))$. Properties of the answer that follow $c_A$ are attributed to attention; those that follow $c_B$ to recurrence. This addresses the concern that split-prefill produces a partly out-of-distribution combination (both caches come from real prefills of well-formed contexts).

### Diagnostic Tasks
Four small synthetic tasks, each isolating one capability:
1. **KV retrieve**: a small dictionary; query asks for a value under a key
2. **List indexing**: position-based retrieval
3. **Language following**: an "answer only in L" instruction; query in a different language; output language scored by a language-ID classifier
4. **Persona**: context describes a persona

### Models
- Qwen3.5-4B (32 layers, Gated DeltaNet)
- Falcon-H1-3B-Instruct (32 layers, Mamba)

Results are reported as means over three seeds with 95% hierarchical bootstrap confidence intervals.

## Empirical Validation / Results

### Quantitative Dissociation

**Table 1: Accuracy under split-prefill on four diagnostic tasks**

| Task | Qwen3.5-4B FULL | Qwen3.5-4B REC | Qwen3.5-4B KV | Falcon-H1-3B FULL | Falcon-H1-3B REC | Falcon-H1-3B KV |
|------|-----------------|----------------|---------------|-------------------|------------------|-----------------|
| KV retrieve | ~0.97 | ~0.00 | ~0.64–0.98×baseline | ~0.94 | ~0.00 | ~0.64–0.98×baseline |
| List indexing | ~0.97 | ~0.00 | ~0.64–0.98×baseline | ~0.94 | ~0.00 | ~0.64–0.98×baseline |
| Language follow | 0.97 | 0.70 | 0.01 | 0.94 | 0.79 | 0.01 |
| Persona | 0.69 | 0.49 | 0.14 | 0.71 | 0.28 | 0.06 |

Key findings:
- **Retrieval tasks**: KV-ONLY retains 64–98% of full-model accuracy; REC-ONLY collapses to zero. The collapse persists even in minimal versions (short dictionaries, short lists), ruling out a state-too-small explanation.
- **Language following**: REC-ONLY retains 70–80% of full accuracy; KV-ONLY collapses to ~1% — a 60–80× gap.
- **Persona**: REC-ONLY is 3–5× higher than KV-ONLY.

### Case Study: Recall, but Not Control
Under KV-ONLY, the model can *name* the requested language correctly (e.g., "French") but answers in English. Under REC-ONLY, the pattern reverses. The KV cache lets the model report what the instruction was; to actually follow it, the recurrent state must be carried over from the context.

### Case Study: Semantic Field Preservation
Using a DRM-style (Roediger and McDermott, 1995) list of words strongly associated with an absent target:
- **KV-ONLY**: both models correctly reject the associated-but-absent target ("No")
- **REC-ONLY**: false-yes rates rise sharply on the associated list while a matched neutral-absent control stays near floor

The recurrent state preserves the **semantic field** of seen items rather than their identities.

### Case Study: Memory-Conjunction Errors
Features of two seen items recombine into an unseen item (e.g., "sun-" from *sunflower* and "-light" from *moonlight* form the compound "sunlight"). Under REC-ONLY, false-yes rates rise sharply on conjunction items but stay near floor on a single-part-absent control. The recurrence signal is driven by the **simultaneous presence of both features** — component features survive, but the binding that ties them together as a specific item is lost.

## Theoretical and Practical Implications

1. **Functional, not static, content**: The results characterize what each channel *causally drives* at inference, not what it statically encodes. The KV cache can let the model name a requested language on demand even when it cannot make the model answer in that language.

2. **Parallel to cognitive psychology**: The pattern parallels the explicit/implicit memory distinction (Tulving, 1985; Schacter, 1987):
   - **KV cache** ≈ explicit memory: addressable, item-specific retrieval
   - **Recurrent state** ≈ implicit memory: shapes behavior without item-level access

3. **Refines prior work**: The paper strengthens the recall-deficiency story for state-space and linear-attention models (Arora et al., 2024) by providing causal evidence from hybrid models.

4. **Design implications**: Hybrid architectures that need exact retrieval should route retrieval queries through attention; the recurrent state should be relied upon for style, language, and response mode control.

## Conclusion

In a trained hybrid LM, the two components do not differ in how long they retain past context — they differ in how the model accesses what they hold:

- The **KV cache** works as an **addressable store**: the model can pull specific items out of it.
- The **recurrent state** works as a **compressed prior** that shapes language, persona, and the semantic field of generation.

State-swap makes the attribution causal at the level of a single answer. Future work includes:
- **Probing** the static information content of attention and recurrent representations using trained probing classifiers on activations from each channel
- Mapping the full information content of each channel's representations

The authors explicitly note they do not claim either channel is empty of the other type of content — the dissociation concerns what each channel *causally contributes to generation*, not what it statically encodes.

---

_Markdown view of https://picx.dev/p/TuhAnW, served by PicX — AI-generated visual whiteboard summaries of research papers._