Summary (Overview)

• This paper investigates the functional roles of attention (KV cache) and recurrent state in hybrid language models that combine both mechanisms, using two novel cache-level interventions: split-prefill (keeping only one channel from a prefilled context) and state-swap (pairing the KV cache from one context with the recurrent state from another).
• The authors demonstrate a sharp functional dissociation: exact retrieval (key-value lookup, list indexing) survives only through the attention KV cache (64–98% of full accuracy) and collapses to zero through the recurrent state, while output language and persona transfer survive only through the recurrent state (70–80% and 3–5× higher than KV-only).
• State-swap experiments confirm this causally: the answer takes its value from the KV side and its language from the recurrent side, with both properties jointly correct on ≥99% of items.
• The recurrent state is shown to preserve semantic fields rather than exact item identities, producing false-yes responses to associated-but-absent words (DRM-style) and memory-conjunction errors (feature recombination).
• The authors frame the distinction as parallel to the explicit/implicit memory distinction in cognitive psychology: attention provides lookup over what was said; the recurrent state shapes how the model says it next.

Introduction and Theoretical Foundation

Hybrid language models combine softmax attention with fixed-size recurrent blocks (e.g., Gated DeltaNet, Mamba) to achieve efficient long-context modeling. Modern open-weight families include:

• Qwen3.5 (Yang et al., 2025a) — 3:1 Gated DeltaNet to Gated Attention pattern
• Falcon-H1 (Zuo et al., 2025) — parallel-hybrid block running attention and SSMs alongside each other
• Jamba (Lieber et al., 2024) — interleaved block types

The central research question: What does each component actually carry at inference?

A natural first guess is temporal: attention is precise but expensive long-term memory; the recurrent state is a cheap short-term buffer. The paper challenges this view by demonstrating that the dissociation is functional — it concerns not the age of information each channel retains, but the type of access each channel provides.

Prior work established that fixed-state models struggle with exact copying, key–value retrieval, and needle-in-a-haystack lookup compared to attention (Arora et al., 2024; Park et al., 2024). This paper makes the stronger claim that the recurrent state does not function as an addressable memory for retrieval at inference, and examines what it does carry.

Methodology

Split-prefill

A prompt is split into a context $c$ and a query $q$. A forward pass over $c$ populates:

• The attention KV cache $\mathcal{K}(c)$
• The recurrent state $\mathcal{R}(c)$ (Gated DeltaNet for Qwen, Mamba for Falcon)

The query $q$ is then processed in three conditions:

1. FULL: keep both channels
2. KV-ONLY: keep $\mathcal{K}(c)$, drop $\mathcal{R}(c)$
3. REC-ONLY: keep $\mathcal{R}(c)$, drop $\mathcal{K}(c)$

Since $q$ is always processed normally, the intervention isolates the carry-over from $c$ in a specific channel.

State-swap

Two contexts $c_A$ and $c_B$ of the same format but different content are prefilled. A shared query $q$ is then run on the hybrid cache $(\mathcal{K}(c_A), \mathcal{R}(c_B))$ and its mirror $(\mathcal{K}(c_B), \mathcal{R}(c_A))$. Properties of the answer that follow $c_A$ are attributed to attention; those that follow $c_B$ to recurrence. This addresses the concern that split-prefill produces a partly out-of-distribution combination (both caches come from real prefills of well-formed contexts).

Diagnostic Tasks

Four small synthetic tasks, each isolating one capability:

1. KV retrieve: a small dictionary; query asks for a value under a key
2. List indexing: position-based retrieval
3. Language following: an "answer only in L" instruction; query in a different language; output language scored by a language-ID classifier
4. Persona: context describes a persona

Models

• Qwen3.5-4B (32 layers, Gated DeltaNet)
• Falcon-H1-3B-Instruct (32 layers, Mamba)

Results are reported as means over three seeds with 95% hierarchical bootstrap confidence intervals.

Empirical Validation / Results

Quantitative Dissociation

Table 1: Accuracy under split-prefill on four diagnostic tasks

Key findings:

• Retrieval tasks: KV-ONLY retains 64–98% of full-model accuracy; REC-ONLY collapses to zero. The collapse persists even in minimal versions (short dictionaries, short lists), ruling out a state-too-small explanation.
• Language following: REC-ONLY retains 70–80% of full accuracy; KV-ONLY collapses to ~1% — a 60–80× gap.
• Persona: REC-ONLY is 3–5× higher than KV-ONLY.

Case Study: Recall, but Not Control

Under KV-ONLY, the model can name the requested language correctly (e.g., "French") but answers in English. Under REC-ONLY, the pattern reverses. The KV cache lets the model report what the instruction was; to actually follow it, the recurrent state must be carried over from the context.

Case Study: Semantic Field Preservation

Using a DRM-style (Roediger and McDermott, 1995) list of words strongly associated with an absent target:

• KV-ONLY: both models correctly reject the associated-but-absent target ("No")
• REC-ONLY: false-yes rates rise sharply on the associated list while a matched neutral-absent control stays near floor

The recurrent state preserves the semantic field of seen items rather than their identities.

Case Study: Memory-Conjunction Errors

Features of two seen items recombine into an unseen item (e.g., "sun-" from sunflower and "-light" from moonlight form the compound "sunlight"). Under REC-ONLY, false-yes rates rise sharply on conjunction items but stay near floor on a single-part-absent control. The recurrence signal is driven by the simultaneous presence of both features — component features survive, but the binding that ties them together as a specific item is lost.

Theoretical and Practical Implications

1. Functional, not static, content: The results characterize what each channel causally drives at inference, not what it statically encodes. The KV cache can let the model name a requested language on demand even when it cannot make the model answer in that language.

2. Parallel to cognitive psychology: The pattern parallels the explicit/implicit memory distinction (Tulving, 1985; Schacter, 1987):

   • KV cache ≈ explicit memory: addressable, item-specific retrieval
   • Recurrent state ≈ implicit memory: shapes behavior without item-level access

3. Refines prior work: The paper strengthens the recall-deficiency story for state-space and linear-attention models (Arora et al., 2024) by providing causal evidence from hybrid models.

4. Design implications: Hybrid architectures that need exact retrieval should route retrieval queries through attention; the recurrent state should be relied upon for style, language, and response mode control.

Conclusion

In a trained hybrid LM, the two components do not differ in how long they retain past context — they differ in how the model accesses what they hold:

• The KV cache works as an addressable store: the model can pull specific items out of it.
• The recurrent state works as a compressed prior that shapes language, persona, and the semantic field of generation.

State-swap makes the attribution causal at the level of a single answer. Future work includes:

• Probing the static information content of attention and recurrent representations using trained probing classifiers on activations from each channel
• Mapping the full information content of each channel's representations

The authors explicitly note they do not claim either channel is empty of the other type of content — the dissociation concerns what each channel causally contributes to generation, not what it statically encodes.