# Triadic Linear Attention: Three-Dimensional Recurrent States for Long-Context Sequence Modeling

> Triadic linear attention raises RNN memory state to a third-order tensor via a second key, matching Transformers at 64k context with minimal parameter overhead.

- **Source:** [arXiv](https://arxiv.org/abs/2609.36529)
- **Published:** 2026-10-03
- **Permalink:** https://picx.dev/p/vGItOx
- **Whiteboard:** https://picx.dev/p/vGItOx/image

## Summary

## Summary (Overview)

- **Triadic linear attention** generalizes linear attention by writing a triadic outer product of a key, a second key, and a value into a third-order (3D) tensor state, increasing state capacity from $d^2$ to $d^2 E$ entries with minimal parameter overhead (only two extra projections).
- The construction is compatible with data-dependent forgetting (per-slice gates along the second-key axis), the delta rule (joint erase before write), and chunkwise-parallel training via state tiling.
- Applied to Gated DeltaNet (GDN) and scalar-gated linear attention (sGLA), triadic attention substantially improves long-context language modeling and recall at 400M and 1.3B parameter scales, outperforming alternatives that enlarge state via larger heads, wider values, more heads, or grouped values.
- A pretrained dyadic model can be "upcycled" to triadic form during long-context extension, recovering 50–75% of the gains of training from scratch.
- In GDN/Transformer hybrids, triadic states outperform larger key-value caches in perplexity while using significantly less memory at long contexts.

---

## Introduction and Theoretical Foundation

### Motivation

Recurrent neural networks (RNNs) compress history into a **fixed-size memory state**, allowing constant-time inference. The state size is crucial for performance—linear attention (Katharopoulos et al., 2020) extended vector-valued hidden states to matrix-valued states via a dyadic outer product of key and value vectors, enabling modern RNNs to rival Transformers. However, state size is fundamentally limited: a model with a fixed-size state cannot even copy sequences beyond a certain length (Jelassi et al., 2024), and modern linear attention models still degrade on recall-intensive and long-context tasks (Arora et al., 2024; Hsieh et al., 2024).

Naively increasing state size (more heads, larger value dimensions) grows parameter count significantly. The paper asks: **How can we increase state size parameter-efficiently?**

### Theoretical Foundation: Tensor Product Representations

The paper frames linear attention as a **tensor product representation** (Smolensky, 1990), where:
- The **value** is a filler, bound to a **role** (the key) via an outer product
- The state $S_t$ is a superposition of bindings
- Readout with the query is unbinding via contraction

Standard linear attention:
$$S_t = S_{t-1} + k_t v_t^\top, \quad o_t = S_t^\top q_t = \sum_{s \leq t} (q_t^\top k_s) v_s \tag{1}$$

This coincides with the **fast-weight programmer** of Schmidhuber (1992). The capacity is determined by the state shape: with $d$-dimensional keys/values, $S_t$ has $d^2$ entries and can separate at most $d$ mutually orthogonal keys.

**Key insight:** A binding itself is a vector (of dimension $d^2$), which can be bound to a further role. Each nesting raises the tensor order by one. With orthonormal roles, an $n$-th order tensor product representation can store $d^{n-1}$ associations exactly. Binding each value to a **second key** increases capacity from $d$ to $d^2$ associations.

---

## Methodology

### Triadic Linear Attention

Each position produces, besides $q_t, k_t, v_t \in \mathbb{R}^d$, a second key $k'_t$ and second query $q'_t \in \mathbb{R}^E$. The state is a third-order tensor $S_t \in \mathbb{R}^{d \times E \times d}$:

$$S_t = S_{t-1} + k_t \otimes k'_t \otimes v_t, \quad o_t = S_t \times_1 q_t \times_2 q'_t = \sum_{s \leq t} (q_t^\top k_s)(q'^\top_t k'_s) v_s \tag{2}$$

where $\otimes$ is the outer product and $\times_n$ is contraction along axis $n$. For $E=1$ with $k'_t = q'_t = 1$, this reduces to standard linear attention (Equation 1).

The state has $d^2 E$ entries (cubic in $d$ for $E=d$), while parameters grow only by the two projections for $k'_t$ and $q'_t$.

### Forgetting and Delta Rule

**Forgetting:** Each slice $S_t[:, e, :]$ gets its own decay gate $\alpha_{t,e}$:
$$S_t = S_{t-1} \times_2 \text{diag}(\alpha_t) + k_t \otimes k'_t \otimes v_t \tag{3}$$

**Delta rule:** The stored association is removed before writing:
$$S_t = S_{t-1} + \beta_t \left( k_t \otimes k'_t \otimes v_t - S_{t-1} \times_1 k_t \times_2 k'_t \right) \tag{4}$$

By flattening the joint key $\kappa_t = k_t \otimes k'_t \in \mathbb{R}^{d \cdot E}$, this can be rewritten as Gated DeltaNet with key dimension $d \cdot E$:
$$S_t = (I - \beta_t \kappa_t \kappa_t^\top) D_t S_{t-1} + \beta_t \kappa_t v_t^\top, \quad o_t = S_t^\top (q_t \otimes q'_t) \tag{5}$$

### Efficient Implementation

**Chunkwise-parallel form:** The sequence is split into chunks of $C$ positions. The masked attention within a chunk factorizes:
$$(q_r \otimes q'_r)^\top (k_s \otimes k'_s) = (q_r^\top k_s)(q'^\top_r k'_s)$$

so the attention matrix becomes $QK^\top \odot R'$ where $R' = \text{tril}(Q'K'^\top)$, reducing cost from $C^2(d \cdot E)$ to $C^2(d + E)$.

**Tiling the state:** The state is split along the value axis into blocks of 32 columns per thread block, so no streaming multiprocessor ever holds the full third-order state. At $E=8$, one head's state is 512 KiB (twice the register file of a Hopper SM), but one block is 128 KiB.

### Experimental Setup

- **Models:** 400M (24 layers, $d_{model}=1024$, 8 heads) and 1.3B (24 layers, $d_{model}=2048$, 16 heads) parameters, head dimension $d=128$.
- **Training:** 50 tokens per parameter (2.5× Chinchilla-optimal), pretrained on Fineweb-Edu with 4k context, then long-context extended to 64k on 5 tokens/parameter.
- **Baselines:** Larger heads, wider values, grouped values, more heads—all matched for state size and parameter count. Transformer with RoPE, QK-norm, GQA-8.
- **Evaluation:** PG19 perplexity by context position, WikiText perplexity, recall suite from Arora et al. (2024), RULER needle-in-a-haystack.

---

## Empirical Validation / Results

### Capacity in Isolation (MQAR)

On multi-query associative recall (MQAR), every doubling of $E$ shifts the accuracy curve right by roughly a doubling of $N$. For $E=16$, the model stores ~16× more associations than ordinary linear attention, with only a 1.08× increase in non-embedding parameters.

### Main Results

**Figure 2** shows that at 400M and 1.3B scales, Triadic GDN with $E=8$ predicts the next token better than the Transformer even at 64k context, and substantially improves recall on tasks requiring copying from long documents (FDA, SWDE). RULER needle-in-a-haystack accuracy stays high up to longer contexts with larger states.

### State-Matched Comparison

**Table 1** compares triadic attention to alternative state-enlargement methods:

| Model | State (MB) | Params (M) | Wiki. ↓ | PG19 ≤4k ↓ | PG19 4k–16k ↓ | PG19 16k–64k ↓ | Recall ↑ |
|---|---|---|---|---|---|---|---|
| GDN (base) | 6.3 | 380.9 | 11.25 | 15.03 | 14.38 | 14.15 | 26.2 |
| Larger heads ($d$=256) | 12.6 | 380.7 | 11.22 | 15.10 | 14.42 | 14.17 | 26.6 |
| Grouped values (2/key) | 12.6 | 378.2 | 11.18 | 15.05 | 14.37 | 14.12 | 27.1 |
| Wider values ($d_v$=256) | 12.6 | 377.9 | 11.20 | 15.07 | 14.39 | 14.14 | 27.4 |
| More heads (16) | 12.6 | 381.6 | 11.32 | 15.25 | 14.54 | 14.28 | 27.5 |
| **Triadic ($E$=2)** | **12.6** | **381.9** | **11.08** | **14.90** | **14.21** | **13.95** | **28.4** |
| **Triadic ($E$=4)** | **25.2** | **383.0** | **10.93** | **14.80** | **14.08** | **13.79** | **31.1** |

Triadic attention achieves consistently lower perplexity on every range and highest recall. Alternative methods barely improve at 2× and degrade at 4× due to MLP width reduction needed to match parameters.

### Upcycling

A pretrained model can be expanded from $E=1$ to $E=8$ by copying forget gates to every slice and initializing second key/query projections from scratch, then performing long-context extension. Upcycled models recover 50–75% of the gains of training from scratch.

### Hybrid Models

In a 3:1 GDN/GQA-8 hybrid, triadic state ($E=4$) achieves the lowest perplexity at every context range and best NIAH average, while using less memory than enlarging the key-value cache beyond ~4.6k tokens (almost half the memory at 64k).

### Training Efficiency

Triadic GDN with $E=8$ is 3% slower than Transformer at 4k tokens and 5.1× faster at 64k. Overhead over vanilla GDN: 28–30% for $E=8$, 14–15% for $E=4$, 9–11% for $E=2$.

### Ablations

- **Key dimensions:** At fixed joint key dimension $d_k \cdot E = 1024$, keeping $d_k$ large (128) and $E$ small (8) slightly outperforms balanced splits.
- **Activation:** Non-negative activations (softplus, sigmoid) outperform signed activations (SiLU, linear) for the second key/query, since signed entries allow cancellation across slices.

---

## Theoretical and Practical Implications

### Theoretical Significance

1. **Tensor product view of memory:** The paper provides a principled framework for increasing RNN state capacity by raising the tensor order, connecting to Smolensky's tensor product representations and higher-order associative memories (Baldi & Venkatesh, 1987), where storable patterns scale as $d^{n-1}$.

2. **Parameter-efficiency:** The ratio of state size to parameter count increases asymptotically—an $E$-dimensional second key gives an $E$-fold state increase with only two extra projections.

3. **Memory dynamics analysis:** Triadic GDN learns substantially larger delta-rule write strengths ($\beta$ mean increases from 0.31 to 0.50) and develops a broad range of memory timescales: median half-lives range from 0.20 tokens (shortest-lived slice) to 1080 tokens (longest-lived), versus a single 5.1-token half-life for vanilla GDN.

### Practical Implications

1. **Long-context modeling:** Triadic linear attention enables pure linear RNNs to match or exceed Transformer perplexity at 64k context with a smaller state than the Transformer's key-value cache.

2. **Hybrid architectures:** Enlarging linear-attention states is more parameter- and memory-efficient than enlarging softmax attention key-value caches in hybrid models.

3. **Upcycling:** Pretrained dyadic models can be converted to triadic during long-context extension, suggesting a staged training recipe where state size grows with context length.

4. **Training cost:** The 15–30% training overhead for $E=4$/$E=8$ is a practical tradeoff, but the models remain faster than Transformers beyond a few thousand tokens.

---

## Conclusion

The paper introduces **triadic linear attention**, which updates a third-order tensor state with the outer product of three vectors (key, second key, value), increasing the capacity of constant-time sequence mixers with minimal parameter overhead. Key findings:

- Triadic Gated DeltaNet consistently lowers perplexity and improves recall at 400M and 1.3B scales, outperforming all alternative state-enlargement methods at matched state size.
- The construction admits efficient chunkwise-parallel kernels (via Kronecker structure factorization and state tiling), keeping training within 1.3× of vanilla GDN for $E=8$.
- The larger state is genuinely used for distant context, with memory slices developing diverse timescales and stronger write strengths.

**Limitations:** Training slowdown (15–30%), performance on some recall tasks still trails Transformers, and the construction is only applied to GDN and sGLA.

**Future directions:** The third-order state opens possibilities for entirely new linear attention variants; revisiting other matrix-state developments (e.g., test-time training) for 3D states; applying triadic construction to mixture-of-experts architectures where smaller key/query projections matter.

---

## Key Formulas

| Equation | Description |
|---|---|
| (1) | Standard linear attention: $S_t = S_{t-1} + k_t v_t^\top$ |
| (2) | Triadic linear attention: $S_t = S_{t-1} + k_t \otimes k'_t \otimes v_t$ |
| (3) | With per-slice forgetting: $S_t = S_{t-1} \times_2 \text{diag}(\alpha_t) + k_t \otimes k'_t \otimes v_t$ |
| (4) | With delta rule: $S_t = S_{t-1} + \beta_t(k_t \otimes k'_t \otimes v_t - S_{t-1} \times_1 k_t \times_2 k'_t)$ |
| (5) | Joint-key form: $S_t = (I - \beta_t \kappa_t \kappa_t^\top) D_t S_{t-1} + \beta_t \kappa_t v_t^\top$ |
| (6) | Chunkwise parallel form (standard) |
| (7) | Factorized masked attention: $QK^\top \odot R'$ |
| (8) | Tiled state chunkwise form |

---

_Markdown view of https://picx.dev/p/vGItOx, served by PicX — AI-generated visual whiteboard summaries of research papers._