# MARCH: Scaling Recurrent Memory with Content-Routed State Anchors

> MARCH augments recurrent state-space models with content-routed checkpoints of historical memory states, improving long-context retrieval and reasoning while maintaining linear-time training efficiency.

- **Source:** [arXiv](https://arxiv.org/abs/2608.12435)
- **Published:** 2026-08-17
- **Permalink:** https://picx.dev/p/lHtyVs
- **Whiteboard:** https://picx.dev/p/lHtyVs/image

## Summary

## Summary (Overview)

- **MARCH (Memory-Anchor Routing across Context History)** is a novel architecture that augments recurrent state-space models with content-routed state anchors, enabling selective retrieval from earlier versions of the evolving recurrent memory without modifying the underlying recurrence.
- The method periodically checkpoints cumulative recurrent states as "state anchors," each paired with a compact content-conditioned key, creating a memory bank that grows with context length.
- MARCH uses an attention-style routing mechanism over state anchors (rather than token-level key-value pairs), including a learned null option to bypass historical memory when the current state is sufficient.
- Extensive experiments show MARCH consistently outperforms Gated DeltaNet and its Log-Linear variant across commonsense reasoning (Avg. 41.5 vs 40.1/40.0), LongBench (Avg. 14.9 vs 11.9/12.5), in-context retrieval (Avg. 23.3 vs 19.2/20.5), and NIAH tasks, including robust extrapolation beyond training context length.
- The architecture achieves competitive training efficiency: at 128K tokens, Top-4 sparse MARCH more than doubles the throughput of dense MARCH and exceeds FlashAttention-2 throughput.

---

## Introduction and Theoretical Foundation

### Background and Motivation

Large language models require effective long-context modeling, which depends on a model's ability to **manage memory**—determining what to preserve, how to represent it, and when to retrieve it. The paper frames sequence models as memory systems with two operations:

- **Writing**: incorporating new input into memory
- **Reading**: retrieving information relevant to the current input

**Standard self-attention** maintains a growing token-level memory in its key-value cache, providing a direct path to every preceding token. However, this incurs:
- Quadratic computation during training: $O(T^2)$
- Key-value cache growing linearly during autoregressive inference: $O(T(d_k + d_\nu))$ per attention head

**Recurrent alternatives** (linear attention, state-space models) compress history into a fixed-size state:
- Enable constant-memory recurrent decoding
- But underperform on recall-intensive tasks since earlier associations get overwritten by subsequent updates

### Theoretical Foundation

**Memory system formulation**: Let $\mathbf{x}_t \in \mathbb{R}^d$ denote the hidden representation at position $t$. The query, key, and value vectors are obtained via:

$$
\mathbf{q}_t = \mathbf{W}_q \mathbf{x}_t, \quad \mathbf{k}_t = \mathbf{W}_k \mathbf{x}_t, \quad \mathbf{v}_t = \mathbf{W}_\nu \mathbf{x}_t,\tag{1}
$$

The general memory system is:

$$
\mathcal{M}_t = \text{Write}(\mathcal{M}_{t-1}; \mathbf{k}_t, \mathbf{v}_t), \quad \mathbf{o}_t = \text{Read}(\mathcal{M}_t; \mathbf{q}_t),\tag{2}
$$

**Linear attention** instantiates memory as a fixed-size matrix $\mathbf{S}_t \in \mathbb{R}^{d_\nu \times d_k}$:

$$
\mathbf{S}_t = \mathbf{S}_{t-1} + \mathbf{v}_t \mathbf{k}_t^\top, \qquad \mathbf{o}_t = \mathbf{S}_t \mathbf{q}_t.\tag{5}
$$

**Gated DeltaNet (GDN)** improves writes with input-dependent retention and delta-rule updates:

$$
\mathbf{S}_t = \alpha_t \mathbf{S}_{t-1} + \beta_t(\mathbf{v}_t - \alpha_t \mathbf{S}_{t-1} \mathbf{k}_t) \mathbf{k}_t^\top, \qquad \mathbf{o}_t = \mathbf{S}_t \mathbf{q}_t,\tag{6}
$$

where $\alpha_t \in (0,1)$ is an input-dependent retention gate, and $\beta_t \in [0,1]$ modulates the delta update strength.

**Two approaches to scaling recurrent memory** (Equation 7):
- **Capacity expansion**: $\mathcal{M}_t^{\text{cap}} := \widetilde{\mathbf{S}}_t = [\mathbf{S}_t^{(1)} \mid \dots \mid \mathbf{S}_t^{(P)}]$ — increases memory dimension
- **Temporal expansion**: $\mathcal{M}_t^{\text{temp}} := (\overline{\mathbf{S}}_{t,1}, \ldots, \overline{\mathbf{S}}_{t,M_t})$ — retains multiple versions of the state along its trajectory

MARCH follows the **temporal expansion** direction by retaining cumulative snapshots of a continuously evolving recurrent state.

---

## Methodology

### 3.1 Continuous Recurrent-State Anchoring

**Anchor placement**: For a sequence of $L$ text tokens, an anchoring policy specifies boundaries $\mathcal{B} = \{b_m\}_{m=1}^{M}$ where $0 = b_0 < b_1 < \dots < b_M \leq L$. An anchor position is inserted after each boundary:

$$
\widehat{\mathcal{T}} = \|_{m=1}^{M} \left([t_{b_{m-1}+1}, \ldots, t_{b_m}] \parallel [\xi_m]\right) \parallel [t_{b_M+1}, \ldots, t_L],\tag{8}
$$

where $\xi_m$ is the $m$-th occurrence of a shared learned anchor embedding $\xi$.

**Cumulative recurrent-state checkpointing**: At each boundary $b_m$, the current state is snapshotted:

$$
\mathbf{A}^{(m,\ell)} = \mathbf{S}_{b_m}^{(\ell)} \in \mathbb{R}^{d_\nu \times d_k}, \qquad m = 1, \ldots, M.\tag{9}
$$

Since the recurrence is not reset between boundaries, $\mathbf{A}^{(m,\ell)}$ encodes the **cumulative prefix** up to position $b_m$.

**Content-conditioned anchor metadata**: The anchor position reads its aligned state checkpoint:

$$
\mathbf{q}_m^{(\ell)} = \mathbf{W}_q^{(\ell)} \mathbf{u}_m^{(\ell)}, \qquad \mathbf{o}_m^{(\ell)} = \mathbf{A}^{(m,\ell)} \mathbf{q}_m^{(\ell)}.\tag{10}
$$

The same input is projected into a compact routing key:

$$
\pmb{\kappa}_m^{(\ell)} = \pmb{\mathsf{W}}_k^{(\ell)} \pmb{\mathsf{u}}_m^{(\ell)} \in \mathbb{R}^{d_r}.\tag{11}
$$

This cross-layer construction makes routing keys explicitly dependent on the **content** of each state anchor, not just its temporal index.

### 3.2 Content-Routed Historical Reading

**Content-based routing**: For a text token at position $t$, the visible anchors are $\mathcal{V}_t = \{m \in \{1,\dots,M\} \mid b_m < t\}$. The routing query scores against anchor keys:

$$
\pmb{\rho}_t = \mathbf{W}_R \mathbf{x}_t, \qquad a_{t,m} = \pmb{\rho}_t^\top \pmb{\kappa}_m, \quad m \in \mathcal{V}_t.\tag{12}
$$

A **null option** $\varnothing$ with zero payload $\mathbf{A}^{(\varnothing)} = \mathbf{0}$ allows bypassing historical memory. The augmented logits become:

$$
s_{t,j} = \begin{cases} a_{t,j}, & j \in \mathcal{V}_t, \\ n_t, & j = \varnothing, \end{cases} \qquad \pi_{t,j} = \frac{\exp(s_{t,j})}{\sum_{r \in \widetilde{\mathcal{V}}_t} \exp(s_{t,r})}.\tag{13}
$$

A **Top-K sparse variant** restricts aggregation to the $K$ highest-scoring visible anchors, reducing computation with minimal performance loss.

**Historical retrieval and residual fusion**:

$$
\mathbf{o}_t = \mathbf{S}_t \mathbf{q}_t + \sum_{j \in \widetilde{\mathcal{V}}_t} \pi_{t,j} \mathbf{A}^{(j)} \mathbf{q}_t.\tag{14}
$$

The additive formulation preserves the original recurrent path while introducing historical retrieval as an auxiliary residual branch.

### 3.3 Implementation

MARCH uses a **two-stage producer–reader computation**:
- **Producer**: Processes recurrent updates in blocks (chunkwise Gated DeltaNet), computes current-state outputs, and checkpoints states at anchor boundaries
- **Reader**: Jointly tiles query tokens and state anchors, reuses anchor tiles across query blocks, and fuses routing-score computation, online softmax updates, and weighted state readout accumulation into a streaming reduction

This avoids materializing the dense token-to-anchor routing matrix or per-anchor candidate readouts.

---

## Empirical Validation / Results

### Experimental Setup

- **Pretraining**: 50B tokens from Long-Data-Collections, sequence length 16K
- **Architecture**: 21 layers, hidden size 1536 (GDN: 793M params; Transformer: 693M)
- **MARCH settings**: routing dimension $d_r = 64$, anchoring interval $C = 512$ tokens
- **Optimization**: fused AdamW, $\beta_1 = 0.9$, $\beta_2 = 0.95$, $\epsilon = 10^{-8}$, weight decay 0.1, peak LR $4 \times 10^{-4}$

### Commonsense Reasoning (Table 1)

| Model | LMB.acc↑ | PIQA.acc↑ | Hella.acc↑ | Wino.acc↑ | ARC-e acc_n↑ | ARC-c acc_n↑ | OBQA acc_n↑ | CSQA acc↑ | Avg. |
|---|---|---|---|---|---|---|---|---|---|
| Transformer | 49.4 | 66.5 | 33.9 | 52.1 | 47.8 | 26.4 | 32.0 | 22.1 | 41.3 |
| w/ 24 Layers | 50.3 | 67.6 | 34.4 | 50.6 | 46.3 | 25.8 | 31.2 | 24.7 | 41.4 |
| Gated DeltaNet | 48.5 | 66.1 | 33.1 | 50.8 | 45.9 | 25.3 | 30.0 | 21.1 | 40.1 |
| w/ Log-Linear | 47.7 | 65.7 | 33.2 | 51.9 | 44.3 | 24.9 | 30.4 | 21.7 | 40.0 |
| **w/ MARCH** | **49.5** | **66.9** | **34.8** | **52.6** | **47.1** | **25.6** | **32.8** | **22.5** | **41.5** |

MARCH improves average accuracy from 40.1/40.0 to **41.5**, with the largest gain on OpenBookQA (+2.8 points), and surpasses both Transformer baselines on average.

### LongBench Results (Table 2)

| Model | NQA | QQA | MFQ | HQA | 2WM | Mus | GvR | QMS | MNs | TRC | TQA | SSM | Avg.↑ |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Transformer | 4.4 | 4.1 | 15.9 | 7.5 | 9.9 | 4.1 | 10.7 | 11.6 | 14.5 | 21.0 | 33.9 | 28.3 | 13.8 |
| w/ 24 Layers | 3.5 | 11.1 | 18.3 | 7.8 | 9.8 | 4.1 | 11.3 | 12.9 | 12.8 | 22.5 | 47.5 | 23.2 | 15.4 |
| Gated DeltaNet | 3.0 | 4.8 | 13.3 | 5.6 | 8.7 | 2.1 | 2.6 | 11.3 | 13.0 | 18.0 | 38.6 | 21.9 | 11.9 |
| w/ Log-Linear | 3.6 | 6.2 | 13.6 | 7.1 | 8.2 | 3.3 | 6.3 | 13.2 | 13.5 | 17.0 | 32.8 | 25.1 | 12.5 |
| **w/ MARCH** | **4.2** | **7.8** | **14.6** | **7.4** | **11.5** | **4.8** | **8.2** | **17.4** | **14.1** | **19.0** | **43.1** | **26.3** | **14.9 (↑25%)** |

Notable gains: 2WikiMultihopQA +32%, MuSiQue +45%, QMSum +32% relative improvements.

### In-Context Retrieval (Table 3)

| Model | SQuAD↑ | SWDE↑ | FDA↑ | TriviaQA↑ | DROP↑ | NQ↑ | Avg.↑ |
|---|---|---|---|---|---|---|---|
| Transformer | 41.3 | 59.3 | 80.4 | 2.2 | 2.9 | 1.7 | 31.3 |
| w/ 24 Layers | 40.4 | 64.9 | 83.7 | 4.0 | 3.4 | 2.5 | 33.2 |
| Gated DeltaNet | 34.8 | 45.0 | 31.4 | 1.1 | 2.2 | 0.8 | 19.2 |
| w/ Log-Linear | 33.7 | 46.1 | 38.2 | 1.3 | 2.6 | 1.0 | 20.5 |
| **w/ MARCH** | **37.7 (↑8%)** | **51.9 (↑13%)** | **44.6 (↑17%)** | **1.6 (↑23%)** | **2.9 (↑12%)** | **1.2 (↑20%)** | **23.3 (↑14%)** |

### NIAH Results

At 32K context (beyond 16K training length), MARCH achieves the best result on all six tasks, retaining perfect accuracy on S-NIAH-1, while both Transformer variants and Log-Linear GDN score zero throughout. MARCH wins in 19 of 24 task–length combinations and 11 of 12 multi-needle settings.

### Ablation: Chunk Size (Table 4)

- **Matched training/inference**: $C = 512$ provides the best balance (32 anchors)
- **Inference-time variation**: Denser anchors (smaller chunks) improve retrieval at higher cost; chunk size 256 at inference gives best retrieval (SQuAD 40.35, SWDE 53.38, FDA 47.46)
- **Fenwick tree organization**: Achieves performance close to Log-Linear Attention, demonstrating router generalizability

### Ablation: Routing Design (Table 5)

| Configuration | CS | LongBench | Retrieval | NIAH Avg. |
|---|---|---|---|---|
| Dense, $d_r=64$, Null (default) | **41.48** | **14.87** | 23.31 | **51.33** |
| Dense, $d_r=192$, Null | 40.94 | 13.88 | **24.52** | 48.47 |
| Top-4, $d_r=64$, Null | 41.38 | 13.79 | 23.17 | 44.85 |
| Dense, $d_r=64$, No Null | 41.04 | 14.11 | 22.86 | 45.98 |

- Increasing $d_r$ to 192 improves retrieval but hurts general performance
- Top-4 nearly matches dense routing on commonsense/retrieval but trails on NIAH (efficiency-oriented)
- Removing the null option degrades all aggregates

### Training Efficiency

At 128K tokens, Top-4 MARCH:
- More than doubles the training throughput of dense MARCH
- Reduces core runtime by roughly an order of magnitude
- Achieves higher throughput than FlashAttention-2

---

## Theoretical and Practical Implications

### Theoretical Significance

1. **Scaling recurrent memory beyond fixed-state bottleneck**: MARCH demonstrates that temporal expansion of recurrent states (rather than only capacity expansion) effectively alleviates the single-state memory bottleneck, providing a theoretical alternative to increasing state dimensionality.

2. **Content-routed retrieval over state anchors**: The paper establishes that attention-style content routing can be applied at the **state level** (over compressed historical states) rather than token level, decoupling memory capacity from dense per-token computation.

3. **Cross-layer state-conditioned routing keys**: The architecture shows that routing keys can be made content-dependent through the recurrent pathway itself, enabling state-aware retrieval without explicit supervision.

### Practical Implications

1. **Long-context efficiency**: MARCH offers a practical trade-off between memory resolution and computational cost, controllable via checkpoint density and sparse routing (Top-K).

2. **Extrapolation beyond training context**: Unlike Transformers (RoPE extrapolation failure) and Log-Linear GDN (state-index-dependent coefficients), MARCH's content-based router generalizes to longer contexts without new parameters.

3. **Compatibility**: MARCH augments existing recurrent architectures without modifying the underlying recurrence, making it a drop-in enhancement for Gated DeltaNet and potentially other linear recurrent models.

---

## Conclusion

### Key Takeaways

MARCH introduces a novel mechanism for scaling recurrent memory by preserving cumulative state checkpoints as **content-routed state anchors**. The architecture:
- Leaves the underlying recurrent update unchanged
- Adds a growing bank of historical memory states
- Uses attention-style routing over compact anchor keys with a learned null option
- Fuses historical readouts with current-state outputs via residual addition

### Results Summary

MARCH consistently outperforms strong recurrent baselines (Gated DeltaNet, Log-Linear Gated DeltaNet) across:
- **Commonsense reasoning**: +1.4 average points
- **LongBench**: +25% relative improvement
- **In-context retrieval**: +14% relative improvement
- **NIAH**: Best performance at all context lengths including 32K extrapolation

### Future Directions

1. **Adaptive anchoring**: Replace fixed-interval checkpointing with content-driven anchoring based on state novelty or update magnitude
2. **Memory consolidation**: Combine state anchoring with larger-capacity memory and multiple specialized partitions (short-term, episodic, consolidated knowledge)
3. **External memory modules**: Support downstream task optimization, knowledge consolidation from experience to parametric information
4. **Scaling directions**: Test-time training and continual learning applications

MARCH establishes **historical-state retrieval** as a practical approach to scaling recurrent memory beyond a single evolving state, opening new directions for efficient long-context modeling.

---

_Markdown view of https://picx.dev/p/lHtyVs, served by PicX — AI-generated visual whiteboard summaries of research papers._