Summary (Overview)

• MARCH (Memory-Anchor Routing across Context History) is a novel architecture that augments recurrent state-space models with content-routed state anchors, enabling selective retrieval from earlier versions of the evolving recurrent memory without modifying the underlying recurrence.
• The method periodically checkpoints cumulative recurrent states as "state anchors," each paired with a compact content-conditioned key, creating a memory bank that grows with context length.
• MARCH uses an attention-style routing mechanism over state anchors (rather than token-level key-value pairs), including a learned null option to bypass historical memory when the current state is sufficient.
• Extensive experiments show MARCH consistently outperforms Gated DeltaNet and its Log-Linear variant across commonsense reasoning (Avg. 41.5 vs 40.1/40.0), LongBench (Avg. 14.9 vs 11.9/12.5), in-context retrieval (Avg. 23.3 vs 19.2/20.5), and NIAH tasks, including robust extrapolation beyond training context length.
• The architecture achieves competitive training efficiency: at 128K tokens, Top-4 sparse MARCH more than doubles the throughput of dense MARCH and exceeds FlashAttention-2 throughput.

---

Introduction and Theoretical Foundation

Background and Motivation

Large language models require effective long-context modeling, which depends on a model's ability to manage memory—determining what to preserve, how to represent it, and when to retrieve it. The paper frames sequence models as memory systems with two operations:

• Writing: incorporating new input into memory
• Reading: retrieving information relevant to the current input

Standard self-attention maintains a growing token-level memory in its key-value cache, providing a direct path to every preceding token. However, this incurs:

• Quadratic computation during training: $O(T^2)$
• Key-value cache growing linearly during autoregressive inference: $O(T(d_k + d_\nu))$ per attention head

Recurrent alternatives (linear attention, state-space models) compress history into a fixed-size state:

• Enable constant-memory recurrent decoding
• But underperform on recall-intensive tasks since earlier associations get overwritten by subsequent updates

Theoretical Foundation

Memory system formulation: Let $\mathbf{x}_t \in \mathbb{R}^d$ denote the hidden representation at position $t$. The query, key, and value vectors are obtained via:

$$\mathbf{q}_t = \mathbf{W}_q \mathbf{x}_t, \quad \mathbf{k}_t = \mathbf{W}_k \mathbf{x}_t, \quad \mathbf{v}_t = \mathbf{W}_\nu \mathbf{x}_t,\tag{1}$$

The general memory system is:

$$\mathcal{M}_t = \text{Write}(\mathcal{M}_{t-1}; \mathbf{k}_t, \mathbf{v}_t), \quad \mathbf{o}_t = \text{Read}(\mathcal{M}_t; \mathbf{q}_t),\tag{2}$$

Linear attention instantiates memory as a fixed-size matrix $\mathbf{S}_t \in \mathbb{R}^{d_\nu \times d_k}$:

$$\mathbf{S}_t = \mathbf{S}_{t-1} + \mathbf{v}_t \mathbf{k}_t^\top, \qquad \mathbf{o}_t = \mathbf{S}_t \mathbf{q}_t.\tag{5}$$

Gated DeltaNet (GDN) improves writes with input-dependent retention and delta-rule updates:

$$\mathbf{S}_t = \alpha_t \mathbf{S}_{t-1} + \beta_t(\mathbf{v}_t - \alpha_t \mathbf{S}_{t-1} \mathbf{k}_t) \mathbf{k}_t^\top, \qquad \mathbf{o}_t = \mathbf{S}_t \mathbf{q}_t,\tag{6}$$

where $\alpha_t \in (0,1)$ is an input-dependent retention gate, and $\beta_t \in [0,1]$ modulates the delta update strength.

Two approaches to scaling recurrent memory (Equation 7):

• Capacity expansion: $\mathcal{M}_t^{\text{cap}} := \widetilde{\mathbf{S}}_t = [\mathbf{S}_t^{(1)} \mid \dots \mid \mathbf{S}_t^{(P)}]$ — increases memory dimension
• Temporal expansion: $\mathcal{M}_t^{\text{temp}} := (\overline{\mathbf{S}}_{t,1}, \ldots, \overline{\mathbf{S}}_{t,M_t})$ — retains multiple versions of the state along its trajectory

MARCH follows the temporal expansion direction by retaining cumulative snapshots of a continuously evolving recurrent state.

---

Methodology

3.1 Continuous Recurrent-State Anchoring

Anchor placement: For a sequence of $L$ text tokens, an anchoring policy specifies boundaries $\mathcal{B} = \{b_m\}_{m=1}^{M}$ where $0 = b_0 < b_1 < \dots < b_M \leq L$. An anchor position is inserted after each boundary:

$$\widehat{\mathcal{T}} = \|_{m=1}^{M} \left([t_{b_{m-1}+1}, \ldots, t_{b_m}] \parallel [\xi_m]\right) \parallel [t_{b_M+1}, \ldots, t_L],\tag{8}$$

where $\xi_m$ is the $m$-th occurrence of a shared learned anchor embedding $\xi$.

Cumulative recurrent-state checkpointing: At each boundary $b_m$, the current state is snapshotted:

$$\mathbf{A}^{(m,\ell)} = \mathbf{S}_{b_m}^{(\ell)} \in \mathbb{R}^{d_\nu \times d_k}, \qquad m = 1, \ldots, M.\tag{9}$$

Since the recurrence is not reset between boundaries, $\mathbf{A}^{(m,\ell)}$ encodes the cumulative prefix up to position $b_m$.

Content-conditioned anchor metadata: The anchor position reads its aligned state checkpoint:

$$\mathbf{q}_m^{(\ell)} = \mathbf{W}_q^{(\ell)} \mathbf{u}_m^{(\ell)}, \qquad \mathbf{o}_m^{(\ell)} = \mathbf{A}^{(m,\ell)} \mathbf{q}_m^{(\ell)}.\tag{10}$$

The same input is projected into a compact routing key:

$$\pmb{\kappa}_m^{(\ell)} = \pmb{\mathsf{W}}_k^{(\ell)} \pmb{\mathsf{u}}_m^{(\ell)} \in \mathbb{R}^{d_r}.\tag{11}$$

This cross-layer construction makes routing keys explicitly dependent on the content of each state anchor, not just its temporal index.

3.2 Content-Routed Historical Reading

Content-based routing: For a text token at position $t$, the visible anchors are $\mathcal{V}_t = \{m \in \{1,\dots,M\} \mid b_m < t\}$. The routing query scores against anchor keys:

$$\pmb{\rho}_t = \mathbf{W}_R \mathbf{x}_t, \qquad a_{t,m} = \pmb{\rho}_t^\top \pmb{\kappa}_m, \quad m \in \mathcal{V}_t.\tag{12}$$

A null option $\varnothing$ with zero payload $\mathbf{A}^{(\varnothing)} = \mathbf{0}$ allows bypassing historical memory. The augmented logits become:

$$s_{t,j} = \begin{cases} a_{t,j}, & j \in \mathcal{V}_t, \\ n_t, & j = \varnothing, \end{cases} \qquad \pi_{t,j} = \frac{\exp(s_{t,j})}{\sum_{r \in \widetilde{\mathcal{V}}_t} \exp(s_{t,r})}.\tag{13}$$

A Top-K sparse variant restricts aggregation to the $K$ highest-scoring visible anchors, reducing computation with minimal performance loss.

Historical retrieval and residual fusion:

$$\mathbf{o}_t = \mathbf{S}_t \mathbf{q}_t + \sum_{j \in \widetilde{\mathcal{V}}_t} \pi_{t,j} \mathbf{A}^{(j)} \mathbf{q}_t.\tag{14}$$

The additive formulation preserves the original recurrent path while introducing historical retrieval as an auxiliary residual branch.

3.3 Implementation

MARCH uses a two-stage producer–reader computation:

• Producer: Processes recurrent updates in blocks (chunkwise Gated DeltaNet), computes current-state outputs, and checkpoints states at anchor boundaries
• Reader: Jointly tiles query tokens and state anchors, reuses anchor tiles across query blocks, and fuses routing-score computation, online softmax updates, and weighted state readout accumulation into a streaming reduction

This avoids materializing the dense token-to-anchor routing matrix or per-anchor candidate readouts.

---

Empirical Validation / Results

Experimental Setup

• Pretraining: 50B tokens from Long-Data-Collections, sequence length 16K
• Architecture: 21 layers, hidden size 1536 (GDN: 793M params; Transformer: 693M)
• MARCH settings: routing dimension $d_r = 64$, anchoring interval $C = 512$ tokens
• Optimization: fused AdamW, $\beta_1 = 0.9$, $\beta_2 = 0.95$, $\epsilon = 10^{-8}$, weight decay 0.1, peak LR $4 \times 10^{-4}$

Commonsense Reasoning (Table 1)

MARCH improves average accuracy from 40.1/40.0 to 41.5, with the largest gain on OpenBookQA (+2.8 points), and surpasses both Transformer baselines on average.

LongBench Results (Table 2)

Notable gains: 2WikiMultihopQA +32%, MuSiQue +45%, QMSum +32% relative improvements.

In-Context Retrieval (Table 3)

NIAH Results

At 32K context (beyond 16K training length), MARCH achieves the best result on all six tasks, retaining perfect accuracy on S-NIAH-1, while both Transformer variants and Log-Linear GDN score zero throughout. MARCH wins in 19 of 24 task–length combinations and 11 of 12 multi-needle settings.

Ablation: Chunk Size (Table 4)

• Matched training/inference: $C = 512$ provides the best balance (32 anchors)
• Inference-time variation: Denser anchors (smaller chunks) improve retrieval at higher cost; chunk size 256 at inference gives best retrieval (SQuAD 40.35, SWDE 53.38, FDA 47.46)
• Fenwick tree organization: Achieves performance close to Log-Linear Attention, demonstrating router generalizability

Ablation: Routing Design (Table 5)

• Increasing $d_r$ to 192 improves retrieval but hurts general performance
• Top-4 nearly matches dense routing on commonsense/retrieval but trails on NIAH (efficiency-oriented)
• Removing the null option degrades all aggregates

Training Efficiency

At 128K tokens, Top-4 MARCH:

• More than doubles the training throughput of dense MARCH
• Reduces core runtime by roughly an order of magnitude
• Achieves higher throughput than FlashAttention-2

---

Theoretical and Practical Implications

Theoretical Significance

1. Scaling recurrent memory beyond fixed-state bottleneck: MARCH demonstrates that temporal expansion of recurrent states (rather than only capacity expansion) effectively alleviates the single-state memory bottleneck, providing a theoretical alternative to increasing state dimensionality.

2. Content-routed retrieval over state anchors: The paper establishes that attention-style content routing can be applied at the state level (over compressed historical states) rather than token level, decoupling memory capacity from dense per-token computation.

3. Cross-layer state-conditioned routing keys: The architecture shows that routing keys can be made content-dependent through the recurrent pathway itself, enabling state-aware retrieval without explicit supervision.

Practical Implications

1. Long-context efficiency: MARCH offers a practical trade-off between memory resolution and computational cost, controllable via checkpoint density and sparse routing (Top-K).

2. Extrapolation beyond training context: Unlike Transformers (RoPE extrapolation failure) and Log-Linear GDN (state-index-dependent coefficients), MARCH's content-based router generalizes to longer contexts without new parameters.

3. Compatibility: MARCH augments existing recurrent architectures without modifying the underlying recurrence, making it a drop-in enhancement for Gated DeltaNet and potentially other linear recurrent models.

---

Conclusion

Key Takeaways

MARCH introduces a novel mechanism for scaling recurrent memory by preserving cumulative state checkpoints as content-routed state anchors. The architecture:

• Leaves the underlying recurrent update unchanged
• Adds a growing bank of historical memory states
• Uses attention-style routing over compact anchor keys with a learned null option
• Fuses historical readouts with current-state outputs via residual addition

Results Summary

MARCH consistently outperforms strong recurrent baselines (Gated DeltaNet, Log-Linear Gated DeltaNet) across:

• Commonsense reasoning: +1.4 average points
• LongBench: +25% relative improvement
• In-context retrieval: +14% relative improvement
• NIAH: Best performance at all context lengths including 32K extrapolation

Future Directions

1. Adaptive anchoring: Replace fixed-interval checkpointing with content-driven anchoring based on state novelty or update magnitude
2. Memory consolidation: Combine state anchoring with larger-capacity memory and multiple specialized partitions (short-term, episodic, consolidated knowledge)
3. External memory modules: Support downstream task optimization, knowledge consolidation from experience to parametric information
4. Scaling directions: Test-time training and continual learning applications

MARCH establishes historical-state retrieval as a practical approach to scaling recurrent memory beyond a single evolving state, opening new directions for efficient long-context modeling.