Replay Memory Breaks the Fixed-State Ceiling of Linear Attention
Highlights of This Issue
The strongest signal this issue is that two routes simultaneously challenge the presupposition that 'the fixed-state capacity of linear attention determines its long-context capability.' History Replays Itself proposes NoteBank, a training-free sparse episodic memory: before a question arises, it archives text segments using the backbone's own key/query features; when the question arrives, it addresses with native features and replays reading evidence through native forward computation. It decouples 'evidence storage' from 'recurrent computation,' improving GDN's average accuracy on six RULER retrieval tasks from 1.7% to 70.3% at 1M tokens, with computational overhead <0.025% and persistent storage about 1/2600 of full MHA KV cache. This directly advances 'Just Let Linear States Forget the Distant Past' from Issue 8—the latter uses near-suffix replay to reconstruct recurrent states, while NoteBank goes further by using the backbone's native features for writing and addressing, without external retrievers or additional training.
On the indexer and sparse attention side, FreeSolo designs the long-range branch as a dual-branch structure where 'summaries only locate, evidence remains in original tokens': local RoPE window + long-range NoPE sparse reads, using maximum key norm as a query-agnostic pruning priority, achieving zero-shot length extrapolation through short CPT. Its linear variant FS-L can retrieve a single needle at 256K under 4K training, and FS-B's hierarchical tree search achieves 100/88/84 on S-NIAH-1/2/3 at 16K. Relative to SPIN's indexer temporal correlation prediction from Issue 9, its increment is providing a training-free indexer family with 'norm pruning + mean key routing + hierarchical tree search' and complete degradation curves.
A new evidence chain emerges for memory interaction in hybrid architectures. GateSight is the first to systematically use the write/forget signals (write gate, decay, write magnitude) of GDN recurrent memory as query-agnostic KV eviction signals: it uses three layers of observation to prove these signals predict future KV utility before the question arises, paired with a lightweight MLP estimator, leading KVzip by 9.5-47.2 points on SCBench/RULER under one-tenth the attention KV budget. Relative to DeltaS from Issue 7, which uses GDN state changes to guide eviction, its increment is using the write/forget signals themselves as cross-memory prediction signals. Meanwhile, AbridgeKV advances trainable compression to online compression of the model's own long CoT outputs: replacing the original KV of generated blocks with the KV rows of learned pause tokens, with code rows located inside the replaced blocks, allocating budget by layer-measured value, matching full KV model accuracy and output length with a quarter cache under 32K generation.
The final two are methodological increments for indexers and compressors. Joint Subspace Sparse Attention advances the low-dimensional scoring objective of token-level sparse attention from query/key reconstruction to joint query-key bilinear score preservation, providing a closed-form spectral solution via a solvable upper bound and proving that directly optimizing the exact objective yields no gain; its gather-free indexed-attention kernel is the first to support per-KV-head independent selection under GQA, achieving 6.43× decoding throughput on B200 at 128K context. Latent Compaction treats KV cache compression itself as a trainable neural network (document-specific KV slots), using test-time training to optimize the compressed cache for each document under frozen model weights, with two-stage training and three initialization types, achieving 96.7% accuracy at 32× compression on RULER (vs. KVZip+'s 40.0%).
Open Questions
- In History Replays Itself, the replayed state is discarded after retrieval, but replay reading temporarily advances the recurrent state—does the replayed text interfere with the recurrent state that subsequent generation depends on? Also, does native feature addressing degrade on tasks with dense competing keys, as SWA NoPE does as revealed in 'Shifting Mechanisms' from Issue 8?
- FreeSolo's norm pruning is query-agnostic, and the paper itself admits it is not robust to 'dense, independently queryable facts'—does this pruning still hold on long contexts with dense decoys? Can it complement SPIN's temporal correlation prediction from Issue 9?
- GateSight's MLP is trained once on general text to predict future KV utility—does this prediction generalize across domains? Can it be combined with trainable indexers (e.g., SPIN from Issue 9), or are the two redundant in signal?
- AbridgeKV's code rows are located inside the replaced blocks—does this in-block positioning limit compression ratio or cross-question reuse? Can it be combined with Latent Compaction's per-document test-time optimization?
Papers in this issue
NoteBank's training-free sparse episodic memory lifts Gated DeltaNet's 1M-token retrieval accuracy from 1.7% to 70.3% with negligible overhead, showing fixed recurrent states need not cap long-context capability.
Editor's noteFirst to use 'sparse episodic memory + native feature addressing + replay reading' as a training-free plug-and-play interface, directly challenging the presupposition that 'fixed-state capacity of linear attention determines long-context capability.' Relative to the sparse anchor replay in 'Just Let Linear States Forget the Distant Past' from Issue 8, its increment is using the backbone's own key/query features for writing and addressing (without external retrievers or additional training), improving GDN's RULER retrieval from 1.7% to 70.3% at 1M tokens, with computational overhead <0.025%, and providing intervention analysis of Mamba-2 state reset.
FreeSolo achieves zero-shot 64x length extrapolation by keeping RoPE local while routing long-range attention through prunable cell summaries, reaching 256k context from 4k training.
Editor's noteFirst to design the long-range branch of sparse attention as a dual-branch structure where 'summaries only locate, evidence remains in original tokens': local RoPE window + long-range NoPE sparse reads, using maximum key norm as a query-agnostic pruning priority, achieving zero-shot length extrapolation through short CPT. Relative to SPIN's indexer temporal correlation prediction from Issue 9, its increment is proposing a training-free indexer family with 'norm pruning + mean key routing + hierarchical tree search,' providing complete degradation curves for 4K training and 256K extrapolation.
GATESIGHT predicts future attention KV cache utility from Gated DeltaNet write and decay signals, beating all query-agnostic baselines at 10% retention with negligible overhead.
Editor's noteFirst to use the write/forget signals (write gate, decay, write magnitude) of GDN recurrent memory in hybrid models as query-agnostic KV eviction signals, predicting future KV utility via a lightweight MLP. Relative to DeltaS from Issue 7, which uses GDN state changes to guide eviction, its increment is systematically using the write/forget signals themselves as cross-memory prediction signals, providing three layers of observation and a trainable estimator, with SCBench/RULER degradation curves and compute-matched comparisons.
ABRIDGEKV trains reasoning models to compress their own chain-of-thought into KV cache entries of learned pause tokens, achieving 4x cache reduction with no accuracy loss and 3.2x faster completion.
Editor's noteFirst to advance trainable compression from fixed inputs to online compression of the model's own long CoT outputs: replacing the original KV of generated blocks with the KV rows of learned pause tokens, with code rows located inside the replaced blocks, and allocating budget by layer-measured value. Relative to iS-KV's online low-rank compression from Issue 9, its increment is shifting the compression target from KV states to semantic summaries of the model's self-generated content, using end-to-end training to let subsequent tokens supervise what information to retain.
JSSA's training-free joint query-key subspace sparse attention preserves bilinear score structure, achieving up to 6.43x decoding throughput at 128K context while matching full-KV accuracy.
Editor's noteFirst to advance the low-dimensional scoring objective of token-level sparse attention from query/key reconstruction to joint query-key bilinear score preservation: providing a closed-form spectral solution via a solvable upper bound and proving that directly optimizing the exact objective yields no gain. Relative to SPIN's indexer scoring cost optimization from Issue 9, its increment is providing a closed-form spectral solution for the scoring subspace rather than training or heuristics, and accompanying it with the first gather-free indexed-attention kernel supporting per-KV-head independent selection under GQA.
Latent Compaction treats the KV cache as a trainable network via test-time training, achieving 96.7% accuracy at 32x compression on RULER versus 40% for KVZip+.
Editor's noteFirst to treat KV cache compression itself as a trainable neural network (document-specific KV slots), using test-time training to optimize the compressed cache for each document under frozen model weights, with two-stage training (attention fitting + end-to-end distillation) and three initialization types. Relative to Still's amortized synthesis from Issue 4 and RestoreKV's LoRA recovery from Issue 2, its increment is making the compressed cache parameters themselves the object of per-document gradient descent.
