Summary (Overview)

• This paper introduces a formal separation between stream stability (whether a model can generate fluently over long streams) and long-term memory (whether old content can still influence outputs) in language models.
• The authors propose three horizons to evaluate long-running models: the stability horizon (how long stable prediction lasts), the access horizon (how far back content causally affects outputs), and the utility horizon (how far back task-relevant information remains usable).
• A key theoretical result (Proposition 1) proves that unbounded stability does not imply unbounded memory — a model can generate perfectly forever while having zero access to content beyond its recent window.
• The paper introduces THREEH, a budget-matched evaluation protocol with three tracks (stability, causal retention, delayed task utility) that must be reported under identical resource constraints.
• Experiments on 128K-token streams show that attention sinks (StreamingLLM) stabilize generation but do not extend semantic access beyond the active cache, while recurrent and retrieval mechanisms extend the semantic horizon at additional state or latency cost.

Introduction and Theoretical Foundation

Background and Motivation

The paper addresses a critical ambiguity in the language model streaming literature. As models are increasingly deployed in continuous settings (multi-day dialogues, unbounded event streams, long-running agents), several capabilities are often conflated:

• Streaming stability: generating fluent text indefinitely
• Long-context processing: attending to very long prompts
• Long-term memory: retaining and using old information

The authors identify that StreamingLLM (Xiao et al., 2024b) — which retains a few initial KV states as "attention sinks" alongside a recent window — solves the stability problem but is frequently over-interpreted as providing memory. The retained sink states stabilize attention normalization but do not absorb the semantics of evicted tokens.

Formal Framework

The paper defines a bounded-state streaming system with state update:

$$M_t = U(M_{t-1}, x_t), \qquad |M_t| \leq B$$

where $M_t$ is the model state after token $t$, and $B$ is the persistent state budget.

Three horizons are formally defined:

1. Stability horizon (for degradation tolerance $\epsilon$):

$$H_{\text{stab}}(\epsilon) = \sup\{T: \Delta\mathcal{L}(t) \leq \epsilon \text{ for all } t \leq T\}$$

where $\Delta\mathcal{L}(t)$ is excess predictive loss relative to a reference.

2. Access horizon: Let $x_i \rightsquigarrow y_t$ mean changing $x_i$ can change the distribution of output $y_t$. Then:

$$H_{\text{access}}(t) = \max\{t - i: x_i \rightsquigarrow y_t\}$$

3. Utility horizon (for task family $\mathcal{T}$, score threshold $\alpha$):

$$H_{\text{util}}(\mathcal{T}, \alpha) = \sup\{d: S_{\mathcal{T}}(d) \geq \alpha\}$$

where $S_T(d)$ is expected task performance when decisive information is placed $d$ tokens before the query.

Key Theoretical Result

Proposition 1 (Horizon separation): There exists a bounded-state autoregressive model with $H_{\text{stab}}(\epsilon) = \infty$ for any $\epsilon > 0$, while $H_{\text{access}}(t) \leq W$ for all non-initial tokens and $H_{\text{util}}(\mathcal{T}, \alpha) \leq W$ for a delayed-recall task.

The proof constructs a stationary order-$W$ token process where the model exactly represents next-token conditionals using only the latest $W$ tokens. A delayed-recall task whose answer depends on a symbol at position $t - W - 1$ cannot be solved, since the retained state contains no function of that symbol.

Methodology

System Architectures Compared

Five conceptual systems are evaluated on a pretrained causal decoder with $W = 2{,}048$-token active cache:

1. Window: retain only the latest $W$ KV states
2. StreamingLLM: retain four initial sinks and $W - 4$ recent states
3. Window recomputation: rebuild a fresh $W$-token cache before each prediction
4. Recurrent: update a fixed-size learned memory at window boundaries
5. Sink + retrieval: use attention sinks locally and retrieve old records from an external store

Evaluation Tracks

Track A (Stability): Stream 128K tokens, normalize perplexity to each method's value at 2K tokens.

Track B (Retention): Insert four-way random bindings at lags from $0.25W$ to $8W$, test recall with counterfactual controls (only the target value changes across matched streams).

Track C (Utility): Delay evidence for a three-action decision task over the same lags.

THREEH Protocol Requirements

The evaluation contract requires reporting:

$$(B_{\text{device}}, B_{\text{external}}, F_{\text{token}}, L_{\text{token}}, L_{\text{query}})$$

where:

• $B_{\text{device}}$: device-resident state
• $B_{\text{external}}$: external stored state
• $F_{\text{token}}$: amortized compute per stream token
• $L_{\text{token}}$: generation latency
• $L_{\text{query}}$: query-time retrieval latency

Empirical Validation / Results

Main Results Table

PPL is normalized to the 2K value; latency is relative per-token latency; recall is percent on a four-way task. M is recurrent state and D is an external store.

Key Findings

1. Stability separation: At 128K tokens, plain window attention reaches 5.40× its short-stream perplexity, while StreamingLLM stays at 1.04× (close to recomputation's 1.02×) — but at 1.03× relative latency versus recomputation's 7.8×.

2. Retention collapse: Inside the cache, Window and StreamingLLM reach 97–98% recall, but at $2W$ both fall to ~26% (near chance for four-way tasks). The recurrent system retains 70% at $2W$ and 46% at $8W$; retrieval reaches 89% and 78%.

3. The critical comparison: StreamingLLM's 1.04× perplexity paired with 25% remote recall demonstrates that stability ≠ memory.

Measured Horizons

Thresholds: stability ≤ 1.1× normalized perplexity, retention ≥ 75%, utility ≥ 75%

Ablation Results

Key ablation conclusions: sink count changes stability but not remote semantic access; cache size and retrieval depth move the access horizon.

Theoretical and Practical Implications

For System Families

1. Sink-window streaming: Effectively unbounded stability horizon with constant device memory, but general access horizon limited to the recent window plus fixed initial positions. Appropriate when relevant information is local or another subsystem handles older state.

2. Recurrent/compressed state: Creates causal paths from old content to future outputs (access horizon can exceed visible window), but suffers from lossy interference — bounded state must decide what to preserve before future queries are known.

3. Retrieval-augmented memory: Physical access horizon can be as long as the store's retention period, but utility depends on indexing, query formation, selection, and downstream use. Storage and retrieval costs must be included in the budget.

4. Hybrid systems: Practical long-running agents should use all three — attention sinks for stable local generation, recurrent state for compact working memory, and retrieval for sparse historical evidence.

Design Recommendations

• Reserve "infinite" for the measured property: "Infinite-duration generation with bounded KV cache" is precise; "infinite context" requires retention evidence beyond the active window.
• Don't train an anchor and call it memory: A dedicated sink token requires a write mechanism or external store to function as memory.
• Match training to horizon objective: Local next-token loss rewards stability without rewarding remote retention; memory requires losses whose targets depend on delayed variables under interference.
• Use failure decomposition: When old facts don't affect decisions, distinguish between eviction, compression loss, retrieval failure, and failure to use available evidence.

Conclusion

The paper establishes that attention sinks solve a real systems problem — keeping bounded-cache generation stable far beyond training windows — but are not semantic memory by themselves. The authors propose replacing vague "context length" claims with three reported quantities: stability horizon, access horizon, and utility horizon.

The THREEH protocol turns this distinction into an evaluation contract with matched budgets, lag sweeps, counterfactual controls, and task-level decisions. This vocabulary enables streaming, recurrent, retrieval, and hybrid systems to receive credit for what they actually achieve while making missing capabilities visible.

Limitations

• Exact horizon measurements depend on the chosen stream, task family, score threshold, and reference system
• Counterfactual streams can become unnatural if edited carelessly
• Perplexity is tokenization-dependent
• External-memory accounting involves policy choices about storage infrastructure
• The access-utility inequality is conceptually useful but not directly observable for arbitrary neural states
• The three horizons don't cover privacy, calibration, factuality, or robustness to malicious memory content