# δ-mem: Efficient Online Memory for Large Language Models > δ-mem introduces a compact online associative memory state updated via delta-rule learning, which generates low-rank corrections to a frozen LLM's attention, enabling efficient dynamic memory without fine-tuning. - **Source:** [arXiv](https://arxiv.org/abs/2605.12357) - **Published:** 2026-05-14 - **Permalink:** https://picx.dev/p/uk5K4a - **Whiteboard:** https://picx.dev/p/uk5K4a/image ## Summary # δ-mem: Efficient Online Memory for Large Language Models ## Summary (Overview) - **Core Contribution**: δ-mem is a lightweight memory mechanism that augments a frozen full-attention LLM backbone with a **compact online state of associative memory (OSAM)**, enabling dynamic maintenance and use of historical information without full fine-tuning or architectural changes. - **Key Mechanism**: It compresses past information into a fixed-size state matrix (e.g., 8×8) updated via **delta-rule learning**, and uses its readout to generate **low-rank corrections** to the backbone's attention computation (query and output sides) during generation. - **Performance**: With only an 8×8 memory state, δ-mem improves the average score to **1.10×** that of the frozen backbone and **1.15×** that of the strongest non-δ-mem baseline. Gains are larger on memory-heavy benchmarks: **1.31×** on MemoryAgentBench and **1.20×** on LoCoMo. - **Efficiency**: Provides effective associative memory without extending explicit context, heavy external retrieval modules, or replacing the backbone architecture. - **Design Flexibility**: Explores three writing granularities: **Token-State Write (TSW)**, **Sequence-State Write (SSW)**, and **Multi-State Write (MSW)**. ## Introduction and Theoretical Foundation Large language models (LLMs) are increasingly deployed in memory-heavy scenarios like long-term personalized assistants and agent systems, requiring the accumulation, updating, and reuse of historical information over extended interactions. Simply expanding the context window is costly (quadratic attention cost) and ineffective due to context degradation/rot. Existing memory mechanisms fall into three paradigms with limitations: 1. **Textual Memory Mechanisms (TMMs)**: Store memory as text injected via input context. Suffer from context-window limits, retrieval noise, and compaction loss. 2. **Outside-channel Memory Mechanisms (OMMs)**: Keep memory in external modules interacting via separate pathways. Introduce overhead, integration complexity, and potential misalignment. 3. **Parametric Memory Mechanisms (PMMs)**: Encode memory into parameters (e.g., prefixes, adapters). Are efficient but static, limiting adaptation to dynamically evolving information. **δ-mem's Motivation**: A need for a mechanism that maintains a **compact, dynamically evolving memory state** while steering the backbone through a pathway **tightly aligned with its internal attention computation**. **Theoretical Foundation**: The online state update is formulated as optimizing an online regression loss using SGD. Given a memory key $k_t \in \mathbb{R}^r$ and value $v_t \in \mathbb{R}^r$ at position $t$, the prediction from the previous state $S_{t-1}$ is: $$\hat{v}_t = S_{t-1} k_t.$$ The update minimizes the loss: $$L_t(S) = \frac{1}{2} \| S k_t - v_t \|^2,$$ leading to the delta-rule update: $$S_t = S_{t-1} - \beta_t \nabla_{S_{t-1}} L_t(S_{t-1}) = S_{t-1} + \beta_t (v_t - S_{t-1} k_t) k_t^\top.$$ Inspired by gated retention, a forget gate $\lambda_t$ is introduced for stable long-range evolution: $$S_t = \lambda_t S_{t-1} + \beta_t (v_t - S_{t-1} k_t) k_t^\top. \tag{3}$$ Here $\lambda_t$ controls retention of previous memory, and $\beta_t$ controls the strength of the residual write. ## Methodology δ-mem operates in a sequence: **read** associative signals from the old state, **steer** attention with low-rank corrections, then **write** current information into the state. The backbone remains frozen. ### 1. Memory Projections The hidden state $x_t \in \mathbb{R}^d$ is projected into a low-dimensional associative memory space ($r \ll d$, e.g., $r=8$): $$ \begin{aligned} q^m_t &= \text{L2\_norm}\left(\tanh(W^m_q x_t)\right), \\ k^m_t &= \text{L2\_norm}\left(\tanh(W^m_k x_t)\right), \\ v^m_t &= W^m_v x_t, \end{aligned} \tag{4} $$ where $q^m_t, k^m_t, v^m_t \in \mathbb{R}^r$. Normalizing query and key reduces scale drift. Write ($\beta_t$) and retention ($\lambda_t$) gates are also derived from $x_t$: $$\beta_t = \sigma(W_\beta x_t + b), \quad \lambda_t = 1 - \beta_t. \tag{5}$$ ### 2. Reading from Online State Before writing, the model reads context-relevant associative signals by querying the old state: $$r_t = S_{t-1} q^m_t. \tag{6}$$ The read vector $r_t \in \mathbb{R}^r$ provides history-dependent steering signals, with cost independent of history length. ### 3. Steering Attention via Low-Rank Corrections The read signal $r_t$ is transformed into corrections for the backbone's attention: $$ \begin{aligned} \Delta q_t &= W^\Delta_q r_t, \\ \Delta o_t &= W^\Delta_o r_t. \tag{7} \end{aligned} $$ These are added to the original query and attention output: $$ \begin{aligned} q^0_t &= W_Q x_t, \quad \tilde{q}_t = q^0_t + \alpha_r \Delta q_t, \tag{8} \\ a_t &= \text{Attn}(\tilde{q}_t, K_{\leq t}, V_{\leq t}), \quad \tilde{y}_t = a_t + \alpha_r \Delta o_t. \tag{9} \end{aligned} $$ The corrections are low-rank and dynamic because $r_t$ comes from the evolving state $S_{t-1}$. ### 4. Writing into Online State After attention, the current information $(k^m_t, v^m_t)$ is written using the gated delta-rule (dimension-wise): $$S_t = \text{Diag}(\lambda_t) S_{t-1} + \text{Diag}(\beta_t) (v^m_t - S_{t-1} k^m_t) (k^m_t)^\top. \tag{10}$$ Expanded, this becomes: $$S_t = \text{Diag}(\lambda_t) S_{t-1} - \text{Diag}(\beta_t) S_{t-1} k^m_t (k^m_t)^\top + \text{Diag}(\beta_t) v^m_t (k^m_t)^\top. \tag{11}$$ Row-wise, for the $i$-th row $s^{(i)}_t$: $$s^{(i)}_t = \lambda_{t,i} s^{(i)}_{t-1} + \beta_{t,i} \left( v^m_{t,i} - s^{(i)}_{t-1} k^m_t \right) (k^m_t)^\top. \tag{12}$$ ### 5. Writing Granularity Strategies - **Token-State Write (TSW)**: Updates state at every token position. $S_t = \text{Update}(S_{t-1}, x_t)$. Captures fine-grained details but susceptible to noise. - **Sequence-State Write (SSW)**: Updates per message/segment. Averages hidden states within segment $M^{(j)}$: $\bar{x}^{(j)} = \frac{1}{|M^{(j)}|} \sum_{t \in M^{(j)}} x_t$, then $S^{(j)} = \text{Update}(S^{(j-1)}, \bar{x}^{(j)})$. Reduces redundancy, smooths evolution. - **Multi-State Write (MSW)**: Uses $N$ parallel sub-states: $S_t = \{S^{(1)}_t, ..., S^{(N)}_t\}$, each updated independently. Readouts are concatenated: $r_t = \text{Concat}(r^{(1)}_t, ..., r^{(N)}_t)$. Reduces interference by separating memory types. ### 6. Training Objective Trained with standard supervised fine-tuning (SFT) loss. Context tokens are written into the online state, producing $S_C$, but are **not replayed** as explicit backbone input during prediction. The backbone receives only query $Q$ and response $Y$, steered by the stored state: $$L_{\text{SFT}} = -\sum_{j=1}^{|Y|} \log p_{\phi,\theta}(y_j | Q, y_{