Summary (Overview)

• Memory Foundation Models: Introduce a new paradigm that internalizes memory as a native capability of foundation models, moving beyond external modules like RAG. Memory is defined by a persistent, dynamically evolving parametric memory state and autonomous memory procedures executed through forward computation.
• Metis Architecture: Proposes the first prototype, Metis, which augments each Transformer block with a Metis block. This consists of a local memory block (dense memory network (M_t^{(l)} \in \mathbb{R}^{d_k \times d_v}) and query-key normalization vector (S_t^{(l)} \in \mathbb{R}^{d_k})) and a hyper memory block (static parameters (\tilde{W}_Q, \tilde{W}_K, \tilde{W}V, \tilde{\mathbf{w}}{\text{agg}})) that compress and update the memory state.
• Native Memory Procedure: Information storage uses top-(\rho) adaptive aggregation of hidden states, followed by a weighted update (linear or Gated Delta Network). Utilization employs a separate memory attention mechanism that reads from the dense memory state, integrated with the original causal attention via a weighted interpolation.
• Training and Optimization: Constructs large-scale memory-specific training data (primary: remember/forget/update/reflect; auxiliary: multi-fact and pollution scenarios) from 27 public benchmarks. Mid-training with three objectives: memory reconstruction (lossless upper bound), memory operation (instruction-driven), and regularization (interference/pollution robustness).
• Experimental Results: Metis achieves strong performance on memory operation tasks (MemOps, OOD benchmarks) and memory-based QA (LoCoMo, NextMem) under no-context settings, significantly outperforming baselines like Temp-LoRA and (\delta)-Mem. Ablations confirm contributions of each component, while capacity studies reveal limitations in long-term and multi-step scenarios.

Introduction and Theoretical Foundation

Background and Motivation

Recent AI agents rely heavily on external memory modules (e.g., RAG) to retain past information. However, external memory suffers from three fundamental limitations:

1. Decoupled from backbone: External memory has separate targets and processing stages, leading to suboptimal alignment between storage and inference.
2. End-to-end optimization difficulty: Gradients cannot propagate effectively through discrete memory operations (e.g., retrieval, reranking, concatenation).
3. Increased latency: External storage and retrieval add explicit operations, increasing online inference time.

Formal Definition

The paper formally defines the memory foundation model under multi-step interactions. Let (t \in {1,2,\dots,T}) denote time steps with input (X_t) and output (Y_t).

• Traditional foundation model: (y_{t,k} \sim P(y | X_t, Y_{t,<k}; \theta)) where (\theta) is static.
• External memory: (y_{t,k} \sim P(y | C_t, X_t, Y_{t,<k}; \theta)) where (C_t) is retrieved from explicit storage.
• Memory foundation model: (y_{t,k} \sim P(y | X_t, Y_{t,<k}; \theta_t)) where (\theta_t) integrates information from previous steps into native parametric space (native memory state). Concurrently, (\theta_{t+1}) is autonomously transformed during forward computation (native memory procedure).

Key Concepts

• Native Memory State: Represented as dynamic parameters (M_t) (subset of (\theta_t)) that are coupled with the backbone. Semantic space of (M_t) and static parameters (\Phi = \theta_t \setminus M_t) must be aligned during pre-training/mid-training.
• Native Memory Procedure: Storage and utilization are modeled within the continuous function space of the backbone, not as discrete rules. Storage predicts how current information will be used; utilization maps instruction and memory state to output.

Comparison with Prior Work

• Test-time training (TTT): Adapts within a single sequence; lacks instruction-driven memory operations and persistent multi-step memory.
• Memory-Augmented Neural Networks (MANNs): Use external differentiable slots; memory is still external to backbone parameters.
• Other methods (Memory 3, MemGen): Still rely on textual memories or explicit retrieval; Metis removes textual context entirely, using parametric states.

Methodology

Metis Architecture

The architecture is built on a causal language model (e.g., Qwen3.5) with (N) Transformer blocks. Each block is augmented with a Metis block containing:

• Local Memory Block: Maintains the dynamic memory state:

  • Dense memory network: (M_t^{(l)} \in \mathbb{R}^{d_k \times d_v})
  • Query-key normalization vector: (S_t^{(l)} \in \mathbb{R}^{d_k})
  • Initialized to zero: (M_1^{(l)} = \mathbf{0}, S_1^{(l)} = \mathbf{0}).

• Hyper Memory Block: Contains static parameters learned during mid-training:

  • Importance vector: (\tilde{\mathbf{w}}_{\text{agg}}^{(l)} \in \mathbb{R}^d)
  • Memory key/value projections: (\tilde{W}_K^{(l)} \in \mathbb{R}^{d \times d_k}, \tilde{W}_V^{(l)} \in \mathbb{R}^{d \times d_v})
  • Memory query projection: (\tilde{W}_Q^{(l)} \in \mathbb{R}^{d \times d_k})

Native Memory Storage Procedure (After step (t))

1. Adaptive Aggregation: Score input hidden states (\tilde{H}t^{(l)}) using (\tilde{\mathbf{w}}{\text{agg}}^{(l)}) and temperature (\tau): [ \mathbf{p}_t^{(l)} = \text{Softmax}\left(\frac{\tilde{H}t^{(l)} \tilde{\mathbf{w}}{\text{agg}}^{(l)}}{\tau}\right) \in \mathbb{R}^L ] Select top-(\rho) tokens to obtain (\bar{H}_t^{(l)} \in \mathbb{R}^{L'_t \times d}) with (L'_t \ll L).

2. Projection: Compute memory keys and values: [ \tilde{K}_t^{(l)} = \bar{H}_t^{(l)} \tilde{W}_K^{(l)}, \quad \tilde{V}_t^{(l)} = \bar{H}_t^{(l)} \tilde{W}_V^{(l)} ]

3. Memory State Update (linear version shown; GDN-based update used in practice): [ M_{t+1}^{(l)} = \lambda M_t^{(l)} + (1-\lambda) \frac{\tilde{K}_t^{(l)\top}}{\sqrt{d_k}} \tilde{V}t^{(l)}, \quad S{t+1}^{(l)} = \lambda S_t^{(l)} + (1-\lambda) \frac{\tilde{K}_t^{(l)\top} \mathbf{1}}{\sqrt{d_k}} ]

Native Memory Utilization Procedure (During step (t))

1. Memory Attention: Compute memory query states: [ \tilde{Q}_t^{(l)} = \tilde{H}_t^{(l)} \tilde{W}_Q^{(l)} ] Then: [ \tilde{A}_t^{(l)} = \text{diag}\left(\tilde{Q}_t^{(l)} S_t^{(l)}\right)^{-1} \tilde{Q}_t^{(l)} M_t^{(l)} ]

2. Combined Attention: Interpolate with original causal attention using weight (\gamma \in [0,1]): [ A_t^{(l)} = \gamma \cdot \text{Softmax}\left(\frac{Q_t^{(l)} K_t^{(l)\top}}{\sqrt{d_k}} + \text{Mask}(L)\right) V_t^{(l)} + (1-\gamma) \cdot \text{Norm}\left(\tilde{A}_t^{(l)}\right) ]

Data Construction

Two types of training data:

Primary Data (Section 4.1)

• Operations: Remember, Forget, Update, Reflect.
• Sources: 27 public benchmarks (LoCoMo, LongMemEval, RULER, etc.).
• Dimensions: Operation type, salience (explicit vs. implicit), noise level (distractor turns).
• Statistics: 357,137 samples, ~406M tokens (see Table 2).

Auxiliary Data (Section 4.2)

• Multi-fact scenarios: Multi-entity binding (76,153 samples), Selective forgetting (76,153).
• Memory pollution: Post-memory dialogue (357,137), Memory-irrelevant dialogue (100,000).
• Total: 609,443 samples.

Optimization Objectives (Section 5)

Mid-training freezes backbone parameters and optimizes only memory parameters. All objectives use per-step negative log-likelihood: [ \ell(s,t) = -\frac{1}{|Y_t|} \sum_{k=1}^{|Y_t|} \log P(y_{t,k} | X_t, Y_{t,<k}; \theta_t) ]

1. Memory Reconstruction Objective ((\mathcal{L}_{\text{rec}})): Regenerate stored reference content (lossless upper bound).
2. Memory Operation Objective ((\mathcal{L}_{\text{op}})): Supervise remember/forget/update/reflect.
3. Regularization Objective ((\mathcal{L}_{\text{reg}})): Constrain multi-fact and pollution scenarios.

Task-weighted sampler with linear annealing: (\pi_\tau(e) = \frac{w_\tau(e)}{\sum_{\tau'} w_{\tau'}(e)}), where (w_\tau(e) = w_\tau^s + (w_\tau^e - w_\tau^s) \cdot \min(\frac{e}{E-1}, 1)).

Empirical Validation / Results

Experimental Setup

• Backbones: Qwen3.5 (4B, 9B, 27B).
• Baselines: Full context, partial context (RAG top-5), Temp-LoRA (TTT-based), (\delta)-Mem.
• Evaluation: LLM-as-a-judge (gpt-4.1-mini) with median over 3 trials.
• Training: 8×H100 GPUs, AdamW, LR 2e-4, BF16.

Main Results

Memory Operation Tasks (Table 5)

No-context setting:

Memory-based QA Tasks (Table 6)

No-context setting:

Ablation Studies (Table 7)

Ablations on Metis-4B show:

• Removing adaptive aggregation ("w/o SA") causes the largest drop (-60.98% overall).
• Removing auxiliary data ("w/o MS+MP") drops overall by -19.31%.
• Removing query-key normalization ("w/o QKN") drops -28.44%.
• Removing optimizable memory query ("w/o OQ") drops -12.23%.

Out-of-Distribution (OOD) Tasks (Table 8)

Metis-27B achieves competitive or best results on ATM-Bench (18.56 Avg.) and MemDaily (59.04 Avg.), demonstrating generalization beyond training distribution.

Memory Capacity Studies (Figure 3)

• Step-level capacity: Performance degrades rapidly as input length exceeds several hundred words.
• Trajectory-level capacity: Accumulation of updates weakens earlier facts; performance declines with more steps.

General Capability (Table 9)

Initial stage: minimal degradation (e.g., -0.80 on MMLU-Pro). Active stage with irrelevant memory: larger drops (e.g., -22.18 on IFEval), indicating interference from stored noise.

Low-Rank Decomposition (Figure 4, Table 10)

Memory states exhibit low effective rank. At rank 64 (6.25% of full 1024), recovery reaches 99.9% overall performance.

Theoretical and Practical Implications

Theoretical Contributions

• Formal definition of memory foundation models and native memory (state + procedure).
• Theoretical insight showing memory attention approximates processing a virtual memory prefix, with error analysis identifying three sources of noise ((\epsilon_1, \epsilon_2, \epsilon_3)).
• Efficiency analysis demonstrating parallelism between original attention, memory utilization, and storage, leading to layer-level latency: [ T_{\text{parallel}}^{(l)} = \max(T_{\text{orig}}, T_{\text{util}}, T_{\text{store}}) + T_{\text{fuse}} ] Unlike external memory, cost does not grow linearly with history length.

Practical Implications

• End-to-end optimization: Memory procedures are trained via gradient-based objectives, enabling domain-specific adaptation.
• Inference efficiency: Gradient-free online memory update (only forward pass), frozen weights during inference.
• Storage compression: Low-rank decomposition reduces memory state storage without major performance loss (64/1024 rank