# Metis: Memory Foundation Model

> Metis introduces a memory foundation model that internalizes memory as dynamic parametric states, achieving strong no-context performance on memory tasks.

- **Source:** [arXiv](https://arxiv.org/abs/2607.26760)
- **Published:** 2026-08-01
- **Permalink:** https://picx.dev/p/Pwp8Fd
- **Whiteboard:** https://picx.dev/p/Pwp8Fd/image

## Summary

## Summary (Overview)

- **Memory Foundation Models**: Introduce a new paradigm that internalizes memory as a native capability of foundation models, moving beyond external modules like RAG. Memory is defined by a persistent, dynamically evolving parametric memory state and autonomous memory procedures executed through forward computation.
- **Metis Architecture**: Proposes the first prototype, Metis, which augments each Transformer block with a Metis block. This consists of a **local memory block** (dense memory network \(M_t^{(l)} \in \mathbb{R}^{d_k \times d_v}\) and query-key normalization vector \(S_t^{(l)} \in \mathbb{R}^{d_k}\)) and a **hyper memory block** (static parameters \(\tilde{W}_Q, \tilde{W}_K, \tilde{W}_V, \tilde{\mathbf{w}}_{\text{agg}}\)) that compress and update the memory state.
- **Native Memory Procedure**: Information storage uses top-\(\rho\) adaptive aggregation of hidden states, followed by a weighted update (linear or Gated Delta Network). Utilization employs a separate memory attention mechanism that reads from the dense memory state, integrated with the original causal attention via a weighted interpolation.
- **Training and Optimization**: Constructs large-scale memory-specific training data (primary: remember/forget/update/reflect; auxiliary: multi-fact and pollution scenarios) from 27 public benchmarks. Mid-training with three objectives: memory reconstruction (lossless upper bound), memory operation (instruction-driven), and regularization (interference/pollution robustness).
- **Experimental Results**: Metis achieves strong performance on memory operation tasks (MemOps, OOD benchmarks) and memory-based QA (LoCoMo, NextMem) under **no-context** settings, significantly outperforming baselines like Temp-LoRA and \(\delta\)-Mem. Ablations confirm contributions of each component, while capacity studies reveal limitations in long-term and multi-step scenarios.

## Introduction and Theoretical Foundation

### Background and Motivation
Recent AI agents rely heavily on external memory modules (e.g., RAG) to retain past information. However, external memory suffers from three fundamental limitations:

1. **Decoupled from backbone**: External memory has separate targets and processing stages, leading to suboptimal alignment between storage and inference.
2. **End-to-end optimization difficulty**: Gradients cannot propagate effectively through discrete memory operations (e.g., retrieval, reranking, concatenation).
3. **Increased latency**: External storage and retrieval add explicit operations, increasing online inference time.

### Formal Definition
The paper formally defines the **memory foundation model** under multi-step interactions. Let \(t \in \{1,2,\dots,T\}\) denote time steps with input \(X_t\) and output \(Y_t\).

- **Traditional foundation model**: \(y_{t,k} \sim P(y | X_t, Y_{t,<k}; \theta)\) where \(\theta\) is static.
- **External memory**: \(y_{t,k} \sim P(y | C_t, X_t, Y_{t,<k}; \theta)\) where \(C_t\) is retrieved from explicit storage.
- **Memory foundation model**: \(y_{t,k} \sim P(y | X_t, Y_{t,<k}; \theta_t)\) where \(\theta_t\) integrates information from previous steps into native parametric space (native memory state). Concurrently, \(\theta_{t+1}\) is autonomously transformed during forward computation (native memory procedure).

### Key Concepts
- **Native Memory State**: Represented as dynamic parameters \(M_t\) (subset of \(\theta_t\)) that are coupled with the backbone. Semantic space of \(M_t\) and static parameters \(\Phi = \theta_t \setminus M_t\) must be aligned during pre-training/mid-training.
- **Native Memory Procedure**: Storage and utilization are modeled within the continuous function space of the backbone, not as discrete rules. Storage predicts how current information will be used; utilization maps instruction and memory state to output.

### Comparison with Prior Work
- **Test-time training (TTT)**: Adapts within a single sequence; lacks instruction-driven memory operations and persistent multi-step memory.
- **Memory-Augmented Neural Networks (MANNs)**: Use external differentiable slots; memory is still external to backbone parameters.
- **Other methods (Memory 3, MemGen)**: Still rely on textual memories or explicit retrieval; Metis removes textual context entirely, using parametric states.

## Methodology

### Metis Architecture
The architecture is built on a causal language model (e.g., Qwen3.5) with \(N\) Transformer blocks. Each block is augmented with a **Metis block** containing:

- **Local Memory Block**: Maintains the dynamic memory state:
  - Dense memory network: \(M_t^{(l)} \in \mathbb{R}^{d_k \times d_v}\)
  - Query-key normalization vector: \(S_t^{(l)} \in \mathbb{R}^{d_k}\)
  - Initialized to zero: \(M_1^{(l)} = \mathbf{0}, S_1^{(l)} = \mathbf{0}\).

- **Hyper Memory Block**: Contains static parameters learned during mid-training:
  - Importance vector: \(\tilde{\mathbf{w}}_{\text{agg}}^{(l)} \in \mathbb{R}^d\)
  - Memory key/value projections: \(\tilde{W}_K^{(l)} \in \mathbb{R}^{d \times d_k}, \tilde{W}_V^{(l)} \in \mathbb{R}^{d \times d_v}\)
  - Memory query projection: \(\tilde{W}_Q^{(l)} \in \mathbb{R}^{d \times d_k}\)

#### Native Memory Storage Procedure (After step \(t\))
1. **Adaptive Aggregation**: Score input hidden states \(\tilde{H}_t^{(l)}\) using \(\tilde{\mathbf{w}}_{\text{agg}}^{(l)}\) and temperature \(\tau\):
   \[
   \mathbf{p}_t^{(l)} = \text{Softmax}\left(\frac{\tilde{H}_t^{(l)} \tilde{\mathbf{w}}_{\text{agg}}^{(l)}}{\tau}\right) \in \mathbb{R}^L
   \]
   Select top-\(\rho\) tokens to obtain \(\bar{H}_t^{(l)} \in \mathbb{R}^{L'_t \times d}\) with \(L'_t \ll L\).

2. **Projection**: Compute memory keys and values:
   \[
   \tilde{K}_t^{(l)} = \bar{H}_t^{(l)} \tilde{W}_K^{(l)}, \quad \tilde{V}_t^{(l)} = \bar{H}_t^{(l)} \tilde{W}_V^{(l)}
   \]

3. **Memory State Update** (linear version shown; GDN-based update used in practice):
   \[
   M_{t+1}^{(l)} = \lambda M_t^{(l)} + (1-\lambda) \frac{\tilde{K}_t^{(l)\top}}{\sqrt{d_k}} \tilde{V}_t^{(l)}, \quad
   S_{t+1}^{(l)} = \lambda S_t^{(l)} + (1-\lambda) \frac{\tilde{K}_t^{(l)\top} \mathbf{1}}{\sqrt{d_k}}
   \]

#### Native Memory Utilization Procedure (During step \(t\))
1. **Memory Attention**: Compute memory query states:
   \[
   \tilde{Q}_t^{(l)} = \tilde{H}_t^{(l)} \tilde{W}_Q^{(l)}
   \]
   Then:
   \[
   \tilde{A}_t^{(l)} = \text{diag}\left(\tilde{Q}_t^{(l)} S_t^{(l)}\right)^{-1} \tilde{Q}_t^{(l)} M_t^{(l)}
   \]

2. **Combined Attention**: Interpolate with original causal attention using weight \(\gamma \in [0,1]\):
   \[
   A_t^{(l)} = \gamma \cdot \text{Softmax}\left(\frac{Q_t^{(l)} K_t^{(l)\top}}{\sqrt{d_k}} + \text{Mask}(L)\right) V_t^{(l)} + (1-\gamma) \cdot \text{Norm}\left(\tilde{A}_t^{(l)}\right)
   \]

### Data Construction
Two types of training data:

#### Primary Data (Section 4.1)
- **Operations**: Remember, Forget, Update, Reflect.
- **Sources**: 27 public benchmarks (LoCoMo, LongMemEval, RULER, etc.).
- **Dimensions**: Operation type, salience (explicit vs. implicit), noise level (distractor turns).
- **Statistics**: 357,137 samples, ~406M tokens (see Table 2).

| Operation | Sources | Explicit | Implicit | Distract | All Samples | Tokens (M) |
|-----------|---------|----------|----------|----------|-------------|------------|
| Remember  | 11      | 14,682   | 13,671   | 28,502   | 56,855      | 362.0      |
| Forget    | 9       | 59,900   | 8,251    | 68,120   | 136,271     | 21.7       |
| Update    | 4       | 33,452   | 7,300    | 40,749   | 81,501      | 11.0       |
| Reflect   | 3       | 20,646   | 20,615   | 41,249   | 82,510      | 11.4       |
| **Total** | 27      | 128,680  | 49,837   | 178,620  | 357,137     | 406.1      |

#### Auxiliary Data (Section 4.2)
- **Multi-fact scenarios**: Multi-entity binding (76,153 samples), Selective forgetting (76,153).
- **Memory pollution**: Post-memory dialogue (357,137), Memory-irrelevant dialogue (100,000).
- **Total**: 609,443 samples.

### Optimization Objectives (Section 5)
Mid-training freezes backbone parameters and optimizes only memory parameters. All objectives use per-step negative log-likelihood:
\[
\ell(s,t) = -\frac{1}{|Y_t|} \sum_{k=1}^{|Y_t|} \log P(y_{t,k} | X_t, Y_{t,<k}; \theta_t)
\]

1. **Memory Reconstruction Objective (\(\mathcal{L}_{\text{rec}}\))**: Regenerate stored reference content (lossless upper bound).
2. **Memory Operation Objective (\(\mathcal{L}_{\text{op}}\))**: Supervise remember/forget/update/reflect.
3. **Regularization Objective (\(\mathcal{L}_{\text{reg}}\))**: Constrain multi-fact and pollution scenarios.

Task-weighted sampler with linear annealing: \(\pi_\tau(e) = \frac{w_\tau(e)}{\sum_{\tau'} w_{\tau'}(e)}\), where \(w_\tau(e) = w_\tau^s + (w_\tau^e - w_\tau^s) \cdot \min(\frac{e}{E-1}, 1)\).

## Empirical Validation / Results

### Experimental Setup
- **Backbones**: Qwen3.5 (4B, 9B, 27B).
- **Baselines**: Full context, partial context (RAG top-5), Temp-LoRA (TTT-based), \(\delta\)-Mem.
- **Evaluation**: LLM-as-a-judge (gpt-4.1-mini) with median over 3 trials.
- **Training**: 8×H100 GPUs, AdamW, LR 2e-4, BF16.

### Main Results

#### Memory Operation Tasks (Table 5)
*No-context setting*:

| Method                | MemOps (Gold) Avg. | Metis Test Set Avg. |
|-----------------------|--------------------|----------------------|
| Qwen3.5-4B (no ctx)   | 1.65               | 16.96                |
| Qwen3.5-9B (no ctx)   | 1.88               | 18.64                |
| Qwen3.5-27B (no ctx)  | 1.69               | 16.87                |
| Temp-LoRA-4B          | 8.85               | 19.34                |
| Temp-LoRA-9B          | 13.51              | 19.51                |
| Temp-LoRA-27B         | 9.70               | 23.86                |
| \(\delta\)-Mem        | 4.38               | 15.03                |
| **Metis-4B**         | **17.84**          | **56.72**            |
| **Metis-9B**         | **19.63**          | **57.92**            |
| **Metis-27B**        | **24.76**          | **73.77**            |

#### Memory-based QA Tasks (Table 6)
*No-context setting*:

| Method                | LoCoMo (Gold) Avg. | NextMem Avg. |
|-----------------------|--------------------|--------------|
| Qwen3.5-4B (no ctx)   | 0.18               | 11.86        |
| Qwen3.5-9B (no ctx)   | 0.07               | 15.93        |
| Qwen3.5-27B (no ctx)  | 0.07               | 17.75        |
| Temp-LoRA-4B          | 9.99               | 24.20        |
| Temp-LoRA-9B          | 11.72              | 28.12        |
| Temp-LoRA-27B         | 4.24               | 30.97        |
| \(\delta\)-Mem        | 10.79              | 20.42        |
| **Metis-4B**         | **16.31**          | **41.69**    |
| **Metis-9B**         | **16.81**          | **43.39**    |
| **Metis-27B**        | **26.74**          | **50.82**    |

### Ablation Studies (Table 7)
Ablations on Metis-4B show:
- Removing adaptive aggregation ("w/o SA") causes the largest drop (-60.98% overall).
- Removing auxiliary data ("w/o MS+MP") drops overall by -19.31%.
- Removing query-key normalization ("w/o QKN") drops -28.44%.
- Removing optimizable memory query ("w/o OQ") drops -12.23%.

### Out-of-Distribution (OOD) Tasks (Table 8)
Metis-27B achieves competitive or best results on ATM-Bench (18.56 Avg.) and MemDaily (59.04 Avg.), demonstrating generalization beyond training distribution.

### Memory Capacity Studies (Figure 3)
- **Step-level capacity**: Performance degrades rapidly as input length exceeds several hundred words.
- **Trajectory-level capacity**: Accumulation of updates weakens earlier facts; performance declines with more steps.

### General Capability (Table 9)
Initial stage: minimal degradation (e.g., -0.80 on MMLU-Pro). Active stage with irrelevant memory: larger drops (e.g., -22.18 on IFEval), indicating interference from stored noise.

### Low-Rank Decomposition (Figure 4, Table 10)
Memory states exhibit low effective rank. At rank 64 (6.25% of full 1024), recovery reaches 99.9% overall performance.

## Theoretical and Practical Implications

### Theoretical Contributions
- **Formal definition** of memory foundation models and native memory (state + procedure).
- **Theoretical insight** showing memory attention approximates processing a virtual memory prefix, with error analysis identifying three sources of noise (\(\epsilon_1, \epsilon_2, \epsilon_3\)).
- **Efficiency analysis** demonstrating parallelism between original attention, memory utilization, and storage, leading to layer-level latency:
  \[
  T_{\text{parallel}}^{(l)} = \max(T_{\text{orig}}, T_{\text{util}}, T_{\text{store}}) + T_{\text{fuse}}
  \]
  Unlike external memory, cost does not grow linearly with history length.

### Practical Implications
- **End-to-end optimization**: Memory procedures are trained via gradient-based objectives, enabling domain-specific adaptation.
- **Inference efficiency**: Gradient-free online memory update (only forward pass), frozen weights during inference.
- **Storage compression**: Low-rank decomposition reduces memory state storage without major performance loss (64/1024 rank

---

_Markdown view of https://picx.dev/p/Pwp8Fd, served by PicX — AI-generated visual whiteboard summaries of research papers._