# MemT-1.0: Towards Memory-Native Evolution

> MemT-1.0 internalizes memory as native model computation via Gated DeltaNet-2 and Mixture-of-Storage, outperforming parametric baselines by 8.9-12.5 points across 11 memory benchmarks.

- **Source:** [arXiv](https://arxiv.org/abs/2610.26825)
- **Published:** 2026-10-21
- **Permalink:** https://picx.dev/p/1TUS03
- **Whiteboard:** https://picx.dev/p/1TUS03/image

## Summary

## Summary (Overview)

- **MemT-1.0** is a family of memory foundation models (MemT-1.0-Nano with 36B parameters/3B activated/10M dynamic memory, and MemT-1.0-Flash with 288B parameters/17B activated/336M dynamic memory) that internalize memory as part of model computation rather than relying on external context.
- The architecture builds on Metis with two key upgrades: a **stable token-level memory update mechanism** based on Gated DeltaNet-2 (GDN-2), and **Mixture-of-Storage (MoS)** for scaling native memory capacity.
- A **two-stage training pipeline** (continued pre-training + post-training) supported by 34B tokens of training data enables the models to achieve memory capabilities; the models outperform parametric memory baselines by 8.9–12.5 points on average across 11 memory benchmarks.
- At 128K token history, MemT-1.0-Nano achieves **1.86× end-to-end inference speedup** and a **66× reduction in persistent storage** compared to full-context KV cache approaches.
- Applied to agentic coding tasks, MemT-1.0 demonstrates that internalizing procedural experience into native memory improves task success (up to +8.8 percentage points on pass^3 over no-experience baselines).

## Introduction and Theoretical Foundation

The paper argues that **memory-native evolution** requires making memory an intrinsic capability of foundation models, enabling them to continuously accumulate and utilize information. The authors build on their previous work **Metis**, which first introduced memory foundation models to internalize memory as a native capability.

Key theoretical foundations include:

- The observation that capabilities like reasoning and multimodality have been progressively internalized from external systems into models themselves; memory represents another fundamental capability.
- The distinction between **parametric memory** (information encoded in model parameters) and **textual memory** (information in external context), with MemT-1.0 positioned as a parametric memory approach that does not require original context during inference.
- The insight that a fixed-size memory state enables **successive compression of accumulated history**, potentially supporting latent reflection and online self-evolution.

The autoregressive generation in the multi-step scenario is formulated as:

$$
y_{t,k} \sim P(\cdot | X_t, Y_{t, < k}; \theta_t),
$$

with the stateful model parameter updated by:

$$
\theta_{t+1} \sim P(\cdot | X_t, Y_t; \theta_t),
$$

## Methodology

### Model Architecture

The architecture augments selected Transformer blocks with **native memory blocks** that perform parallel original and memory attention:

$$\mathbf{A}_{t,k}^{(l)} = \mathbf{A}_{\text{ori},t,k}^{(l)} + \mathbf{A}_{\text{mem},t,k}^{(l)}, \quad \mathbf{A}_{\text{mem},t,k}^{(l)} = \text{MemoryAttention}(\tilde{\mathbf{H}}_{t,k}^{(l)}; \cdot)$$

**Native Memory State** comprises:
- **Local Memory Block**: Dynamic memory state $\mathbf{M}_t^{(l)} \in \mathbb{R}^{H_{kv} \times d_k \times d_v}$ with multi-channel structure
- **Hyper Memory Block**: Trainable parameters including erase gate $\mathbf{W}_\beta^{(l)}$, write gate $\mathbf{W}_\omega^{(l)}$, forget gate components, and memory output gating

**Native Memory Storage** uses a stable token-level update based on GDN-2:

$$\tilde{\mathbf{M}}_{t,i,h}^{(l)} = \left[\mathbf{I} - \mathbf{U}_{t,i,h}^{(l)\top} \mathbf{U}_{t,i,h}^{(l)}\right] \cdot \mathbf{D}_{t,i,h}^{(l)} \tilde{\mathbf{M}}_{t,i-1,h}^{(l)} + \mathbf{U}_{t,i,h}^{(l)\top} \left(\boldsymbol{\omega}_{t,i,h}^{(l)} \odot \mathbf{V}_{t,i,h}^{(l)}\right)$$

where $\mathbf{U}_{t,i}^{(l)} = \sqrt{\beta_{t,i}^{(l)}} \odot \mathbf{K}_{t,i}^{(l)}$ (with L2-normalized keys) and $\mathbf{D}_{t,i,h}^{(l)} = \mathrm{Diag}[\exp(\boldsymbol{\lambda}_{t,i,h}^{(l)})]$.

**Mixture-of-Storage (MoS)** expands each native memory block into E parallel memory experts with dense routing. The final memory attention is computed as:

$$\mathbf{A}_{\text{mem},t,i}^{(l)} = \sum_{e=1}^{E} \boldsymbol{\alpha}_{\text{u},t,i,e}^{(l)} \mathbf{A}_{\text{mem},t,i,e}^{(l)}$$

### Training Pipeline

1. **Backbone Preparation**: For MemT-1.0-Flash, DeepSeek-V4-Flash undergoes post-training through a three-step pipeline (capability profiling, attribution by intervention, capability-conditioned synthesis) across three tracks (general, agentic coding, agentic working).

2. **Continued Pre-Training (CPT)**: Establishes language modeling with native memory; sequences divided into segments with causal language modeling within each segment.

3. **Post-Training**: Adapts the model to memory-based response generation using **off-policy distillation** from the frozen backbone as teacher. The distillation loss is:

$$\ell_{t,k}^{\mathrm{KL}} = T_d^2 \sum_{v \in \mathcal{K}_{t,k}} \hat{q}_{t,k}(v) \log \frac{\hat{q}_{t,k}(v)}{p_{t,k}^{T_d}(v)}$$

### System Infrastructure

- **Chunked WY updates** for efficient token-level state updates
- **Complete-step windows for MoS** to bound memory usage (reduces peak memory by ~80% with only 1.6% time increase at 4096-token windows)
- **Stateful inference** with shared weights and trajectory-specific memory states

## Empirical Validation / Results

### Memory Capabilities

Across 11 memory benchmarks grouped by trajectory length, MemT-1.0 achieves:

| Method | Backbone | <16K | 16-32K | 32-64K | >64K | All AVG |
|---|---|---|---|---|---|---|
| **MemT-1.0-Flash** | - | 73.5/49.2 | 61.6/97.5/65.1 | 33.5/20.5/64.1 | 37.2/17.4/37.0 | **50.6** |
| **MemT-1.0-Nano** | - | 71.8/43.1 | 52.3/95.7/57.5 | 38.7/23.9/49.9 | 32.6/15.5/35.8 | **47.0** |
| Temp-LoRA | - | 44.3/32.0 | 57.4/98.0/25.1 | 35.7/16.2/52.5 | 13.4/12.0/32.3 | 38.1 |
| Doc-to-LoRA | - | 57.2/42.4 | 34.0/70.2/24.5 | 16.5/15.3/36.0 | 14.6/3.54/26.0 | 30.9 |
| MemoryLLM | - | 22.7/17.8 | 41.1/37.4/46.6 | 3.33/16.3/46.8 | 15.6/6.26/24.0 | 25.3 |
| ICAE | - | 31.9/24.5 | 17.8/70.4/22.4 | 2.80/11.4/40.4 | 11.0/7.35/11.7 | 22.9 |

MemT-1.0-Flash and MemT-1.0-Nano outperform the strongest parametric baseline by **12.5 and 8.9 points** respectively.

### General Capabilities

Under the noise setting (with irrelevant information injected into memory):

- **MemT-1.0-Flash**: 0.0 average performance difference vs. backbone (gains on BigCodeBench-Hard +2.0, IFEval +0.9, LongBench v2 +0.2; decreases on MMLU-Pro -1.8, GPQA -0.6, LiveCodeBench -0.7)
- **MemT-1.0-Nano**: -0.9 average performance difference vs. backbone (gains on IFEval +3.3, BigCodeBench-Hard +2.0; decreases on LiveCodeBench -5.8, LongBench -2.0)

### Ablation Studies

| Variant | AVG | ΔAVG |
|---|---|---|
| MemT-1.0-Nano | 47.0 | 0.0 |
| w/ GDN-2 (no stable mechanism) | 46.4 | -0.5 |
| w/ GDN | 46.7 | -0.3 |
| w/ Bidirectional Encoding | 45.4 | -1.5 |
| **w/ Turn-Level updates** | 41.0 | **-6.0** |
| w/o CPT | 44.9 | -2.1 |
| w/o PT | 36.3 | -10.7 |

The **token-level update granularity** has the largest architectural effect, while **post-training** contributes the largest training gain (+10.7 points).

### Inference Efficiency and Storage

At 128K tokens history length, MemT-1.0-Nano achieves:
- **1.86× end-to-end inference speedup** over full context (3.99× at 256K)
- **2.44× query TTFT speedup** at 128K (4.19× at 256K)
- **66× reduction in persistent storage** vs. full context

### Agentic Coding Results

| Benchmark | Store Timing | Method | pass^3 | Avg. Success | pass@3 |
|---|---|---|---|---|---|
| Verified | Harness-Triggered | Qwen3.6-35B-A3B (No Exp.) | 54.00 | 66.07 | 77.60 |
| Verified | Harness-Triggered | Qwen3.6-35B-A3B (Textual Exp.) | 61.80 | 69.13 | 76.80 |
| Verified | Harness-Triggered | **MemT-1.0-Nano (Native Mem.)** | **62.80** | **69.73** | **77.80** |
| Multilingual | Harness-Triggered | Qwen3.6-35B-A3B (No Exp.) | 45.33 | 60.00 | 73.67 |
| Multilingual | Harness-Triggered | **MemT-1.0-Nano (Native Mem.)** | **55.33** | **63.56** | **73.00** |

## Theoretical and Practical Implications

- **Memory as a scaling dimension**: The paper demonstrates that memory capacity (via MoS experts) and memory-related training data exhibit scaling behavior, with the 4-expert configuration outperforming 2-expert by up to 2.69 average points and 3.25 points on LoCoMo.

- **Fixed-size memory states enable predictable resource usage**: The constant state size makes per-trajectory budgets predictable and bounds the logical payload for resuming interactions, while enabling successive compression of accumulated history.

- **Native memory can internalize procedural experience**: Beyond information recall, MemT-1.0 demonstrates that procedural guidance for multi-step agentic execution can be stored in and retrieved from native memory without textual context, with performance matching or exceeding explicit textual experience.

- **Preparation vs. utilization tradeoff**: The paper highlights that memory construction costs matter: Temp-LoRA's end-to-end cost grows due to optimization-based adaptation, while 𝛿-mem's history preparation becomes prohibitive at longer lengths, making MemT-1.0's write-once-read-many design attractive.

- **Memory fidelity remains a challenge**: The gap between parametric memory (MemT-1.0) and textual memory systems (MemOS, Full Context) grows on long-trajectory benchmarks, highlighting that preserving detailed evidence through recurrent updates remains difficult.

## Conclusion

The paper presents **MemT-1.0**, a family of memory foundation models that internalize memory as part of model computation through:
1. Stable token-level memory updates based on GDN-2
2. Mixture-of-Storage for scalable memory capacity
3. Deliberate backbone preparation and a two-stage training pipeline with 34B tokens
4. Efficient distributed infrastructure with stateful inference

The models outperform parametric memory baselines across 11 memory benchmarks while largely preserving general capabilities under noise. Applications to coding agents demonstrate that native memory can internalize procedural experience to improve task success and consistency.

**Future directions** include:
- Improving long-term memory fidelity and selective retention
- Scaling memory capacity further
- Reducing the cost of memory updates
- Autonomous experience acquisition and continual memory consolidation
- Exploring the relationship between local memory (information accumulation) and hyper memory (learned capabilities), potentially enabling consolidation of experience into memory parameters across tasks

The authors position this work as "a practical step toward memory-native evolution" where agents learn through accumulated experience stored in native memory rather than growing external context.

---

_Markdown view of https://picx.dev/p/1TUS03, served by PicX — AI-generated visual whiteboard summaries of research papers._