Summary (Overview)

• MemT-1.0 is a family of memory foundation models (MemT-1.0-Nano with 36B parameters/3B activated/10M dynamic memory, and MemT-1.0-Flash with 288B parameters/17B activated/336M dynamic memory) that internalize memory as part of model computation rather than relying on external context.
• The architecture builds on Metis with two key upgrades: a stable token-level memory update mechanism based on Gated DeltaNet-2 (GDN-2), and Mixture-of-Storage (MoS) for scaling native memory capacity.
• A two-stage training pipeline (continued pre-training + post-training) supported by 34B tokens of training data enables the models to achieve memory capabilities; the models outperform parametric memory baselines by 8.9–12.5 points on average across 11 memory benchmarks.
• At 128K token history, MemT-1.0-Nano achieves 1.86× end-to-end inference speedup and a 66× reduction in persistent storage compared to full-context KV cache approaches.
• Applied to agentic coding tasks, MemT-1.0 demonstrates that internalizing procedural experience into native memory improves task success (up to +8.8 percentage points on pass^3 over no-experience baselines).

Introduction and Theoretical Foundation

The paper argues that memory-native evolution requires making memory an intrinsic capability of foundation models, enabling them to continuously accumulate and utilize information. The authors build on their previous work Metis, which first introduced memory foundation models to internalize memory as a native capability.

Key theoretical foundations include:

• The observation that capabilities like reasoning and multimodality have been progressively internalized from external systems into models themselves; memory represents another fundamental capability.
• The distinction between parametric memory (information encoded in model parameters) and textual memory (information in external context), with MemT-1.0 positioned as a parametric memory approach that does not require original context during inference.
• The insight that a fixed-size memory state enables successive compression of accumulated history, potentially supporting latent reflection and online self-evolution.

The autoregressive generation in the multi-step scenario is formulated as:

$$y_{t,k} \sim P(\cdot | X_t, Y_{t, < k}; \theta_t),$$

with the stateful model parameter updated by:

$$\theta_{t+1} \sim P(\cdot | X_t, Y_t; \theta_t),$$

Methodology

Model Architecture

The architecture augments selected Transformer blocks with native memory blocks that perform parallel original and memory attention:

$$\mathbf{A}_{t,k}^{(l)} = \mathbf{A}_{\text{ori},t,k}^{(l)} + \mathbf{A}_{\text{mem},t,k}^{(l)}, \quad \mathbf{A}_{\text{mem},t,k}^{(l)} = \text{MemoryAttention}(\tilde{\mathbf{H}}_{t,k}^{(l)}; \cdot)$$

Native Memory State comprises:

• Local Memory Block: Dynamic memory state $\mathbf{M}_t^{(l)} \in \mathbb{R}^{H_{kv} \times d_k \times d_v}$ with multi-channel structure
• Hyper Memory Block: Trainable parameters including erase gate $\mathbf{W}_\beta^{(l)}$, write gate $\mathbf{W}_\omega^{(l)}$, forget gate components, and memory output gating

Native Memory Storage uses a stable token-level update based on GDN-2:

$$\tilde{\mathbf{M}}_{t,i,h}^{(l)} = \left[\mathbf{I} - \mathbf{U}_{t,i,h}^{(l)\top} \mathbf{U}_{t,i,h}^{(l)}\right] \cdot \mathbf{D}_{t,i,h}^{(l)} \tilde{\mathbf{M}}_{t,i-1,h}^{(l)} + \mathbf{U}_{t,i,h}^{(l)\top} \left(\boldsymbol{\omega}_{t,i,h}^{(l)} \odot \mathbf{V}_{t,i,h}^{(l)}\right)$$

where $\mathbf{U}_{t,i}^{(l)} = \sqrt{\beta_{t,i}^{(l)}} \odot \mathbf{K}_{t,i}^{(l)}$ (with L2-normalized keys) and $\mathbf{D}_{t,i,h}^{(l)} = \mathrm{Diag}[\exp(\boldsymbol{\lambda}_{t,i,h}^{(l)})]$.

Mixture-of-Storage (MoS) expands each native memory block into E parallel memory experts with dense routing. The final memory attention is computed as:

$$\mathbf{A}_{\text{mem},t,i}^{(l)} = \sum_{e=1}^{E} \boldsymbol{\alpha}_{\text{u},t,i,e}^{(l)} \mathbf{A}_{\text{mem},t,i,e}^{(l)}$$

Training Pipeline

1. Backbone Preparation: For MemT-1.0-Flash, DeepSeek-V4-Flash undergoes post-training through a three-step pipeline (capability profiling, attribution by intervention, capability-conditioned synthesis) across three tracks (general, agentic coding, agentic working).

2. Continued Pre-Training (CPT): Establishes language modeling with native memory; sequences divided into segments with causal language modeling within each segment.

3. Post-Training: Adapts the model to memory-based response generation using off-policy distillation from the frozen backbone as teacher. The distillation loss is:

$$\ell_{t,k}^{\mathrm{KL}} = T_d^2 \sum_{v \in \mathcal{K}_{t,k}} \hat{q}_{t,k}(v) \log \frac{\hat{q}_{t,k}(v)}{p_{t,k}^{T_d}(v)}$$

System Infrastructure

• Chunked WY updates for efficient token-level state updates
• Complete-step windows for MoS to bound memory usage (reduces peak memory by ~80% with only 1.6% time increase at 4096-token windows)
• Stateful inference with shared weights and trajectory-specific memory states

Empirical Validation / Results

Memory Capabilities

Across 11 memory benchmarks grouped by trajectory length, MemT-1.0 achieves:

MemT-1.0-Flash and MemT-1.0-Nano outperform the strongest parametric baseline by 12.5 and 8.9 points respectively.

General Capabilities

Under the noise setting (with irrelevant information injected into memory):

• MemT-1.0-Flash: 0.0 average performance difference vs. backbone (gains on BigCodeBench-Hard +2.0, IFEval +0.9, LongBench v2 +0.2; decreases on MMLU-Pro -1.8, GPQA -0.6, LiveCodeBench -0.7)
• MemT-1.0-Nano: -0.9 average performance difference vs. backbone (gains on IFEval +3.3, BigCodeBench-Hard +2.0; decreases on LiveCodeBench -5.8, LongBench -2.0)

Ablation Studies

The token-level update granularity has the largest architectural effect, while post-training contributes the largest training gain (+10.7 points).

Inference Efficiency and Storage

At 128K tokens history length, MemT-1.0-Nano achieves:

• 1.86× end-to-end inference speedup over full context (3.99× at 256K)
• 2.44× query TTFT speedup at 128K (4.19× at 256K)
• 66× reduction in persistent storage vs. full context

Agentic Coding Results

Theoretical and Practical Implications

• Memory as a scaling dimension: The paper demonstrates that memory capacity (via MoS experts) and memory-related training data exhibit scaling behavior, with the 4-expert configuration outperforming 2-expert by up to 2.69 average points and 3.25 points on LoCoMo.

• Fixed-size memory states enable predictable resource usage: The constant state size makes per-trajectory budgets predictable and bounds the logical payload for resuming interactions, while enabling successive compression of accumulated history.

• Native memory can internalize procedural experience: Beyond information recall, MemT-1.0 demonstrates that procedural guidance for multi-step agentic execution can be stored in and retrieved from native memory without textual context, with performance matching or exceeding explicit textual experience.

• Preparation vs. utilization tradeoff: The paper highlights that memory construction costs matter: Temp-LoRA's end-to-end cost grows due to optimization-based adaptation, while 𝛿-mem's history preparation becomes prohibitive at longer lengths, making MemT-1.0's write-once-read-many design attractive.

• Memory fidelity remains a challenge: The gap between parametric memory (MemT-1.0) and textual memory systems (MemOS, Full Context) grows on long-trajectory benchmarks, highlighting that preserving detailed evidence through recurrent updates remains difficult.

Conclusion

The paper presents MemT-1.0, a family of memory foundation models that internalize memory as part of model computation through:

1. Stable token-level memory updates based on GDN-2
2. Mixture-of-Storage for scalable memory capacity
3. Deliberate backbone preparation and a two-stage training pipeline with 34B tokens
4. Efficient distributed infrastructure with stateful inference

The models outperform parametric memory baselines across 11 memory benchmarks while largely preserving general capabilities under noise. Applications to coding agents demonstrate that native memory can internalize procedural experience to improve task success and consistency.

Future directions include:

• Improving long-term memory fidelity and selective retention
• Scaling memory capacity further
• Reducing the cost of memory updates
• Autonomous experience acquisition and continual memory consolidation
• Exploring the relationship between local memory (information accumulation) and hyper memory (learned capabilities), potentially enabling consolidation of experience into memory parameters across tasks

The authors position this work as "a practical step toward memory-native evolution" where agents learn through accumulated experience stored in native memory rather than growing external context.