# Agent Memory Distillation: Empowering Small LLM Agents with Hierarchical Teacher Memory

> Agent Memory Distillation transfers teacher knowledge via hierarchical workflow, subtask, and function memories, boosting small agent accuracy by up to 34.5 percentage points without training.

- **Source:** [arXiv](https://arxiv.org/abs/2608.07169)
- **Published:** 2026-08-12
- **Permalink:** https://picx.dev/p/A7f2AC
- **Whiteboard:** https://picx.dev/p/A7f2AC/image

## Summary

## Summary (Overview)

- **Agent Memory Distillation (AMD)** is a training-free framework that transfers structured knowledge from a large teacher agent to small student agents (4B–8B parameters) through **hierarchical memory** comprising three complementary types: Workflow, Subtask, and Function memory.
- AMD achieves **average accuracy gains of 27.2%p, 11.2%p, and 3.4%p** on AppWorld, BFCL V3, and ToolSandbox respectively, consistently outperforming existing memory-based baselines (ReasoningBank, MemP, SASM).
- Some small students **match or surpass teacher-level performance**: Gemma4-E4B (54.17%) and Qwen3-8B (51.79%) exceed GPT-5-mini's 50.00% on AppWorld.
- **Subtask memory contributes the largest gains** to performance, particularly for complex, long-horizon tasks; teacher effectiveness depends on both teacher capability and teacher–student compatibility.
- The **optimal memory representation varies by granularity**: high-level planning transfers best as natural language, while low-level execution knowledge is better conveyed through concrete code examples.

---

## Introduction and Theoretical Foundation

### Background and Motivation

Memory systems have shown promise for improving agent performance in tool-use tasks, enabling the reuse of successful behavioral patterns and avoidance of past failures. However, the potential of memory remains largely unexplored for **small language models**, which face two fundamental challenges:

1. **Low task success rates** produce memory repositories dominated by unsuccessful trajectories, with only limited successful experiences available for reuse.
2. **A capability gap** between teacher and student agents limits the effectiveness of naive memory transfer—a small student may receive a high-level strategy like *"Log in first to start the playlist"* but lack the prerequisite knowledge to execute the login procedure itself.

**Key Insight:** The authors identify that naive teacher memory transfer yields only marginal improvements comparable to student-generated memory alone, analogous to the knowledge gap observed in conventional knowledge distillation (Mirzadeh et al., 2020; Guo et al., 2020). Small models exhibit limited in-context learning and weaker instruction-following capacity, making it difficult to interpret and apply teacher memories directly.

### Theoretical Foundation

The work draws on two research threads:

- **LLM Agents with Memory**: Prior work (Reflexion, ExpeL, AWM, ReasoningBank, MemP, ReflecTool, SASM) demonstrates memory's value, but predominantly with large proprietary models (GPT-4, Gemini 2.5), overlooking the limitations of small agents.
- **Knowledge Distillation**: While parameter-update approaches (SAD, Agent Distillation, SCoRe) transfer teacher behaviors through training, AMD takes a **training-free approach** that directly transfers teacher experiences as structured memory.

---

## Methodology

### Problem Formulation

In multi-turn tool-use reasoning, an agent $\pi$ operates in an interactive environment by issuing tool calls from a predefined set $\mathcal{F}$. Given a task $s$, the agent produces a trajectory $\tau = ((a_1, o_1), \dots, (a_T, o_T))$ where $a_t \in \mathcal{F}$ is a tool call and $o_t$ is the observation. The goal of AMD is to maximize:

$$\max_{\mathcal{M}} \mathbb{E}_{s \sim \mathcal{S}} \left[ R \big(\pi^{S} (s; \mathcal{M}) \big) \right], \tag{1}$$

where $R(\pi^{S}(s; \mathcal{M}))$ is the task success reward of student $\pi^{S}$ on task $s$ given memory $\mathcal{M}$.

### Hierarchical Memory Generation

AMD constructs three memory types from successful teacher trajectories $\mathcal{D}_{+}^{T} \subseteq \mathcal{D}^{T}$:

#### Workflow Memory ($\mathcal{M}^{wf}$)
Captures the teacher's high-level task-completion strategy as a **verbalized insight** in natural language. Runtime values (identifiers, credentials, file paths) are replaced with typed placeholders (e.g., `<ID>`, `<EMAIL>`, `<FILE_PATH>`) for generalizability. Each entry $m_i^{wf} = (q_i, \bar{ins}_i)$ pairs a natural language query with the insight, encoded into dense vectors for retrieval.

#### Subtask Memory ($\mathcal{M}^{st}$)
Provides concrete behavioral examples at an intermediate granularity. Each successful trajectory is decomposed into coherent subtask segments $\{e_{i,1}, \ldots, e_{i,K_i}\}$ via teacher LLM segmentation. Each entry $m_{i,k}^{st} = (\ell_{i,k}, d_{i,k}, e_{i,k})$ contains a label, description, and concrete execution examples (tool calls or code with corresponding observations).

#### Function Memory ($\mathcal{M}^{fn}$)
Captures fine-grained tool invocation knowledge per function. Each record $m_{i,j}^{fn} = (f_{i,j}, E_{i,j}, \text{doc}(f_{i,j}))$ stores the function name, a concrete teacher example $E_{i,j}$ with surrounding context, and optional API documentation. Function entries are indexed by function name rather than dense vectors.

### Memory Retrieval and Injection

- **Proactive Injection**: Workflow memory is retrieved using the task instruction as query (top-k entries); Subtask memory is retrieved by having the student decompose the task into up to six subtask labels, each used as an independent retrieval query with deduplication. Both are injected into the system prompt at task start.
- **Reactive Injection**: Function memory is retrieved only when a tool call returns an error, using the failing function name for lookup, ranked by cosine similarity with the current task instruction.

---

## Methodology

**Benchmarks:**
- **AppWorld** (Trivedi et al., 2024): Multi-app agent benchmark with Python API calls, 168 tasks in `test_normal` subset, scored by database-state unit tests.
- **BFCL V3** (Patil et al., 2025): Function-calling benchmark, 200 multi-turn base tasks, correct only if every turn matches expected API state.
- **ToolSandbox** (Lu et al., 2024): Stateful conversational tool-use, 129 base scenarios with GPT-5-mini as user simulator.

**Teacher & Students:**
- Teacher: GPT-5-mini
- Students: Qwen3-4B, Qwen3-8B, Gemma4-E4B, Llama3.1-8B
- Memory encoding: OpenAI's text-embedding-3-small
- Retrieval: top-1 for all memory types (k=1), each experiment repeated twice with average reported

---

## Empirical Validation / Results

### Main Results

**Table 1: Main results across three benchmarks. Δ shows absolute accuracy gains of AMD over zero-shot.**

| Method | AppWorld | BFCL V3 | ToolSandbox | Average |
|---|---|---|---|---|
| **Teacher Agent** | | | | |
| GPT-5-mini | 50.00 | 36.50 | 28.68 | 38.39 |
| **Qwen3-4B** | | | | |
| Zero-shot | 14.88 | 15.50 | 16.28 | 15.55 |
| ReasoningBank | 10.71 | 24.25 | 16.28 | 17.08 |
| MemP | 16.67 | 28.25 | 14.73 | 19.88 |
| SASM | 15.48 | 15.25 | 18.22 | 16.32 |
| **AMD** | **49.40** | **38.50** | **20.16** | **36.02** |
| Δ | +34.52 | +23.00 | +3.88 | +20.47 |
| **Gemma4-E4B** | | | | |
| Zero-shot | 24.40 | 37.25 | 18.22 | 26.62 |
| **AMD** | **54.17** | **46.00** | **21.71** | **40.63** |
| Δ | +29.77 | +8.75 | +3.49 | +14.00 |
| **Qwen3-8B** | | | | |
| Zero-shot | 25.60 | 38.00 | 20.16 | 27.92 |
| **AMD** | **51.79** | **45.50** | **25.58** | **40.96** |
| Δ | +26.19 | +7.50 | +5.42 | +13.04 |
| **Llama3.1-8B** | | | | |
| Zero-shot | 8.93 | 9.00 | 5.43 | 7.79 |
| **AMD** | **27.38** | **14.50** | **6.20** | **16.03** |
| Δ | +18.45 | +5.50 | +0.77 | +8.24 |

### Key Findings

**AMD vs. Baselines:** AMD consistently outperforms all baselines. Baseline methods (ReasoningBank, MemP, SASM) yield inconsistent improvements and sometimes degrade performance (e.g., ReasoningBank reduces Qwen3-4B AppWorld accuracy from 14.88% to 10.71%), suggesting that flat or inadequately structured teacher memory introduces noise exceeding small students' comprehension capabilities.

**Matching/Surpassing Teacher:** Gemma4-E4B (54.17%) and Qwen3-8B (51.79%) surpass GPT-5-mini on AppWorld; three students outperform the teacher on BFCL V3. This indicates that AMD distills **transferable decision-making patterns** that students re-instantiate under their own inductive biases.

**Interaction Efficiency:** AMD reduces student turn counts closer to teacher levels (e.g., Qwen3-4B on AppWorld: 23.8 → 14.9 turns vs. teacher's 10.1).

### Ablation Study

**Table 2.** Ablation study across models and benchmarks. WF, ST, and FN denote workflow, subtask, and function memory.

| Method | Qwen3-4B | Qwen3-8B | LLaMA3.1-8B | Gemma4-E4B | Qwen3-4B | Qwen3-8B | LLaMA3.1-8B | Gemma4-E4B |
|---|---|---|---|---|---|---|---|---|
| | AppWorld | | | | BFCL V3 | | | |
| Zero-shot | 14.88 | 25.60 | 8.93 | 24.40 | 15.50 | 38.00 | 9.00 | 37.25 |
| WF | 22.02 | 30.36 | 11.31 | 30.36 | 35.50 | 40.00 | 11.50 | 45.50 |
| WF + FN | 24.11 | 33.93 | 14.88 | 40.48 | 35.50 | 41.50 | 12.50 | 46.00 |
| WF + ST | 47.02 | 51.19 | 30.36 | 53.57 | 37.50 | 45.50 | 14.00 | 45.00 |
| **WF + ST + FN** | **49.40** | **51.79** | **27.38** | **54.17** | **38.50** | **45.50** | **14.50** | **46.00** |
| Student Memory | 16.07 | 29.76 | 8.93 | 25.60 | 27.00 | 43.00 | 9.50 | 44.50 |

**Key findings from ablation:**
- **Subtask memory** yields the largest incremental improvement (e.g., +25.0%p for Qwen3-4B over WF alone on AppWorld)
- Student-generated memory performs close to zero-shot, confirming teacher trajectory quality is essential
- All three memory types are necessary for optimal performance

### Teacher Agent Effect

**Table 3: Effect of teacher agents on AppWorld. GPT-5-mini shows the strongest transfer effectiveness for Qwen3-4B student.**

| Teacher | Teacher Acc | Qwen3-4B | Qwen3-8B |
|---|---|---|---|
| Zero-shot | - | 14.88 | 25.60 |
| GPT-5.5 | 91.08 | 47.02 | 58.93 |
| DeepSeek V4 Pro | 81.55 | 38.10 | 57.14 |
| GPT-5-mini | 50.00 | 49.40 | 51.79 |
| Qwen3-32B | 34.42 | 29.76 | 39.29 |

For the stronger Qwen3-8B student, **teacher accuracy predicts distillation quality** (GPT-5.5 → 58.93% student accuracy). However, for the weaker Qwen3-4B, this ordering breaks down: GPT-5-mini (49.40%) outperforms DeepSeek V4 Pro (38.10%) despite lower teacher accuracy, highlighting the importance of **teacher–student compatibility**.

### Student Model Size Effect

Accuracy increases with model size (21.43%, 49.40%, 51.79%, 52.68% for 1.7B, 4B, 8B, 14B), but the **accuracy gain peaks at 4B** (+34.52%p). Larger models (8B, 14B) match or slightly surpass teacher performance, with diminishing room for improvement.

### Retrieval Count Effect

Performance at $k=1$ is optimal or near-optimal for all memory types. Increasing $k$ degrades performance, most dramatically for Subtask memory (49.40% → 33.34% as k rises from 1 to 5). This highlights the importance of **precise, high-confidence memory injection** over breadth for small models.

### Memory Representation

**Table 4: Effect of memory representation on App-World. Our design choice achieves the best accuracy.**

| Workflow | Subtask | Function | Acc |
|---|---|---|---|
| Code | Code | Code | 44.05 |
| **Text** | **Code** | **Code** | **49.40** |
| Text | Text | Code | 23.21 |
| Text | Code | Text | 47.62 |
| Text | Text | Text | 26.19 |

The optimal representation varies by granularity: **natural language for Workflow** (49.40% vs 44.05% for code), **code-centric for Subtask** (49.40% vs 23.21% for text), and **code for Function** (49.40% vs 47.62% for text). Replacing all three with natural language drops accuracy to 26.19%.

---

## Theoretical and Practical Implications

### Theoretical Contributions

1. **First systematic study of teacher-to-student memory distillation** for small agents, revealing that naive memory transfer fails due to the capability gap.
2. **Hierarchical memory organization** (task → subtask → function) bridges the gap between teacher and student comprehension levels, analogous to curriculum structure in learning.
3. **Memory representation matters**: The finding that planning knowledge transfers best as natural language while execution knowledge requires concrete code challenges the assumption that a single representation works for all memory types.

### Practical Implications

1. **Training-free knowledge transfer**: AMD requires no parameter updates, making it a cost-effective way to enhance small agents.
2. **Teacher–student compatibility** is more important than raw teacher accuracy for weaker students—practitioners should consider model families and inductive bias alignment when selecting teachers.
3. **Precision over breadth**: The k=1 result suggests that for small models, injecting fewer high-confidence memories is more effective than providing a larger memory pool.
4. **4B-scale students benefit most**: The framework is most impactful for models that are capable enough to leverage transferred knowledge but have substantial room for improvement over their zero-shot baseline.

---

## Conclusion

AMD demonstrates that **hierarchically structured memory distillation** enables small LLM agents to achieve teacher-level performance on tool-use benchmarks without any training. The three memory types (Workflow, Subtask, Function) form a complementary hierarchy spanning task planning, execution strategy, and API interaction—each resolving distinct failure modes.

### Future Directions

- **Generalization** to multimodal environments and open-ended coding tasks where the action space is not a fixed set of callable operations
- **Adaptive memory**: Incorporating student's own successes/failures at test time and adapting to distribution shifts
- **Adaptive teacher selection**: Determining which teacher best suits a given student remains an open problem, as stronger teachers do not always benefit smaller students
- **Scaling**: Extending the framework to larger student models and more complex task domains

### Limitations

- Evaluated only on text-based tool-use benchmarks (Python APIs, structured function calls)
- Memory is frozen at inference time—no incorporation of student's test-time experience
- Teacher quality and student compatibility jointly determine transfer effectiveness; adaptive teacher selection remains unresolved

---

> **Key Quote:** "The three memory types form a complementary hierarchy operating at distinct layers of agent competence, spanning task planning, execution strategy, and API interaction. All three are necessary for the agent to reach a correct solution on tasks requiring their combined coverage."

---

_Markdown view of https://picx.dev/p/A7f2AC, served by PicX — AI-generated visual whiteboard summaries of research papers._