Summary (Overview)

• EvoMem introduces a persistent memory architecture for LLM-based evolutionary code search, enabling the reuse of successful mutation knowledge across runs and tasks.
• The system operates in two phases: an offline write phase that extracts, filters, deduplicates, and stores successful mutation ideas with provenance, and an online read phase that retrieves bounded, task-relevant advice during mutation.
• Evaluated across geometric optimization, multi-hop question answering, GPU kernel optimization, and scientific-code tuning, EvoMem shows an average target-metric gain of 6.40% and an average search speedup of 5.93× across benchmarks.
• A mechanism-level audit shows that relevance-based retrieval achieves a 4.05% idea-acceptance rate versus 0.25% for random memory selection — a 16.2× improvement in the likelihood that retrieved advice is incorporated into generated code.
• The design preserves the underlying evolutionary loop unchanged; memory enters only as an optional, bounded advice channel, limiting negative transfer.

---

Introduction and Theoretical Foundation

Background

LLM-based evolutionary code search (e.g., AlphaEvolve) combines LLM-generated code mutations with evolutionary selection: retrieve candidate programs, prompt an LLM to propose a modification, evaluate the result, and retain successful variants in a population. This evaluator-in-the-loop structure has proven effective for matrix multiplication, mathematical search, and systems optimization.

The Problem

Current evolutionary code-search systems treat each run as a self-contained search process. Successful runs produce more than a final program — they reveal reusable tactics such as:

• Useful decompositions
• Numerical stabilization tricks
• Pruning rules
• Data-layout changes
• Prompting strategies

In standard pipelines, this knowledge remains implicit in evolved programs and logs, forcing new runs to rediscover similar ideas through broad search.

The Core Challenge

Naively reusing prior artifacts is not enough. A concrete program is usually tied to a particular task, metric, and interface, while simply mixing many experiments into one population can make retrieval less precise and can expose the mutation model to irrelevant context.

What is needed is a mechanism that:

1. Abstracts useful mutation knowledge from past runs
2. Preserves provenance to keep it auditable
3. Retrieves only a bounded amount of relevant advice during future mutations

---

Methodology

Architecture Overview

The data flow is formalized as:

$$P_{r} \xrightarrow{\mathrm{extract}} C_{r} \xrightarrow{\mathrm{write}} M \xrightarrow{\mathrm{select}(p, \tau, \mu)} I_{p} \hookrightarrow \mathrm{mutationprompt}$$

Where:

• $P_r$ = set of programs from run $r$
• $C_r$ = set of cards extracted from that run
• $M$ = persistent memory bank
• $I_p$ = small list of instructions selected for parent program $p$ on task $\tau$ with metric description $\mu$

Write Phase (Offline)

Insights Extraction: The analysis stage collects completed programs, filters out root programs and invalid outputs, and converts candidates into normalized records containing fitness, generation, parentage, task context, mutation strategy, code, and improvement descriptions.

Two Analysis Modes:

1. Default mode: An LLM compares new improvements against a working collection of previously extracted ideas, classifying them as genuinely new, revisions, or reformulations.
2. Fast analyzer: Uses DBSCAN clustering over embedding space to group semantically related candidates, then asks an LLM to refine ambiguous clusters.

Usage Tracking: The contribution of a selected memory item is computed as:

$$\Delta f = f(\text{child}) - \max_{p \in \text{parents}} f(p)$$

computed only when both child and parent fitness values are valid. Median aggregation is used because evolutionary fitness changes are typically sparse and heavy-tailed.

Memory Generation

The write pipeline stores two kinds of entries:

1. Abstract ideas/tactics — description, task context, explanations, provenance, usage statistics
2. High-performing programs — fitness, code, task context, links to associated ideas

Deduplication Score:

$$\operatorname{score}(c) = \sum_{q \in Q} w_q s_q(c)$$

where $s_q(c)$ is the retriever score for candidate card $c$ under query view $q$. The highest-scoring candidates are passed to an LLM decision policy that chooses whether to store, discard, or merge.

Promotion Rule (conservative): An idea is favored when it has:

• An identifiable introduction point
• Improvement over its strongest available parent
• Rarely produces worse descendants
• Supported by at least one additional signal (recurrence, sibling comparison, elite-lineage spread)

Read Phase (Online)

Task-Scope Filtering: An LLM selects source tasks potentially relevant to the current task, restricting retrieval to ideas from those tasks.

Retrieval: Ranks memory cards using lexical and embedding-based similarity over multiple semantic views:

• Mechanism description
• Source task context
• Explanation of why the idea helped
• Compositions of these fields

Key Design Properties:

1. Memory is optional — control and treatment runs differ only in the memory advice channel
2. Retrieval is bounded — fixed item and iteration budgets
3. System is auditable — selected cards are recorded with candidates that used them

---

Empirical Validation / Results

Experimental Setup

Memory-Generation Set: Circle Packing (26), Heilbronn-style point placement, Kissing Number (11D), HotpotQA, HoVer, GSM8K, selected AlgoTune tasks, and selected KernelBench kernels.

Evaluation Set (cross-task transfer): Circle Packing (26 and 32), Hexagon Packing, Heilbronn-style, Kissing Number (12D), HoVer, HotpotQA, AlgoTune Power Control, AlgoTune Kalman, KernelBench L1 P66 and L3 P21.

Key Protocol: Memories from the target benchmark are excluded from the memory bank at test time — only memories from other benchmarks are retrievable.

Main Results

Relative gain is calculated per matched pair as $g_i = (F_i^{memory} - F_i^{baseline}) / |F_i^{baseline}|$.

Key observations:

• Kissing Number illustrates an acceleration-only case: final quality unchanged, but memory-enabled runs reach the baseline best score ~6.8× faster.
• KernelBench shows the highest mean gain (16.89%) but also the highest variability (0% to 65.93%).

Memory Utilization

• Full memory-bank coverage: Only HoVer
• High coverage: Heilbronn (81.3%), Circle Packing (26) (69.9%), Hexagon Packing (64.4%)
• Low coverage: KernelBench (5.4%), Kissing Number (3.3%)

Retrieval Quality Audit

Examples of Reused Memory

Retrieved memories encoded transferable execution strategies rather than narrow task-specific tricks:

• Circle Packing ← Heilbronn: "Use annealed force-directed repulsion and low-discrepancy initialization"
• Hexagon Packing ← Circle/geometric: "Use basin hopping, simulated annealing, and temperature-scaled repulsive forces"
• HotpotQA ← Multi-hop QA traces: "Extract only claim-relevant facts, preserve intermediate entities"
• KernelBench ← Kernel traces: "Avoid over-customized kernels when vendor primitives are faster"

---

Theoretical and Practical Implications

Theoretical Significance

1. Knowledge transfer in evolutionary search: EvoMem demonstrates that some mutation knowledge is more general than the single program in which it first appeared — strategies transfer across related and sometimes substantially different domains.

2. Memory as a first-class component: The results suggest evolutionary coding systems should treat memory as a core architectural element, not an afterthought. Artifacts of a run include not just final code but also reusable design decisions, optimization heuristics, and failure-avoidance patterns.

3. Bounded advice vs. hard constraints: The design shows that injecting memory as suggestions (not constraints) preserves the search's ability to explore novel strategies while biasing toward previously productive directions.

Practical Implications

1. Reduced redundant exploration: Memory-enabled runs consistently reached baseline final scores with fewer evaluated candidates (average 5.93× speedup), reducing compute cost.

2. Cross-task transfer: The system works when target-benchmark memories are excluded, showing that knowledge genuinely transfers across problem families.

3. Auditability: Provenance tracking enables tracing which memories contributed to which improvements, supporting scientific reproducibility.

4. Compute overhead: Memory generation is relatively cheap: ~683K input tokens and ~25K output tokens per run for post-processing, compared to ~7.2M input and ~893K output tokens for evolution.

---

Conclusion

EvoMem introduces a persistent memory architecture for LLM-based evolutionary code optimization that converts successful mutation events into structured, auditable memory cards and retrieves a bounded set of relevant cards during later mutations. Across geometry, reasoning, scientific-code, and GPU-kernel benchmarks, EvoMem is associated with improved efficiency or final target metrics in most tested settings.

The central hypothesis — that LLM-driven evolution can benefit from persistent cross-run memory — receives initial support: memory reuse reduces redundant rediscovery and biases search toward previously productive directions while preserving evaluator-based selection.

Future Directions

1. Scale memory across more runs and domains
2. Adaptive memory reliance based on task and population state
3. Repository-scale evolution capturing project conventions, interface constraints, and cross-file optimization strategies
4. Reduce cost of repeated LLM calls, execution, validation, and selection

Limitations

• High variance across runs (partly inherent to evolutionary search with high-temperature LLM sampling)
• Sensitivity to memory-pipeline quality (extraction, clustering, summarization, retrieval hyperparameters)
• Small memory bank and fixed memory influence during mutation
• End-to-end evaluation rather than component-level ablations (each ablation changes the search trajectory, requiring fresh evaluator-in-the-loop runs)