Summary (Overview)

• SwarmEvolve is a novel evolutionary search framework where hundreds of low-cost LLM agents per round reason in latent space (without decoding to text) and communicate by passing KV caches rather than text messages.
• The framework achieves 5–13× higher throughput than state-of-the-art text-based frameworks (OpenEvolve, ShinkaEvolve, CodeEvolve), evaluating far more programs per hour on a single GPU.
• On 21 problems spanning mathematical discovery and scientific computing, SwarmEvolve achieves the highest average score with all tested LLMs (Qwen3 8B, 14B, 32B-FP8), and reaches baselines' 60-minute scores in a median of 3–8 minutes.
• With a small Qwen3 8B model, SwarmEvolve improves on several constructions originally found by AlphaEvolve using frontier Gemini models, including a best-known circle packing result.
• Key innovations include a novelty filter based on cosine distance between agents' hidden states, and built-in crossover in latent space via a summarizer agent that reads concatenated KV caches.

---

Introduction and Theoretical Foundation

Background

Many problems in mathematics and scientific computing can be framed as searching for a good program (e.g., symbolic regression, optimizing constants in inequalities, building PDE solvers). Evolutionary computation iteratively improves candidate programs through variation, crossover, and selection. Recent work uses LLMs as mutation/crossover operators:

• FunSearch found new cap set constructions
• AlphaEvolve found a more efficient algorithm for multiplying 4×4 complex matrices
• Later frameworks add MAP-Elites archives, novelty rejection, and multi-agent coordination

Motivation

Existing LLM-guided evolutionary frameworks are slow and expensive because:

• Each agent writes a long text chain of thought
• Growing archives and evaluator feedback make contexts very long
• Scaling to many agents or larger models multiplies cost

The central research question: Can swarms of low-cost agents make evolutionary search faster and better?

Theoretical Foundation

SwarmEvolve builds on two recent developments:

1. Latent reasoning: LLMs reason with continuous vectors instead of sampled tokens. The last hidden state $\mathbf{h}_t$ is fed back as the next input embedding $\mathbf{e}_{t+1}$ without decoding text.

2. Latent communication: LLM agents exchange KV caches instead of text messages, enabling a summarizer agent to attend directly to other agents' reasoning traces.

---

Methodology

LLM and KV Cache Preliminaries

For a decoder-only transformer, the input token at position $t$ is embedded as $\mathbf{e}_t \in \mathbb{R}^d$ (where $d$ is the hidden dimension). The LLM stores keys and values in a KV cache $\{(\mathbf{K}^{(\ell)}, \mathbf{V}^{(\ell)})\}_{\ell=1}^{L}$ for each layer.

Rotary Position Embeddings (RoPE) — Qwen3 models use RoPE, where the attention score depends only on position offset:

$$(\mathbf{R}_i \mathbf{q})^\top (\mathbf{R}_j \mathbf{k}) = \mathbf{q}^\top \mathbf{R}_{j-i} \mathbf{k}. \tag{1}$$

Moving a KV cache block by $\delta$ positions requires multiplying each cached key by $\mathbf{R}_\delta$.

Latent Reasoning

Each latent reasoning step maps the last hidden state to the next input embedding:

$$\mathbf{e}_{t+1} = \gamma \frac{\mathbf{W}_{\mathrm{a}} \mathbf{h}_t}{\|\mathbf{W}_{\mathrm{a}} \mathbf{h}_t\|}, \qquad \mathbf{W}_{\mathrm{a}} = \mathbf{W}_{\mathrm{in}}^\top \mathbf{W}_{\mathrm{out}} \left(\mathbf{W}_{\mathrm{out}}^\top \mathbf{W}_{\mathrm{out}} + \rho \mathbf{I}\right)^{-1}, \tag{2}$$

where $\mathbf{W}_{\mathrm{in}}, \mathbf{W}_{\mathrm{out}} \in \mathbb{R}^{|\mathcal{V}| \times d}$ are input/output embedding matrices, $|\mathcal{V}|$ is vocabulary size, $\gamma$ is the mean norm of input token embeddings, and $\rho$ is a ridge regression regularization weight.

Latent Communication

The summarizer's KV cache concatenates all agents' caches along the sequence dimension:

$$\mathbf{K}^{(\ell)} = [\mathbf{K}_1^{(\ell)}; \ldots; \mathbf{K}_J^{(\ell)}; \mathbf{K}_{\text{sum}}^{(\ell)}], \qquad \mathbf{V}^{(\ell)} = [\mathbf{V}_1^{(\ell)}; \ldots; \mathbf{V}_J^{(\ell)}; \mathbf{V}_{\text{sum}}^{(\ell)}] \quad \text{for each layer } \ell. \tag{3}$$

SwarmEvolve Workflow (per round)

1. Form swarms: 10 swarms × 45 latent agents = 450 agents total:

   • 4 refinement swarms: improve the two best programs (2 swarms each)
   • 3 hybridization swarms: combine ideas from 5 random top-10 programs
   • 2 exploration swarms: explore less-considered directions
   • 1 fresh-start swarm: find new approaches from the initial program only

2. Reason in latent space: Each agent takes 40 latent reasoning steps; all 450 run in one batch with paged attention.

3. Filter for novelty: Each agent is represented by its mean hidden state:

   $$\bar{\mathbf{h}}_j = \frac{1}{40} \sum_{t=1}^{40} \mathbf{h}_{j,t}, \qquad j = 1, \ldots, 45. \tag{4}$$

   Greedy farthest-point selection in cosine distance keeps the 30 most distinct agents:

   $$j^{\star} = \arg\max_{j \notin \mathcal{K}} \min_{i \in \mathcal{K}} \left(1 - \mathbf{z}_j^\top \mathbf{z}_i\right), \tag{5}$$

   where $\mathbf{z}_j = (\bar{\mathbf{h}}_j - \boldsymbol{\mu}) / \|\bar{\mathbf{h}}_j - \boldsymbol{\mu}\|$ is the centered, normalized hidden state.

4. Transfer KV caches: The summarizer's KV cache concatenates problem statement, 30 kept agents' reasoning, parent programs, and instruction:

   $$\mathbf{K}^{(\ell)} = [\mathbf{K}_{\mathrm{prob}}^{(\ell)}; \mathbf{K}_1^{(\ell)}; \ldots; \mathbf{K}_{30}^{(\ell)}; \mathbf{K}_{p_1}^{(\ell)}; \ldots; \mathbf{K}_{p_Q}^{(\ell)}; \mathbf{K}_{\mathrm{ins}}^{(\ell)}], \tag{6}$$

   with keys rotated by $\mathbf{R}_\delta$ to account for position offsets.

5. Write programs: Summarizer decodes one complete program per swarm.

6. Tune parameters: A parameter tuning agent proposes up to 8 single-number edits per round.

7. Evaluate: Deduplicate, syntax-check, auto-import, and score on CPU cores.

---

Empirical Validation / Results

Experimental Setup

• 21 problems across 5 domains: packing, geometry, analysis, combinatorics, scientific computing
• 16 problems from AlphaEvolve's Appendix B; 5 new scientific computing problems (sparse linear systems, eigenvalues, Burgers/convection-diffusion/Helmholtz equations)
• 60 minutes per run, 1 GPU, 8 CPU cores, 3 seeds per problem
• Models: Qwen3 8B, 14B, 32B-FP8

Key Results (Table 1 Summary)

Key findings:

• SwarmEvolve with 8B outperforms all baselines with 32B-FP8
• Solves all 5 sparse linear systems to full accuracy with 32B-FP8 (no baseline ever achieves this)
• Evaluates 5–13× more programs per hour than baselines
• Reaches baselines' 60-minute scores in median 3–8 minutes
• With Qwen3 8B, surpasses AlphaEvolve's published constructions on 3 problems (circle packing of 26 circles reaching the best known sum of radii 2.635983, and two autocorrelation inequalities)

Latent Space Analysis

Principal component analysis of agents' mean hidden states shows:

• Across swarms: Each swarm occupies a distinct region of latent space, ordered by role (refinement → exploration → fresh-start)
• Within swarms: Agents cluster by parent program; the novelty filter keeps agents from every cluster, ensuring coverage

---

Theoretical and Practical Implications

Significance

1. Latent reasoning is critical: Replacing latent reasoning with text reasoning (even with higher token budgets) drops scores significantly because text makes rounds slower, reducing the number of programs evaluated.

2. Scaling agents helps early search: Larger swarms explore more design space per round, which matters most when evaluations are expensive (e.g., large simulations).

3. Affordable large-scale search: SwarmEvolve makes large-scale program search feasible on a single GPU, democratizing AI-driven scientific discovery beyond frontier-model labs.

Practical Applications

• Mathematics: Searching for new constructions and proof strategies
• Computing: Improving algorithms for speed and accuracy
• Scientific computing: Writing better PDE solvers, symbolic regression, efficient numerical methods

Limitations

1. All agents must share one LLM (KV caches are model-specific); learned translations could extend to heterogeneous swarms
2. Requires standard attention; linear-attention/state-space models need state-merging mechanisms
3. KV cache memory grows linearly with agents × steps × layers, requiring large GPUs for big models

---

Conclusion

SwarmEvolve demonstrates that evolutionary search can be fast, cheap, and strong simultaneously. By having hundreds of low-cost latent agents reason in embedding space and communicate via KV caches, the framework:

• Evaluates far more programs per hour than text-based frameworks
• Achieves higher final scores with smaller models
• Discovers results competitive with frontier-LLM frameworks using only 8B open-weight models

The work shows that large-scale search for new mathematical constructions, algorithms, and scientific computing methods does not necessarily require frontier models or large compute budgets. Swarms of small latent agents make AI-driven scientific discovery fast, cheap, capable, and widely available.

Future directions include:

• Learned KV-cache translations for multi-model swarms
• Compressed latent messages to reduce memory
• Distributed rounds across multiple GPUs for larger models and longer runs