# SwarmEvolve: Latent Agent Swarms for Evolutionary Discovery and Optimization

> SwarmEvolve uses hundreds of low-cost LLM agents reasoning in latent space and communicating via KV caches to achieve 5–13x higher throughput and top scores, matching frontier-model results with an 8B model.

- **Source:** [arXiv](https://arxiv.org/abs/2611.08483)
- **Published:** 2026-11-14
- **Permalink:** https://picx.dev/p/828NYj
- **Whiteboard:** https://picx.dev/p/828NYj/image

## Summary

## Summary (Overview)

- **SwarmEvolve** is a novel evolutionary search framework where hundreds of low-cost LLM agents per round reason in **latent space** (without decoding to text) and communicate by passing **KV caches** rather than text messages.
- The framework achieves **5–13× higher throughput** than state-of-the-art text-based frameworks (OpenEvolve, ShinkaEvolve, CodeEvolve), evaluating far more programs per hour on a single GPU.
- On **21 problems** spanning mathematical discovery and scientific computing, SwarmEvolve achieves the **highest average score** with all tested LLMs (Qwen3 8B, 14B, 32B-FP8), and reaches baselines' 60-minute scores in a **median of 3–8 minutes**.
- With a small **Qwen3 8B** model, SwarmEvolve improves on several constructions originally found by AlphaEvolve using frontier Gemini models, including a best-known circle packing result.
- Key innovations include a **novelty filter** based on cosine distance between agents' hidden states, and **built-in crossover** in latent space via a summarizer agent that reads concatenated KV caches.

---

## Introduction and Theoretical Foundation

### Background
Many problems in mathematics and scientific computing can be framed as searching for a good program (e.g., symbolic regression, optimizing constants in inequalities, building PDE solvers). Evolutionary computation iteratively improves candidate programs through variation, crossover, and selection. Recent work uses LLMs as mutation/crossover operators:

- **FunSearch** found new cap set constructions
- **AlphaEvolve** found a more efficient algorithm for multiplying 4×4 complex matrices
- Later frameworks add MAP-Elites archives, novelty rejection, and multi-agent coordination

### Motivation
Existing LLM-guided evolutionary frameworks are **slow and expensive** because:
- Each agent writes a long text chain of thought
- Growing archives and evaluator feedback make contexts very long
- Scaling to many agents or larger models multiplies cost

The central research question: **Can swarms of low-cost agents make evolutionary search faster and better?**

### Theoretical Foundation
SwarmEvolve builds on two recent developments:

1. **Latent reasoning**: LLMs reason with continuous vectors instead of sampled tokens. The last hidden state $\mathbf{h}_t$ is fed back as the next input embedding $\mathbf{e}_{t+1}$ without decoding text.

2. **Latent communication**: LLM agents exchange KV caches instead of text messages, enabling a summarizer agent to attend directly to other agents' reasoning traces.

---

## Methodology

### LLM and KV Cache Preliminaries

For a decoder-only transformer, the input token at position $t$ is embedded as $\mathbf{e}_t \in \mathbb{R}^d$ (where $d$ is the hidden dimension). The LLM stores keys and values in a KV cache $\{(\mathbf{K}^{(\ell)}, \mathbf{V}^{(\ell)})\}_{\ell=1}^{L}$ for each layer.

**Rotary Position Embeddings (RoPE)** — Qwen3 models use RoPE, where the attention score depends only on position offset:

$$
(\mathbf{R}_i \mathbf{q})^\top (\mathbf{R}_j \mathbf{k}) = \mathbf{q}^\top \mathbf{R}_{j-i} \mathbf{k}. \tag{1}
$$

Moving a KV cache block by $\delta$ positions requires multiplying each cached key by $\mathbf{R}_\delta$.

### Latent Reasoning

Each latent reasoning step maps the last hidden state to the next input embedding:

$$
\mathbf{e}_{t+1} = \gamma \frac{\mathbf{W}_{\mathrm{a}} \mathbf{h}_t}{\|\mathbf{W}_{\mathrm{a}} \mathbf{h}_t\|}, \qquad \mathbf{W}_{\mathrm{a}} = \mathbf{W}_{\mathrm{in}}^\top \mathbf{W}_{\mathrm{out}} \left(\mathbf{W}_{\mathrm{out}}^\top \mathbf{W}_{\mathrm{out}} + \rho \mathbf{I}\right)^{-1}, \tag{2}
$$

where $\mathbf{W}_{\mathrm{in}}, \mathbf{W}_{\mathrm{out}} \in \mathbb{R}^{|\mathcal{V}| \times d}$ are input/output embedding matrices, $|\mathcal{V}|$ is vocabulary size, $\gamma$ is the mean norm of input token embeddings, and $\rho$ is a ridge regression regularization weight.

### Latent Communication

The summarizer's KV cache concatenates all agents' caches along the sequence dimension:

$$
\mathbf{K}^{(\ell)} = [\mathbf{K}_1^{(\ell)}; \ldots; \mathbf{K}_J^{(\ell)}; \mathbf{K}_{\text{sum}}^{(\ell)}], \qquad \mathbf{V}^{(\ell)} = [\mathbf{V}_1^{(\ell)}; \ldots; \mathbf{V}_J^{(\ell)}; \mathbf{V}_{\text{sum}}^{(\ell)}] \quad \text{for each layer } \ell. \tag{3}
$$

### SwarmEvolve Workflow (per round)

1. **Form swarms**: 10 swarms × 45 latent agents = 450 agents total:
   - **4 refinement swarms**: improve the two best programs (2 swarms each)
   - **3 hybridization swarms**: combine ideas from 5 random top-10 programs
   - **2 exploration swarms**: explore less-considered directions
   - **1 fresh-start swarm**: find new approaches from the initial program only

2. **Reason in latent space**: Each agent takes 40 latent reasoning steps; all 450 run in one batch with paged attention.

3. **Filter for novelty**: Each agent is represented by its mean hidden state:
   $$
   \bar{\mathbf{h}}_j = \frac{1}{40} \sum_{t=1}^{40} \mathbf{h}_{j,t}, \qquad j = 1, \ldots, 45. \tag{4}
   $$
   Greedy farthest-point selection in cosine distance keeps the 30 most distinct agents:
   $$
   j^{\star} = \arg\max_{j \notin \mathcal{K}} \min_{i \in \mathcal{K}} \left(1 - \mathbf{z}_j^\top \mathbf{z}_i\right), \tag{5}
   $$
   where $\mathbf{z}_j = (\bar{\mathbf{h}}_j - \boldsymbol{\mu}) / \|\bar{\mathbf{h}}_j - \boldsymbol{\mu}\|$ is the centered, normalized hidden state.

4. **Transfer KV caches**: The summarizer's KV cache concatenates problem statement, 30 kept agents' reasoning, parent programs, and instruction:
   $$
   \mathbf{K}^{(\ell)} = [\mathbf{K}_{\mathrm{prob}}^{(\ell)}; \mathbf{K}_1^{(\ell)}; \ldots; \mathbf{K}_{30}^{(\ell)}; \mathbf{K}_{p_1}^{(\ell)}; \ldots; \mathbf{K}_{p_Q}^{(\ell)}; \mathbf{K}_{\mathrm{ins}}^{(\ell)}], \tag{6}
   $$
   with keys rotated by $\mathbf{R}_\delta$ to account for position offsets.

5. **Write programs**: Summarizer decodes one complete program per swarm.

6. **Tune parameters**: A parameter tuning agent proposes up to 8 single-number edits per round.

7. **Evaluate**: Deduplicate, syntax-check, auto-import, and score on CPU cores.

---

## Empirical Validation / Results

### Experimental Setup
- **21 problems** across 5 domains: packing, geometry, analysis, combinatorics, scientific computing
- 16 problems from AlphaEvolve's Appendix B; 5 new scientific computing problems (sparse linear systems, eigenvalues, Burgers/convection-diffusion/Helmholtz equations)
- 60 minutes per run, 1 GPU, 8 CPU cores, 3 seeds per problem
- Models: Qwen3 8B, 14B, 32B-FP8

### Key Results (Table 1 Summary)

| Metric | SwarmEvolve (8B) | OpenEvolve (8B) | ShinkaEvolve (8B) | CodeEvolve (8B) |
|---|---|---|---|---|
| **Average score** | **0.804** | 0.693 | 0.642 | 0.700 |
| **Average score (14B)** | **0.837** | 0.771 | 0.686 | 0.739 |
| **Average score (32B-FP8)** | **0.869** | 0.774 | 0.733 | 0.731 |

**Key findings:**
- SwarmEvolve with **8B** outperforms all baselines with **32B-FP8**
- Solves all 5 sparse linear systems to full accuracy with 32B-FP8 (no baseline ever achieves this)
- Evaluates **5–13× more programs per hour** than baselines
- Reaches baselines' 60-minute scores in **median 3–8 minutes**
- With Qwen3 8B, surpasses AlphaEvolve's published constructions on **3 problems** (circle packing of 26 circles reaching the best known sum of radii 2.635983, and two autocorrelation inequalities)

### Latent Space Analysis
Principal component analysis of agents' mean hidden states shows:
- **Across swarms**: Each swarm occupies a distinct region of latent space, ordered by role (refinement → exploration → fresh-start)
- **Within swarms**: Agents cluster by parent program; the novelty filter keeps agents from every cluster, ensuring coverage

---

## Theoretical and Practical Implications

### Significance
1. **Latent reasoning is critical**: Replacing latent reasoning with text reasoning (even with higher token budgets) drops scores significantly because text makes rounds slower, reducing the number of programs evaluated.

2. **Scaling agents helps early search**: Larger swarms explore more design space per round, which matters most when evaluations are expensive (e.g., large simulations).

3. **Affordable large-scale search**: SwarmEvolve makes large-scale program search feasible on a **single GPU**, democratizing AI-driven scientific discovery beyond frontier-model labs.

### Practical Applications
- **Mathematics**: Searching for new constructions and proof strategies
- **Computing**: Improving algorithms for speed and accuracy
- **Scientific computing**: Writing better PDE solvers, symbolic regression, efficient numerical methods

### Limitations
1. All agents must share one LLM (KV caches are model-specific); learned translations could extend to heterogeneous swarms
2. Requires standard attention; linear-attention/state-space models need state-merging mechanisms
3. KV cache memory grows linearly with agents × steps × layers, requiring large GPUs for big models

---

## Conclusion

SwarmEvolve demonstrates that evolutionary search can be **fast, cheap, and strong simultaneously**. By having hundreds of low-cost latent agents reason in embedding space and communicate via KV caches, the framework:

- Evaluates far more programs per hour than text-based frameworks
- Achieves higher final scores with smaller models
- Discovers results competitive with frontier-LLM frameworks using only 8B open-weight models

The work shows that large-scale search for new mathematical constructions, algorithms, and scientific computing methods **does not necessarily require frontier models or large compute budgets**. Swarms of small latent agents make AI-driven scientific discovery fast, cheap, capable, and widely available.

**Future directions** include:
- Learned KV-cache translations for multi-model swarms
- Compressed latent messages to reduce memory
- Distributed rounds across multiple GPUs for larger models and longer runs

---

_Markdown view of https://picx.dev/p/828NYj, served by PicX — AI-generated visual whiteboard summaries of research papers._