Summary (Overview)

• Novel Framework: Dream-RSI introduces a framework for recursive self-improvement (RSI) of exploration policies in AI-driven discovery, addressing the critical bottleneck of delayed and expensive feedback in meta-level policy optimization.
• Key Insight: Completed discovery histories can be repurposed as "replay simulators" — structured trees of past exploration decisions and outcomes that enable fast, zero-execution-cost evaluation of alternative exploration policies.
• Core Mechanism: The system operates in a three-stage loop: (1) online exploration to collect discovery trees, (2) construction of replay simulators from recorded history, and (3) "dreaming" — simulating thousands of candidate policies against the replay simulator before redeploying the best one online.
• Empirical Results: Across 8 tasks in algorithm engineering, mathematical optimization, and GPU kernel engineering, Dream-RSI achieves competitive or superior discovery quality while reducing discovery cost by up to 162× compared to SimpleTES and 1.7× compared to fixed-exploration baselines.
• Lightweight Design: Only the exploration policy code changes; the underlying coding agent, evaluator, and execution interfaces remain unchanged.

Introduction and Theoretical Foundation

Background and Motivation

Recursive self-improvement (RSI) has become an ambitious goal for autonomous AI systems, with iterative discovery loops driving progress across algorithm design, mathematical optimization, systems design, and agent self-improvement. As agent capabilities improve, discovery increasingly requires long-horizon exploration over vast search spaces, often spanning thousands of proposal–evaluation cycles.

The Core Dilemma

The paper identifies a fundamental dilemma in existing approaches:

"Fixed strategies fail to adapt as search spaces scale, while online policy optimization requires navigating vast meta-search spaces under delayed and expensive feedback over long-horizon rollouts."

Two fundamental bottlenecks in online policy optimization:

1. Delayed and expensive feedback: Assessing an exploration policy requires observing how it shapes the subsequent discovery process over many proposal–evaluation cycles.
2. Vast meta-policy space: A newly proposed policy may perform poorly, so many alternatives may need to be tried.

Theoretical Foundation: World Models Analogy

The paper draws a formal analogy to model-based reinforcement learning and World Models (Ha and Schmidhuber, 2018; Hafner et al., 2023). Just as the Dreamer family of models learns a compact dynamics model from collected experience and improves policies by imagining trajectories within that model, Dream-RSI treats completed discovery histories as empirical replay simulators — grounded models of the observed portion of the discovery space.

The key intuition is stated as:

"A fast and inexpensive simulator of discovery would allow many exploration policies to be evaluated before costly online deployment. Surprisingly, completed discovery histories already provide such a simulator."

Methodology

System Architecture

Dream-RSI operates through three core stages (Figure 1):

1. O1 Online Explore: The current exploration policy guides a coding agent to expand a discovery tree and log historical traces.
2. O2 Construct Replay Simulator: The generated discovery tree is converted into a reusable simulator pool.
3. O3 Dreaming-based Policy Improvement: The agent "dreams" up a massive pool of alternative policies, feeds them into the replay simulator for simulated executions, derives rapid feedback, and refines its strategy.

Formal Framework

Discovery Trees and Decision Interface: A discovery tree $\mathcal{T}$ is rooted at $r$, with each non-root node $\nu$ having exactly one primary parent. The exploration policy observes a tree and selects nodes from which to continue exploration. The eligible nodes form the set:

$$A(\mathcal{T}) = \{r\} \cup \{\nu \in \mathcal{T} : \nu \text{ is a leaf}\}$$

With $W \geq 1$ parallel workers, the policy's action is a batch $C \in A(\mathcal{T}; W)$, where:

$$A(\mathcal{T}; W) = \{C \subseteq A(\mathcal{T}) : |C| \leq W\}$$

Online Rollout: At outer iteration $t$, policy $\pi_t$ guides a new online rollout. The rollout allows at most $K_1$ rounds. At round $k$, the policy selects a node batch $C_t^k \in A(\mathcal{T}_t^k; W)$, and each selected node is assigned to a worker for parallel generation–evaluation attempts.

Offline Replay: During the offline phase, the history $\mathcal{H}_t$ remains fixed while the method constructs and evaluates $M \geq 1$ policy versions. For each policy–tree pair $(m, i)$, replay resets the policy's state and starts from $\mathcal{T}_i^{m,0} = \{r\}$. The transition is:

$$\mathcal{T}_i^{m,k+1} = \mathcal{T}_i^{m,k} \cup \bigcup_{\nu \in C_i^{m,k}} \text{Child}(\nu; \mathcal{T}_i, \mathcal{T}_i^{m,k})$$

where $\text{Child}(\nu; \mathcal{T}_i, \mathcal{T}_i^{m,k})$ denotes the set of unobserved children of $\nu$ on the recorded tree.

Replay Objective: The replay score balances discovery quality, execution cost, and parallelism:

$$V_i^m = \underbrace{\max_{\nu \in \mathcal{T}_i^{m, k_i^{m,\star}}} s_\nu}_{\text{discovery quality}} - \underbrace{\beta_1 N_i^m}_{\text{execution cost}} + \underbrace{\beta_2 \frac{N_i^m}{\max\{1, k_i^{m,\star}\}}}_{\text{parallelism bonus}} \tag{1}$$

where $N_i^m = |\mathcal{T}_i^{m, k_i^{m,\star}}| - 1$ is the number of revealed non-root nodes, and $\beta_1, \beta_2 \geq 0$ are fixed coefficients.

Policy Selection: The evaluation score of policy version $\pi_t^m$ is its average replay score across the fixed history:

$$V^m = \frac{1}{t} \sum_{i=1}^{t} V_i^m$$

The next online policy is selected as $\pi_{t+1} = \pi_t^{m^\star}$, where $m^\star \in \arg\max_{m \in \{0, \ldots, M-1\}} V^m$, guaranteeing the selected policy is no worse than the current policy on the fixed history.

Experimental Setup

• Models: Gemini-3.1 Pro and Gemini-3.7-Flash via the Gemini CLI
• Baselines: Recursive Fixed Exploration (primary controlled baseline), SimpleTES, sklearn, glmnet, AlphaEvolve, and others
• Domains: Algorithm engineering (Lasso path solver), mathematical optimization (sum-difference, autocorrelation, circle packing), GPU kernel engineering (KernelBench tasks)

Empirical Validation / Results

Algorithm Engineering (Lasso Regularization Path)

Key Results (Figure 3a):

Dream-RSI reduces agent calls by up to 162× over SimpleTES and 1.7× over fixed-exploration baselines while achieving lower average downstream runtime.

Mathematical Optimization (Table 1)

Dream-RSI outperforms SimpleTES on Sum-Diff with fewer than 1,000 generations (vs. 51,200 for SimpleTES), achieving over 50× budget savings.

GPU Kernel Engineering (Figure 4)

• VGG16: Dream-RSI reaches comparable performance with 2.43× fewer generations
• LayerNorm: Dream-RSI reaches comparable performance with 1.79× fewer generations
• ConvDiv: Dream-RSI achieves 2.09× higher performance under comparable budgets
• ConvMax: Dream-RSI achieves 1.44× higher performance under comparable budgets

Further Analysis

Historical Inductive Biases: Explicit semantic guidance (prompt-level insights from history) consistently underperforms the replay-simulator approach, suggesting that strong semantic inductive biases over-constrain the search space in long-horizon discovery.

Evolution of Exploration Behavior: The learned policy exhibits adaptive patterns — conserving compute initially (reducing attempts from 110 to 50), then increasing exploration effort when progress plateaus, coinciding with further performance gains.

Theoretical and Practical Implications

Theoretical Contributions

1. History as Replay Simulator: The paper conceptualizes completed discovery histories as replay simulators, transforming delayed exploration feedback into reusable, immediate, low-cost feedback for meta-exploration policy evaluation.

2. Meta-Layer RSI Loop: Dream-RSI establishes a recursive self-improvement loop at the meta-exploration layer, addressing the fundamental challenge that "meta-level strategies are difficult to improve because their quality is often revealed only after costly long-horizon rollouts."

3. Off-Policy Evaluation: The framework enables off-policy evaluation of exploration strategies without invoking repetitive, expensive online evaluations — a significant theoretical advance for meta-optimization.

Practical Implications

• Cost Reduction: Substantial reduction in discovery compute (up to 162×) while maintaining or improving discovery quality
• Generalizability: Demonstrated effectiveness across three distinct scientific discovery domains
• Lightweight Integration: The orchestration layer leaves the underlying coding agent unchanged, making it easy to integrate with existing discovery systems
• Adaptive Behavior: The system automatically adjusts exploration effort based on progress, conserving compute when performance improves and intensifying exploration during plateaus

Conclusion

Dream-RSI presents a novel framework for recursive self-improvement of exploration policies in AI-driven discovery. The key innovation is converting accumulated discovery history from static context into an active, replayable simulator, enabling "dreaming" — rapid, zero-execution-cost evaluation of alternative exploration policies.

Main takeaways:

1. Completed discovery histories contain valuable structural information that can serve as empirical replay simulators
2. Meta-level exploration policies can be recursively improved through simulated evaluation without rerunning expensive online rollouts
3. The framework achieves competitive or improved discovery quality while substantially reducing discovery cost across multiple domains

Future directions implied by the work include:

• Extending the replay simulator concept to more complex discovery structures
• Investigating the trade-offs between semantic guidance and replay-based learning
• Scaling the approach to even larger discovery systems and more diverse domains

The paper concludes that Dream-RSI "addresses the core bottleneck of meta-optimization: delayed and expensive feedback, which is especially severe in long-horizon discovery settings."