Summary (Overview)

• New Evaluation Target: RSIBench-Data introduces a controlled benchmark for evaluating LLM agents as data-centric researchers—agents that must form hypotheses about model failures, design training-data strategies, and revise them based on checkpoint feedback—while keeping the post-training stack (training, serving, evaluation) fixed.
• Isolation of Research Capability: Unlike existing benchmarks that entangle research decisions with optimization, serving, and system implementation, RSIBench-Data fixes the infrastructure so that performance differences are attributable to the agent's data-centric research policy.
• Discovery–Reliability Gap: Agents improve on their first valid attempt in 58.33% of settings, but among searches that continue after reaching their best score, 78.26% finish with a lower-scoring final attempt, revealing that feedback-driven revisions are unreliable.
• Agent–Benchmark Interaction: No single agent dominates; the three SWE-style tasks are won by three different agents, and best-to-worst spreads range from 8 to 20 percentage points, showing strong task-dependent interactions.
• Open-Source Testbed: The benchmark is released with code at https://github.com/evolvent-ai/RSIBench-Data, providing a measurable and auditable testbed for studying data-centric research capabilities.

---

Introduction and Theoretical Foundation

Background and Motivation

Recursive self-improvement requires AI systems to repeatedly convert evidence about model failures into effective model improvements. While LLM agents are increasingly evaluated on long-horizon tasks involving software repositories, terminals, and external tools, these environments expose evidence about model behavior rather than training experience that can directly improve the model.

The central research question is: Can LLM agents reliably automate data-centric post-training research? Specifically, an agent must:

1. Formulate a hypothesis about a model failure
2. Turn it into executable and verifiable training data
3. Revise the synthesis strategy from controlled training and evaluation feedback

Gap in Existing Benchmarks

The paper identifies a fundamental tradeoff in existing benchmarks (Table 1):

• Data-centric benchmarks (DataComp, DataComp-LM, AgoraBench, DCA-Bench) isolate individual data operations but lack closed-loop evolution
• End-to-end post-training benchmarks (PostTrainBench, Agent² RL-Bench) give broad control but obscure the contribution of data-centric research decisions

RSIBench-Data occupies the middle ground: it evaluates whether an LLM agent can act as a data-centric researcher while the surrounding training, serving, and official evaluation stack remains fixed.

Formal Problem Definition

Given a fixed base model $M_0$, benchmark evidence $S$, and resource budget $\mathcal{B}$, a researcher policy $\pi$ observes $S$ and its attempt history $H_{<t}$, then proposes training data $D_t$ and a whitelisted training config $c_t$:

$$(D_t, c_t) = \pi(S, H_{<t}), \quad M_t = \text{Train}(M_0, D_t; c_t)$$

Each $M_t$ is evaluated by a fixed evaluation service, producing permitted selection feedback $h_t = \text{Eval}_{\text{sel}}(M_t)$. After $T$ realized attempts, the agent selects a checkpoint $M_{t^*}$ using only $H_T$, before observing official outcomes. The official performance and improvement over the base model are:

$$s_{\text{off}}(\pi) = \text{Eval}_{\text{off}}(M_{t^*}), \quad \Delta_{\text{off}}(\pi) = s_{\text{off}}(\pi) - \text{Eval}_{\text{off}}(M_0)$$

---

Methodology

Fixed Infrastructure

RSIBench-Data fixes the infrastructure that turns a training-data strategy into an evaluated checkpoint:

• Tinker-backed LoRA SFT: All agents train through the same shared SFT backend from the fixed base model Qwen/Qwen3.5-35B-A3B-Base
• Harbor-orchestrated E2B evaluation: Checkpoints are evaluated in sandboxed environments with fixed verifiers and scoring protocols
• Bounded configuration interface: Agents submit training artifacts (not training code) with whitelisted configurations

Agent Inputs and Data-Use Constraints

Each agent receives:

• Target benchmark description and success criteria
• Task-matched public seed repositories or seed examples
• Tool and environment schemas
• Available base-model diagnostics
• Resource budget (16-hour wall-clock, $500 Tinker budget)

Critical constraint: Evaluation-only tasks, labels, trajectories, and protected benchmark materials may not be used as supervision, preventing agents from converting evaluation information into training data.

Closed-Loop Data-Synthesis Research

The loop consists of:

1. Data synthesis: Agents choose data sources, task construction, trajectory representation, filtering rules, verification methods, difficulty curricula, and data mixtures
2. Shared SFT training: Artifacts are validated and trained through the shared LoRA backend
3. Harbor and E2B evaluation: Checkpoints are evaluated with official verifiers, returning selection scores, trajectories, verifier outcomes, and execution diagnostics

Experimental Setup

• 4 × 6 matrix: Four researcher agents (Claude Code Opus-4.8, Claude Code Sonnet-5, Codex gpt-5.6-sol, Codex gpt-5.6-terra) × six benchmarks
• Benchmarks: SWE-bench Verified, SWE-bench Multilingual, SWE-bench Pro, Terminal-Bench 2.0, GPQA Diamond, AIME 2026
• Evaluation: Mini-SWE-Agent for SWE variants, Terminus-2 for others; fixed subsets (100 instances for SWE variants and GPQA, full sets for Terminal-Bench and AIME)
• Rollout model: Claude Opus 4.8 fixed as external rollout model for generating reasoning traces

---

Empirical Validation / Results

Main Results (Table 2)

Key Findings

1. Iteration improves the candidate frontier in most settings: A later candidate outperforms the first in 14 of 24 settings (58.33%).

2. Feedback-driven improvement is rarely monotonic: Among 23 settings that continue after reaching their best selection score, 18 (78.26%) finish with a final attempted candidate below that peak, and 5 only return to it.

3. Resource efficiency varies dramatically: Median cost is $62.51 (range:$4.80–$363.77) and median time is 6.55 hours (range: 1.14–14.91 hours). High-scoring runs can be inexpensive (e.g., Codex gpt-5.6-sol reaches 65$10.37).

4. Task families do not determine common evolution trajectories: The three SWE-style tasks are won by three different agents, showing strong agent–benchmark interactions.

Reasoning-Effort Diagnostic

A controlled comparison of Claude Code Sonnet-5 on SWE-bench Verified under high vs. max reasoning effort showed:

• Max effort raises mean selection score from 35.5% to 44.0%
• First candidate improves from 25% to 36%
• Historical best moves from 44% to 49%
• Official score increases from 35% to 52%
• Max effort constructs larger datasets (149 training records vs. 50; $401.70 vs.$181.70) but completes one fewer valid attempt, suggesting a depth–breadth tradeoff

Early RSI Experiment

A same-family RSI experiment with Kimi K2.6 as researcher on SWE-bench Pro showed clear strategy evolution (from 8% to 21%) but no checkpoint exceeded the 33% unadapted-model reference, suggesting the current researcher's RSI capability remains insufficient for capable instruction-tuned targets.

---

Theoretical and Practical Implications

Mechanism Analysis: Four Patterns of Stronger Runs

The trajectory analysis identifies four recurring patterns in stronger runs:

1. Accurate hypotheses: Correctly diagnosing the target capability gap
2. Validation-grounded supervision: Grounding data in validation signals rather than assumptions
3. Behavior-aligned data: Matching supervision to evaluated behavior
4. Preservation of strong checkpoints: Historical-best selection protecting earlier discoveries

Implications for Recursive Self-Improvement

• Discovery capability exists but is fragile: Agents can make useful discoveries but cannot yet translate feedback into improvements consistently
• Selection discipline matters: Historical-best checkpoint selection can mitigate regression effects, but effective stopping and strategy revision remain necessary for efficient search
• Cost-performance tradeoffs: Final scores must be interpreted together with search time and training cost; higher cost does not imply stronger final checkpoints

Benchmark Contributions

RSIBench-Data uniquely satisfies all six criteria in Table 1 (executable experience synthesis, reusable data-synthesis policy, capability-gap diagnosis, feedback-driven revision, train/eval data isolation, service-isolated train/serve/eval), unlike all prior benchmarks.

---

Conclusion

RSIBench-Data provides a controlled benchmark for evaluating data-centric research capability required for recursive self-improvement. The main findings reveal a discovery–reliability gap: while agents exhibit core researcher capabilities (improving on first valid attempts in 58.33% of settings), they do not consistently convert feedback into better training-data strategies (78.26% of continued searches end below their peak).

Limitations:

• One representative run per setting (no repeated trials)
• Selection and official evaluation use the same task subset (no held-out generalization measurement)
• Researcher identity bundles LLM, scaffold, and reasoning effort (no component isolation)

Future directions:

• Repeated trials and evaluator-private splits for measuring stability
• Refreshed tasks for testing transferability
• Developing more dependable data-centric researcher agents by making their research process measurable and auditable

The benchmark is open-sourced, providing a foundation for studying and improving the data-centric research capabilities essential for recursive self-improvement.