# Benchmarking Data-Centric Research for Recursive Self-Improvement (RSIBench-Data) > RSIBench-Data isolates data-centric research from training infrastructure, revealing a discovery-reliability gap where agents improve initial attempts but regress in 78% of continued feedback-driven searches. - **Source:** [arXiv](https://arxiv.org/abs/2607.25886) - **Published:** 2026-09-19 - **Permalink:** https://picx.dev/p/jseytu - **Whiteboard:** https://picx.dev/p/jseytu/image ## Summary ## Summary (Overview) - **New Evaluation Target**: RSIBench-Data introduces a controlled benchmark for evaluating LLM agents as *data-centric researchers*—agents that must form hypotheses about model failures, design training-data strategies, and revise them based on checkpoint feedback—while keeping the post-training stack (training, serving, evaluation) fixed. - **Isolation of Research Capability**: Unlike existing benchmarks that entangle research decisions with optimization, serving, and system implementation, RSIBench-Data fixes the infrastructure so that performance differences are attributable to the agent's data-centric research policy. - **Discovery–Reliability Gap**: Agents improve on their first valid attempt in 58.33% of settings, but among searches that continue after reaching their best score, 78.26% finish with a lower-scoring final attempt, revealing that feedback-driven revisions are unreliable. - **Agent–Benchmark Interaction**: No single agent dominates; the three SWE-style tasks are won by three different agents, and best-to-worst spreads range from 8 to 20 percentage points, showing strong task-dependent interactions. - **Open-Source Testbed**: The benchmark is released with code at https://github.com/evolvent-ai/RSIBench-Data, providing a measurable and auditable testbed for studying data-centric research capabilities. --- ## Introduction and Theoretical Foundation ### Background and Motivation Recursive self-improvement requires AI systems to repeatedly convert evidence about model failures into effective model improvements. While LLM agents are increasingly evaluated on long-horizon tasks involving software repositories, terminals, and external tools, these environments expose *evidence about model behavior* rather than *training experience* that can directly improve the model. The central research question is: **Can LLM agents reliably automate data-centric post-training research?** Specifically, an agent must: 1. Formulate a hypothesis about a model failure 2. Turn it into executable and verifiable training data 3. Revise the synthesis strategy from controlled training and evaluation feedback ### Gap in Existing Benchmarks The paper identifies a fundamental tradeoff in existing benchmarks (Table 1): - **Data-centric benchmarks** (DataComp, DataComp-LM, AgoraBench, DCA-Bench) isolate individual data operations but lack closed-loop evolution - **End-to-end post-training benchmarks** (PostTrainBench, Agent² RL-Bench) give broad control but obscure the contribution of data-centric research decisions RSIBench-Data occupies the middle ground: it evaluates whether an LLM agent can act as a data-centric researcher while the surrounding training, serving, and official evaluation stack remains fixed. ### Formal Problem Definition Given a fixed base model $M_0$, benchmark evidence $S$, and resource budget $\mathcal{B}$, a researcher policy $\pi$ observes $S$ and its attempt history $H_{