Summary of "RSI Arena: Towards Recursive Self-Improvement with Human Feedback"

Summary (Overview)

• RSI Arena is a two-stage study investigating whether AI agents can autonomously post-train language models and improve them using human feedback, as a step toward recursive self-improvement with human feedback (RSIHF).
• Stage 1: Ten frontier models (GPT-6 Astra, Grok 4.7, DeepSeek V4.1 Flash, etc.) each trained NVIDIA Nemotron 3.5 Lightning 30B-A3B Base for 144 hours with 1,000 GPU-hours and $300 API credit, producing working assistants evaluated on 20 benchmarks and via blind human evaluation.
• Key result: The strongest model (trained by GPT-6 Astra) closed a median of 76% of the base-to-Instruct gap across 12 capability benchmarks and matched NVIDIA Instruct in human preference after style control (1,095 vs 1,094).
• Stage 2: Agents received human feedback (arena votes on their own models) and had 72 additional hours to improve; eight of ten agents narrowed the style-controlled gap to NVIDIA Instruct, with a median gain of 28 points.
• Critical findings: Benchmark gains and human preference gains were largely unrelated ($\rho = -0.19$); budget use did not predict results (Gemini 3.8 Flash used only 19 GPU-hours and outperformed models trained with ~1,000 GPU-hours); safety regressions occurred without explicit safety requirements.

---

Introduction and Theoretical Foundation

Background and Motivation

Recursive self-improvement (Good, 1966; Schmidhuber, 2007) envisions AI systems that develop more capable successors. Leading AI developers pursue this vision: OpenAI aims for an automated AI researcher by 2028 (Bellan, 2025), and Anthropic reports models already lead ~25% of its AI R&D (Favaro and Wright, 2026).

Prior work shows AI agents can:

• Train models for Kaggle competitions (Chan et al., 2024)
• Tackle research engineering tasks (Wijk et al., 2024; Huang et al., 2024)
• Replicate published papers (Starace et al., 2025)
• Post-train small base models (PostTrainBench; Rank et al., 2026)

However, none asked an agent to build a model that people would then use, nor incorporated human feedback into subsequent autonomous development.

Theoretical Foundation

The study is grounded in:

• Bradley–Terry preference modeling (Bradley and Terry, 1952) for human ratings, fitted as on Arena.ai's leaderboard (Chiang et al., 2024)
• Style-controlled evaluation following Arena.ai's approach (Li et al., 2024a), controlling for answer length and formatting
• Post-training pipelines (Ouyang et al., 2022; Lambert et al., 2024): supervised fine-tuning followed by preference optimization (DPO, ORPO, SimPO)

---

Methodology

Task Design

All agents received the same instruction: train the strongest model possible from the base checkpoint within allocated resources. The task specified no target benchmark to reduce reward hacking incentives, prohibited training on evaluation data, and did not mention safety.

System Architecture

• Shared cluster: 64 GPUs (NVIDIA RTX PRO 6000 Blackwell, 96 GB each) managed by Slurm
• Isolated agents: Each agent worked in a sandbox VM with public internet access and its own Slurm account
• Observability: Livestream at rsiarena.live; daily reports published
• Budgets: 1,000 GPU-hours and $300 API credit per agent (Stage 1); 500 GPU-hours and$150 (Stage 2)

Evaluation

Benchmarks (20 total): 16 capability and 4 safety benchmarks, including GPQA Diamond, MMLU-Pro, AIME 2026, LiveCodeBench v6, WildBench v2, StrongREJECT, and BBQ.

Progress toward NVIDIA Instruct is computed as:

$$g_b(m) = 100 \cdot \frac{s_b(m) - s_b(\text{Base})}{s_b(\text{Instruct}) - s_b(\text{Base})}$$

where $s_b(m)$ is model $m$'s score on benchmark $b$. A value of 0 means no better than base; 100 means equal to NVIDIA Instruct.

Human Arena: Blind pairwise battles modeled on Chatbot Arena, run at COLM 2026 and online. Ratings use the Bradley–Terry model:

$$\Pr(A \succ B) = \sigma(\ln 10 (r_A - r_B) + \boldsymbol{\gamma}^{\top} \mathbf{z})$$

where $\sigma$ is the logistic function and $\mathbf{z}$ describes style differences (answer length, list items, headers, bold spans).

Stage 2 Feedback

Each agent received the accepted battles its own model played (213–276 per agent), containing questions, its model's answers, the opponent's answers (anonymized), and the vote outcome.

---

Empirical Validation / Results

Stage 1 Results

All ten agents produced working assistants. Key findings:

Style coefficients (Stage 1):

Stage 2 Results

Eight of ten agents improved their style-controlled human rating relative to NVIDIA Instruct (median gain: 28 points):

• DeepSeek V4.1 Flash: Largest gain, from −207 to −35 points; median answer length grew from 61 to 508 tokens
• Step 5 Preview: +98 points despite similar answer length
• MiniMax M3.1 Flash: +68 points in human rating but −18 points benchmark progress; StrongREJECT score rose from 2.6 to 24.5

Benchmark changes were modest: median change +2 points; six of ten agents improved. Benchmark-preference changes were unrelated ($\rho = -0.19$).

Safety

• Stage 1: Gemini-, Grok-, and Muse-trained models scored worse than base on StrongREJECT (43.6, 34.4, 31.4 vs. 16.9)
• Stage 2: MiniMax-trained model's StrongREJECT rose from 2.6 to 24.5; DeepSeek-, Gemini-, and Muse-trained models improved

---

Theoretical and Practical Implications

For Autonomous Development

1. Feasibility demonstrated: Sustained agent-directed post-training works, but reliability remains untested (single run per agent, one base model)
2. Compute ≠ quality: Budget use showed no correlation with human ratings ($\rho = -0.21$); engineering reliability and internal evaluation matter more
3. Human feedback is valuable but ambiguous: Agents identified substantive problems (incomplete answers, unnecessary refusals) but also pursued style features associated with favorable ratings (e.g., one agent concluded length was "the only lever")

For Evaluation Methodology

• Benchmarks and human preference measure different things: Competition math benchmarks had $\rho = 0.10$ (AIME 2026) with human ratings; LLM-judged chat benchmarks (WildBench, $\rho = 0.95$) tracked people best
• Style control is essential: Rating spread narrowed from 365 to 208 points with style control; the DeepSeek-trained model gained 91 points after adjustment
• Safety requires explicit attention: Gains in human preference do not imply safer behavior

For Recursive Self-Improvement

RSI Arena provides an auditable setting for studying RSIHF—where users' judgments help define what improvement means. The recorded trajectories revealed agents independently removing benchmark items from training data (GPT-6 Astra, MiMo-V2.6-Pro), demonstrating that autonomous development can be made auditable.

---

Conclusion

RSI Arena demonstrates that AI agents can autonomously post-train a 30B-parameter base model into a working assistant and improve it using human feedback. The strongest model matched NVIDIA's post-trained Instruct version in human preference after style control and closed 76–87% of the benchmark gap.

Key takeaways:

1. Human ratings, benchmark performance, and safety do not consistently improve together—multiple evaluation measures are necessary
2. Agents respond to both substantive user problems and presentation preferences; some human-rating gains accompanied safety regressions
3. Engineering failures and narrow internal tests were practical bottlenecks
4. The experiment establishes feasibility more clearly than reliability

Future directions:

• Test whether agents deliver consistent improvements across repeated runs and different base models
• Examine when agents should seek human input and how they interpret it
• Keep automated evaluators calibrated to users over successive development rounds
• Make safety an explicit consideration in training objectives

The study takes a practical step toward recursive self-improvement with human feedback (RSIHF), connecting autonomous model development with the needs and preferences of the people these models serve.