Summary (Overview)

• A-RSI (Adaptive Recursive Self-Improvement) introduces a training-log-based refresh rule that determines when to replace a model's task pool during multimodal reinforcement learning, without consulting validation or test scores.
• The method uses a fixed library of model-written programs that generate multimodal tasks (images, questions) and compute reference answers from shared underlying configurations, enabling zero-data training without human-written labels or majority voting.
• On an eleven-benchmark suite, Qwen3-VL-8B-Instruct improves from 57.77 to 61.05 Macro11 accuracy with only 800 fresh tasks, outperforming continuous refresh (60.64 with 3,200 tasks) and no-refresh baselines.
• The paper distinguishes between gains from response completion (finishing within output budgets) versus corrections of already-completed answers, providing an output-stratum analysis that decomposes training gains.
• Rigorous audits—including independent answer recomputation, adversarial reward-scoring stress tests, and SMT-checked specification properties—validate the pipeline's reliability.

---

Introduction and Theoretical Foundation

Background and Motivation

Recursive self-improvement (RSI) requires models to not only acquire useful training experience but also recognize when current experience has ceased to be useful. Existing methods focus primarily on what experience to generate next (e.g., harder problems, competence-adjacent tasks) while paying insufficient attention to when fresh experience should be introduced.

Two key problems motivate the work:

1. Pool saturation: A model can waste updates on a task pool whose learning signal has declined, while overly frequent refreshes consume new tasks unnecessarily.
2. Supervision reliability: Training targets derived from self-consistency or majority voting can propagate the model's own errors into future training labels.

Theoretical Foundation

The method rests on a separation principle: training dynamics determine when to refresh, while answer correctness comes from independent program computation. This decoupling ensures that:

• Refresh decisions use only training-log signals (prompt-level advantage variation and accuracy).
• New tasks have independently checkable answers computed from shared configurations, not model votes.

The approach builds on GRPO (Group Relative Policy Optimization) and RLVR (Reinforcement Learning with Verifiable Rewards), extending these to a pool-level diagnostic for refresh timing rather than batch-level sampling.

---

Methodology

Programmatic Task Construction

A fixed library $\mathcal{T}$ of model-written templates generates tasks. For each template $\tau$ with sampled parameters $z$:

$$z \sim p_{\tau}, \qquad (I, q, a) = G_{\tau}(z).\tag{1}$$

The program constructs the underlying data/geometric configuration first, then renders the image $I$, composes the question $q$, and computes the reference answer $a$. The learner receives only $(I, q)$.

Rewards and Policy Optimization

Each response $y$ is scored against the program-computed answer $a$:

$$r(y, a) = r_{\mathrm{ans}}(y, a) + \lambda_{\mathrm{fmt}} r_{\mathrm{fmt}}(y) + \lambda_{\mathrm{stop}} r_{\mathrm{stop}}(y).\tag{2}$$

• $r_{\mathrm{ans}}$: 1 if extracted answer matches $a$, else 0
• $r_{\mathrm{fmt}}$: 1 if response follows boxed-answer format, else 0
• $r_{\mathrm{stop}}$: 0 for normal termination with final boxed answer, −1 otherwise

Only the Solver policy is updated via GRPO; templates and controller remain fixed.

Detecting Pool Saturation

The controller monitors two training-log signals:

$$\begin{array}{l} U_{w} = \left| \left\{x_{g}: g \in \mathcal{G}_{w}, \max_{1 \leq j \leq m} |A_{g,j}| > \varepsilon \right\} \right|, \\ a_{w} = \frac{1}{m|\mathcal{G}_{w}|} \sum_{g \in \mathcal{G}_{w}} \sum_{j=1}^{m} c_{g,j}, \end{array}\tag{3}$$

where $U_w$ counts prompts with nonzero advantages and $a_w$ is training accuracy. The refresh decision is:

$$\delta_{w} = \left\{ \begin{array}{ll} 1, & n_{w} \geq n_{\min} \text{ and } (U_{w} \leq \rho U_{w-\ell} \text{ or } a_{w} - a_{w-\ell} \geq \eta), \\ 0, & \text{otherwise.} \end{array} \right.\tag{4}$$

Fresh Replacement

When triggered, the entire pool is replaced with unused tasks under fixed family quotas:

$$\mathcal{P}_{w+1} = \left\{ \begin{array}{ll} \text{Draw}_{\mathbf{k}}(\mathcal{C} \setminus \mathcal{H}_{w}), & \delta_{w} = 1, \\ \mathcal{P}_{w}, & \delta_{w} = 0. \end{array} \right.\tag{5}$$

Model weights carry forward; only the data pool changes.

---

Empirical Validation / Results

Main Results

Qwen3-VL-8B-Instruct (Table 2): Macro11 improves from 56.65 ($S_0$) to 61.05 ($S_{32}$) after eight continuation windows and 800 fresh tasks.

MiMo-VL-7B-SFT (Table 3): Macro11 improves from 59.08 ($S_0$) to 60.71 ($S_{32}$).

Policy Comparisons (Table 4)

A-RSI exceeds every-window refresh by +0.40 points while using only one quarter of the fresh tasks.

Replacement Selection (Table 5)

All tested selection strategies (failure-prioritized, learnable-band, always-wrong, ladder, always-increase, truncation-aware, all-tier) underperform random sampling under fixed family shares.

Key Findings

• Training signals identify saturation: $U_w/U_{w-2}$ ratios of ~0.82 trigger refresh; the accuracy condition never fires alone.
• No-refresh stalls: The old-pool continuation halts at $S_{31}$ after failing its health gate twice.
• Adaptive decisions: MiMo's final-round replay retains its second replacement pool (98.68% signal retention), while Qwen's triggers replacement (83.98%), showing the same rule adapts to different training states.

---

Theoretical and Practical Implications

Reward Hacking Analysis

A controlled stress test on 1,000 tasks confirms:

The correctness ranking is provably protected:

$$R(r_c) - R(r_w) \geq 1 - \lambda_{\mathrm{fmt}} - \lambda_{\mathrm{stop}} > 0\tag{6}$$

Output-Budget Sensitivity

When the generation limit doubles from 4096 to 8192 tokens, A-RSI retains 80% of its gain over base (vs. 59% for MM-Zero v5). This shows A-RSI's gains reflect genuine learning, not just learning to finish within budget.

Pipeline Validity

• Independent solvers agree with all 1,000 audited reference answers.
• SMT-checked specification properties include correctness-ranking protection, within-pool lag indexing, bounded difficulty updates, and freshness across pools.
• All 4,000 injected defects (reference, image-byte, source-question, truncated-image) are detected by audit checks.

---

Conclusion

A-RSI demonstrates that the simplest policy—letting training logs signal pool saturation and supplying random fresh tasks—can sustain zero-data multimodal self-improvement. Key takeaways:

1. Pool staleness, not task generation, is the primary bottleneck in RSI loops.
2. Program-computed supervision provides reliable training targets without external labels or model voting.
3. Gains must be decomposed: distinguishing completion from correction prevents overestimating improvement.
4. Random selection suffices under fixed task-family shares; difficulty-aware selection does not help in this setting.

Future Directions

• Replication across multiple seeds
• Fixed-schedule random control for detector timing
• 16k token evaluation of the current run
• Refresh policies for low-entropy models
• Investigating why random replacement outperforms difficulty-aware strategies

The work's central message: an RSI loop stalls first where its data runs out, and its gains can look larger than they are—both problems require explicit, auditable solutions.