Full text not available for this paper

BaseRSI: Recursive Self-Improvement of a Base Vision-Language Model via Training-Free Harness Internalization

Summary (Overview)

  • New paradigm: BaseRSI introduces "Recursive Harness Internalization," a framework where a base vision-language model (VLM) improves its own weights by internalizing gains from its harness (prompts/tools) through training-free edits, round after round.
  • Two novel mechanisms: ReGap supervises every transformer block with high-dimensional activation differences (teacher–student gaps) and ReKey writes these gaps into a few key–value neurons per block via regression, with measured site and strength.
  • Key results: On ten visual reasoning benchmarks (5 in-distribution, 5 held-out), BaseRSI improves the base model's average accuracy by +11.7 points (ID) and +6.1 points (OOD), outperforming prompting, visual zoom-in, harness evolution, and fine-tuning baselines.
  • Efficiency: The entire internalization process uses only forward passes and linear solves—no backward passes or API calls—requiring just 45.5 GPU hours, the lowest cost among all compared methods.
  • Composability: Edits written at different sites compose in a single checkpoint without refitting, and the edited model answers original requests alone, exceeding even the harness-in-the-request performance.

Introduction and Theoretical Foundation

Background and Motivation

Recursive self-improvement (RSI) systems have traditionally improved the harness—prompts, tools, and context—around a frozen base model. However, three key challenges persist:

  1. Harness-bound gains (Figure 2a): Improvements from harness evolution live in the request, costing context on every request and disappearing when the harness is removed.
  2. Black-box supervision (Figure 2b): Co-evolution and distillation supervise weights only at the model's output, so the signal must travel back through every block via gradient training—slow and prone to internalizing noise.
  3. Imprecise edits (Figure 2c): Weight editing methods preset the site and strength of edits, but localization does not reveal where an edit works best, leading to missed targets and harmed general abilities.

Theoretical Foundation

Definition 1 (Base model and harness): A base model fθf_\theta has weights θ\theta and LL residual blocks, each with attention and SwiGLU MLP. A harness H={c1,…,cm}\mathcal{H} = \{c_1, \ldots, c_m\} wraps the model with entries c=(κc,ac)c = (\kappa_c, a_c), where scope κc(q)∈{0,1}\kappa_c(q) \in \{0,1\} determines applicability and action aca_c rewrites the request.

Definition 2 (Response trajectory): The model responds with token sequence y1:Ty_{1:T} where each token is drawn conditioned on the harness-rewritten request:

yt∼pθ(⋅∣H(q),y<t),H(q)=(cm∘⋯∘c1)(q)y_t \sim p_\theta\left(\cdot \mid \mathcal{H}(q), y_{<t}\right), \quad \mathcal{H}(q) = (c_m \circ \cdots \circ c_1)(q)

Problem statement: Given base weights θ0\theta_0, empty harness H0\mathcal{H}_0, development requests QQ, and edit family E\mathcal{E}, seek:

max⁡e0,…,eN−1∈EE(q,y⋆)∼T[1{fθN(q)=y⋆}]\max_{e_0,\ldots,e_{N-1} \in \mathcal{E}} \mathbb{E}_{(q,y^\star) \sim \mathcal{T}} \left[\mathbb{1}\{f_{\theta_N}(q) = y^\star\}\right]

where θn+1=θn⊕en\theta_{n+1} = \theta_n \oplus e_n and only the weights enter the deployed objective.


Methodology

4.1 Recursive Harness Internalization

The paradigm treats the harness as a fast system recording new experience and weights as a slow system consolidating it through replay.

Outer loop (harness evolution): The base model proposes entries from its own failures; an acceptance test keeps those whose gain generalizes:

An={c:pgen(c)=max⁡b∈Bp({db′(c)}b′≠b)≤αc}\mathcal{A}_n = \left\{c : p_{\text{gen}}(c) = \max_{b \in \mathcal{B}} p\left(\{d_{b'}(c)\}_{b' \neq b}\right) \leq \alpha_c\right\}

where db(c)d_b(c) is the paired accuracy gain on benchmark bb, p(⋅)p(\cdot) is the one-sided sign-flip pp-value, and αc\alpha_c is the Holm threshold.

Inner loop (internalization): The model with entry cc is the teacher πθ0c=pθ0(⋅∣c(q),y<t)\pi^c_{\theta_0} = p_{\theta_0}(\cdot \mid c(q), y_{<t}); the model on the original request is the student πθ0=pθ0(⋅∣q,y<t)\pi_{\theta_0} = p_{\theta_0}(\cdot \mid q, y_{<t}):

en∈arg⁡min⁡e∈EEq∼QEy∼πe(⋅∣q)KL(πθ0c∥πe)e_n \in \arg\min_{e \in \mathcal{E}} \mathbb{E}_{q \sim Q} \mathbb{E}_{y \sim \pi_e(\cdot|q)} \text{KL}\left(\pi^c_{\theta_0} \| \pi_e\right)

Rounds on own responses: Edits are refitted in rounds on the student's own responses to address covariate shift:

D(j)=D(j−1)∪D(y),y∼πe(j−1)(⋅∣q)\mathcal{D}^{(j)} = \mathcal{D}^{(j-1)} \cup \mathcal{D}(y), \quad y \sim \pi_{e^{(j-1)}}(\cdot \mid q)

4.2 ReGap: Per-Block Supervision

High-dimensional signal: The gap of block ll is the difference the entry makes at that block alone:

gl=Fl(sl−1c;zl−1c)−Fl(sl−1c;zl−1)g_l = F_l\left(s^c_{l-1}; z^c_{l-1}\right) - F_l\left(s^c_{l-1}; z_{l-1}\right)

where superscript cc marks teacher states. This gap is the exact descent direction of the distance between block ll and its teacher.

Low-dimensional update: The edit is a set of key–value neurons:

El(x)=γ ωβ(kl⊤x)∑i=0r(ul,i⊤x)vl,iE_l(x) = \gamma \, \omega_\beta\left(k_l^\top x\right) \sum_{i=0}^{r} \left(u_{l,i}^\top x\right) v_{l,i}

where ωβ(z)=zσ(βz)\omega_\beta(z) = z\sigma(\beta z) is the key's scope function, γ\gamma is the step, and r≪Hr \ll H (width).

Proposition 2: Adding every block's gap to the student's activations makes its output identical to the teacher's.

4.3 ReKey: Measured Site and Strength

Site measurement:

η(c)=1−KL(πθc∥πθlin←c)KL(πθc∥πθ)\eta(c) = 1 - \frac{\text{KL}\left(\pi^c_\theta \| \pi^{\text{lin} \leftarrow c}_\theta\right)}{\text{KL}\left(\pi^c_\theta \| \pi_\theta\right)}

where πθlin←c\pi^{\text{lin} \leftarrow c}_\theta is the student with teacher's linear-attention outputs patched in. High η\eta → write to MLP neurons; low η\eta → write to patch embedding (resampling kernel K′=B⊤KK' = B^\top K).

Writing (ridge regression):

W^l=arg⁡min⁡WEDl∥g−bl−W⊤Plx∥2+λ∥W∥F2\hat{W}_l = \arg\min_W \mathbb{E}_{\mathcal{D}_l} \left\| g - b_l - W^\top P_l x \right\|^2 + \lambda \|W\|^2_F

reduced to the top-rr directions, each filling one key–value neuron.

Strength (line search):

γj=arg⁡min⁡γ∈Γδj(γ),δj(γ)=KL(πθc∥πγe(j))\gamma_j = \arg\min_{\gamma \in \Gamma} \delta_j(\gamma), \quad \delta_j(\gamma) = \text{KL}\left(\pi^c_\theta \| \pi_{\gamma e^{(j)}}\right)

Empirical Validation / Results

Main Results (RQ1, RQ2)

MethodID Avg.OOD Avg.
Qwen3.8-27B (base)60.464.5
CoT62.467.1
Self-Refine62.268.4
Meta-Harness63.167.7
RRSI65.168.3
SFT60.965.1
GRPO61.263.9
BaseRSI (ours)72.170.7

BaseRSI achieves the highest mean accuracy on all ten benchmarks, with gains of +21.6 on BabyVision, +16.4 on VLMs Are Biased, and +10.4 on VisuLogic.

Key Findings

  • Harness vs. weights (RQ3): The edited base model on original requests exceeds the same model with its harness in the request on all ten benchmarks (+2.5 points ID, +2.3 OOD).
  • Rounds matter: A one-shot edit fitted on teacher's responses loses effect on student's own outputs (restoration drops from 0.652 to 0.504); three rounds of refitting on own responses suffice.
  • ReGap vs. distillation (RQ4): Output-only distillation gains less than half of ReGap's gain (ID: 0.9 vs. 4.1 points; OOD: −0.9 vs. 1.9 points).
  • Measured choices (RQ5): Preset site (view in MLP) loses 7.2 points in-scope; preset strength loses 4.3 points; preset layers lose 4.6 points.
  • Cross-model generalization: BaseRSI improves all ten benchmarks on Gemma-4-31B (+11.4 ID, +5.3 OOD), Qwen3.6-27B, and Qwen3.5-9B.

Fine-Grained Ablations

  • Signal quality: A unit step along the gap removes divergence to 0.0003 (320× below no-edit); negated, shuffled, or other-block gaps leave it.
  • Blocks written: All 64 blocks contribute; the last 32 blocks carry most of the effect, with the 7 largest single blocks all being full-attention.
  • Neurons: Fitted readouts on least-active neurons work best; random directions restore only 0.250 vs. 0.593 for fitted ones; most-active neurons harm performance (−2.119 restoration).

Theoretical and Practical Implications

Theoretical Contributions

  1. Per-block supervision replaces output-only supervision: ReGap demonstrates that high-dimensional activation gaps provide a more precise supervision signal than output-level losses, with Proposition 1 guaranteeing convergence of the round-based fitting.

  2. Training-free weight editing as recursive self-improvement: The framework shows that weight edits can be computed entirely from forward passes (Proposition 2, 3), making RSI feasible without gradient computation.

  3. Measured site and strength: ReKey formalizes that edit placement (site) and magnitude (strength) should be measured from the model's own responses rather than preset, addressing the imprecision of prior weight-editing methods.

Practical Implications

  • Cost efficiency: BaseRSI requires 45.5 GPU hours—the least of all compared methods—with no API calls, making it accessible for modest compute budgets.
  • Composability: Edits fitted independently on base weights compose in one checkpoint without refitting, enabling incremental accumulation of improvements.
  • Deployment: The edited model answers original requests in a single call on standard inference engines, eliminating per-request harness overhead (34% fewer tokens than harness-in-request).

Conclusion

BaseRSI unifies harness evolution, weight editing, and internalization into one recursive loop, enabling a base VLM to improve its own weights through training-free edits. Key takeaways:

  1. The harness-to-weights pipeline works: Verified harness gains can be written into the model's weights via per-block gaps, with the edited model exceeding the harness itself.
  2. Internal supervision beats output supervision: Per-block gaps carry substantially more information than output-level distillation.
  3. Measured edits outperform preset ones: Site, strength, and key all benefit from being measured on the model's own responses.

Future directions include extending to larger models, more complex harness entries, and exploring whether the recursive loop can discover entirely new capabilities rather than just consolidating harness gains. The authors note that Claude (Anthropic) was used for writing polish and LaTeX typesetting.

Related papers