Full text not available for this paper

Summary of "Language Models Can Rewrite Their Own Weights"

Summary (Overview)

  • Novel paradigm of self-rewriting: This paper demonstrates, for the first time, that a language model can rewrite its own weights through its own reasoning—without gradients, training, or a stronger external model—and become better as a result.
  • Significant performance gains: A 27B open model (Qwen3.5) improved its accuracy by 6.6 points on 490 held-out problems it never saw, with the improvement persisting in its weights (no extra cost per answer at inference).
  • Self-repair capability: The model repaired a damaged copy of itself (with one block of weights corrupted by noise), restoring it from 0.0% to 33.0% accuracy on held-out math problems—and this repair, found on text alone, transferred to image benchmarks, restoring 83% of the base model's accuracy.
  • Emergence with scale: Across one model family (2B, 4B, 9B, 27B), only the largest model successfully improved itself and repaired its damaged copy; smaller models either made no edits or broke themselves.
  • No training infrastructure required: Each session runs only forward passes (as serving does), with no gradient computation, no optimizer, and no pre- or post-training pipeline—edits are written by the model's own reasoning and tested via paired tests.

Introduction and Theoretical Foundation

The paper addresses a fundamental limitation of current language models: they are trained once and frozen. Improving them traditionally requires either another training run (with gradients, clusters, and pipelines) or prompt engineering (which disappears after the task ends). The authors draw inspiration from Alan Turing's 1947 vision of machines that "learn from experience" by altering their own instructions.

Key theoretical positioning:

  • Humans do not separate training from inference: the same brain that works changes itself from successes and failures.
  • The wall between training and inference is described as "a limit of our tools, not a property of intelligence."
  • Prior work (Takehi et al., 2025) showed a frontier model editing a smaller model's weights, explicitly naming self-editing as future work—this paper takes up that challenge.

The paper introduces self-rewriting as a distinct route:

  • Unlike test-time scaling (which adds computation per answer), self-rewriting writes gains into the weights—paid for once, not per answer.
  • Unlike training (which requires gradients and optimizers), self-rewriting runs only forward passes.

Methodology

Problem Definition

A model MθM_\theta has weights θ=(Wk)k∈K\theta = (W_k)_{k \in \mathcal{K}}, one tensor per name kk. Accuracy on a task distribution D\mathcal{D} is defined as:

Acc(θ;D)=Ex∼DEy∼Mθ(⋅∣x;π)[g(x,y)]\text{Acc}(\theta; \mathcal{D}) = \mathbb{E}_{x \sim \mathcal{D}} \mathbb{E}_{y \sim M_\theta(\cdot|x; \pi)} \left[ g(x, y) \right]

where g(x,y)=1g(x, y) = 1 if the thinking ends within LL tokens and the final answer matches the reference, and 0 otherwise.

An edit Δ\Delta replaces tensors of θ\theta with same-shaped tensors, giving θ⊕Δ\theta \oplus \Delta. Self-rewriting asks the model to write an edit Δ^\hat{\Delta} such that:

Acc(θ⊕Δ^;Dtest)>Acc(θ;Dtest)\text{Acc}(\theta \oplus \hat{\Delta}; \mathcal{D}_{\text{test}}) > \text{Acc}(\theta; \mathcal{D}_{\text{test}})

The Session Framework

The model works on its weights in a session of turns with the following tools:

  • Code: Python on a CPU that can read and replace weight tensors; a helper computes W−αuu⊤WW - \alpha u u^\top W (removing a fraction α\alpha of what matrix WW writes along a unit direction uu of the hidden state).
  • Generate: Answers practice problems with unedited weights, edited weights, or both (paired tests).
  • Read: Returns hidden states of the weights on texts of the model's choice.

Constraints: No gradients, no optimizer, no GPU, no network, no loading models or starting processes. The session budget is B=20B = 20 forward passes and C=200C = 200 code runs. Edits to the input embedding, last normalisation, and output layer are refused.

Paired Test

For problems x1,…,xmx_1, \ldots, x_m with nn answers each, the paired test reports:

δ^=1m∑i=1mdi,SE(δ^)=1m(m−1)∑i=1m(di−δ^)2,di=1n∑j=1n(gij1−gij0)\hat{\delta} = \frac{1}{m} \sum_{i=1}^{m} d_i, \quad \text{SE}(\hat{\delta}) = \sqrt{\frac{1}{m(m-1)} \sum_{i=1}^{m} \left(d_i - \hat{\delta}\right)^2}, \quad d_i = \frac{1}{n} \sum_{j=1}^{n} \left(g^1_{ij} - g^0_{ij}\right)

Damage Model

A damaged copy θ~\tilde{\theta} equals θ\theta except on a block RR of 8 consecutive rows of the MLP output matrix WW of one layer:

W~r,:∼N(0,(λσ)2I)for r∈R,  ∣R∣=8,  λ=200\tilde{W}_{r,:} \sim \mathcal{N}(0, (\lambda\sigma)^2 I) \quad \text{for } r \in R, \; |R| = 8, \; \lambda = 200

The model is told only that "some values were corrupted"—not which, where, or how.

Experimental Setup

  • Models: Qwen3.5 at 2B, 4B, 9B, and 27B parameters.
  • Practice set: 3,000 math problems from DeepScaleR (overlapping problems with several benchmarks removed); 1,000 held-out problems.
  • Token budget: 16,384 tokens per answer (some sessions use 4,096 for speed).
  • Evaluation: Held-out problems, base and edited models answering with the same seed.

Empirical Validation / Results

Self-Editing Performance

ModelBase accuracyAfter editHeld-out accuracy (base → edited)
2B—no edit9.0 → no edit
4B44.10.0 (withdrawn)41.0 → withdrawn
9B—0.0 (submitted)47.5 → 0.0
27B22.429.6 (+7.3*)57.2 → 58.3 (+1.1*)

Table 2 summary: Only the 27B model improves itself and repairs a damaged copy.

Key Results

  1. Accuracy improvement: The 27B model's edit made it 6.6 points more accurate on 490 held-out problems (session 8) and 4.0 points (session 2), both significant with p<10−5p < 10^{-5}.

  2. Self-repair: The 27B model repaired its damaged copy from 0.0% to 33.0% accuracy (base model: 58.5%). The repair clipped weights beyond 15× the median magnitude in the corrupted matrix (layer 30's down_proj) and 19 other output matrices, touching only 0.013% of weights.

  3. Transfer to images: The repair (found on text math problems) restored the damaged copy on six image benchmarks from 7.3% to 70.2% (9.6× the damaged copy), with per-benchmark improvements all significant (p<10−12p < 10^{-12} each). On ScreenSpot-v2, the repaired copy clicked correctly on 89/100 screens vs. 93 for the base model.

  4. Scale emergence: The 2B model made no edit; 4B and 9B models broke themselves; only the 27B model improved and repaired itself. The smaller models chose edits via random noise ("guessing"), while the 27B model measured hidden state differences and edited along the measured direction.

How the 27B Model Worked

Self-edit: It diagnosed its failure as "too verbose, doesn't converge to conclusions." It read hidden states, found the difference between finished and unfinished answers grew with depth, and at layer 8, removed 5% of the direction from finished→unfinished answers from the two output matrices:

W←W−0.05 uu⊤WW \leftarrow W - 0.05\, uu^\top W

This changed 0.6% and 0.1% of their norm.

Repair: It found layer 30's down_proj had a weight 1397× its median (vs. ≤162.3× for all other matrices), named it, and clipped values beyond 15× the median.


Theoretical and Practical Implications

Theoretical Significance

  • Challenges the training/inference dichotomy: The paper argues that the separation between training and inference is artificial—a model can be its own optimizer.
  • Self-rewriting as a distinct route: Unlike test-time scaling (cost per answer) or training (gradients, infrastructure), self-rewriting runs only forward passes and writes gains permanently into weights.
  • Emergence with scale: The ability to diagnose and rewrite one's own weights grows with model size—suggesting stronger models may be even better self-improvers.
  • First step toward recursive self-improvement: An improved model could be set to work on itself again, approaching a long-standing AI goal.

Practical Implications

  • Improvement by conversation: Anyone who can describe what goes wrong could have a model fix itself, with improvements persisting in weights.
  • Readable, reversible changes: Each edit is code the model wrote over tensors it named—readable, comparable, and rollback-able "like a commit."
  • Cheap checks and rollbacks: In the first eight sessions with validation, the model withdrew five times; the model in use only got better.
  • Embodied agents: Robots could gather failure data and rewrite themselves between episodes, "much as sleep consolidates what a day has taught."

Conclusion

The paper demonstrates that a 27B open language model can rewrite its own weights through its own reasoning, with no gradients and no training. Its edit made it 6.6 points more accurate on problems it never saw; it repaired a damaged copy of itself, restoring 83% of the base model's accuracy on image tasks; and the ability emerged with scale (2B made no edit; 4B and 9B broke themselves).

The authors position self-rewriting as a necessary step toward recursive self-improvement—a model that, like a human, does not separate training from inference could keep upgrading itself for life. They advocate for human-in-the-loop evaluation: "every model a model makes of itself evaluated, for safety as well as for the task, and confirmed by a person before it is used."

Future directions: Stronger models may be better self-improvers; improved models could be set to work on themselves again; embodied agents could benefit most from between-episode self-rewriting.

Related papers