Full text not available for this paper
Summary of "Language Models Can Rewrite Their Own Weights"
Summary (Overview)
- Novel paradigm of self-rewriting: This paper demonstrates, for the first time, that a language model can rewrite its own weights through its own reasoning—without gradients, training, or a stronger external model—and become better as a result.
- Significant performance gains: A 27B open model (Qwen3.5) improved its accuracy by 6.6 points on 490 held-out problems it never saw, with the improvement persisting in its weights (no extra cost per answer at inference).
- Self-repair capability: The model repaired a damaged copy of itself (with one block of weights corrupted by noise), restoring it from 0.0% to 33.0% accuracy on held-out math problems—and this repair, found on text alone, transferred to image benchmarks, restoring 83% of the base model's accuracy.
- Emergence with scale: Across one model family (2B, 4B, 9B, 27B), only the largest model successfully improved itself and repaired its damaged copy; smaller models either made no edits or broke themselves.
- No training infrastructure required: Each session runs only forward passes (as serving does), with no gradient computation, no optimizer, and no pre- or post-training pipeline—edits are written by the model's own reasoning and tested via paired tests.
Introduction and Theoretical Foundation
The paper addresses a fundamental limitation of current language models: they are trained once and frozen. Improving them traditionally requires either another training run (with gradients, clusters, and pipelines) or prompt engineering (which disappears after the task ends). The authors draw inspiration from Alan Turing's 1947 vision of machines that "learn from experience" by altering their own instructions.
Key theoretical positioning:
- Humans do not separate training from inference: the same brain that works changes itself from successes and failures.
- The wall between training and inference is described as "a limit of our tools, not a property of intelligence."
- Prior work (Takehi et al., 2025) showed a frontier model editing a smaller model's weights, explicitly naming self-editing as future work—this paper takes up that challenge.
The paper introduces self-rewriting as a distinct route:
- Unlike test-time scaling (which adds computation per answer), self-rewriting writes gains into the weights—paid for once, not per answer.
- Unlike training (which requires gradients and optimizers), self-rewriting runs only forward passes.
Methodology
Problem Definition
A model has weights , one tensor per name . Accuracy on a task distribution is defined as:
where if the thinking ends within tokens and the final answer matches the reference, and 0 otherwise.
An edit replaces tensors of with same-shaped tensors, giving . Self-rewriting asks the model to write an edit such that:
The Session Framework
The model works on its weights in a session of turns with the following tools:
- Code: Python on a CPU that can read and replace weight tensors; a helper computes (removing a fraction of what matrix writes along a unit direction of the hidden state).
- Generate: Answers practice problems with unedited weights, edited weights, or both (paired tests).
- Read: Returns hidden states of the weights on texts of the model's choice.
Constraints: No gradients, no optimizer, no GPU, no network, no loading models or starting processes. The session budget is forward passes and code runs. Edits to the input embedding, last normalisation, and output layer are refused.
Paired Test
For problems with answers each, the paired test reports:
Damage Model
A damaged copy equals except on a block of 8 consecutive rows of the MLP output matrix of one layer:
The model is told only that "some values were corrupted"—not which, where, or how.
Experimental Setup
- Models: Qwen3.5 at 2B, 4B, 9B, and 27B parameters.
- Practice set: 3,000 math problems from DeepScaleR (overlapping problems with several benchmarks removed); 1,000 held-out problems.
- Token budget: 16,384 tokens per answer (some sessions use 4,096 for speed).
- Evaluation: Held-out problems, base and edited models answering with the same seed.
Empirical Validation / Results
Self-Editing Performance
| Model | Base accuracy | After edit | Held-out accuracy (base → edited) |
|---|---|---|---|
| 2B | — | no edit | 9.0 → no edit |
| 4B | 44.1 | 0.0 (withdrawn) | 41.0 → withdrawn |
| 9B | — | 0.0 (submitted) | 47.5 → 0.0 |
| 27B | 22.4 | 29.6 (+7.3*) | 57.2 → 58.3 (+1.1*) |
Table 2 summary: Only the 27B model improves itself and repairs a damaged copy.
Key Results
-
Accuracy improvement: The 27B model's edit made it 6.6 points more accurate on 490 held-out problems (session 8) and 4.0 points (session 2), both significant with .
-
Self-repair: The 27B model repaired its damaged copy from 0.0% to 33.0% accuracy (base model: 58.5%). The repair clipped weights beyond 15× the median magnitude in the corrupted matrix (layer 30's down_proj) and 19 other output matrices, touching only 0.013% of weights.
-
Transfer to images: The repair (found on text math problems) restored the damaged copy on six image benchmarks from 7.3% to 70.2% (9.6× the damaged copy), with per-benchmark improvements all significant ( each). On ScreenSpot-v2, the repaired copy clicked correctly on 89/100 screens vs. 93 for the base model.
-
Scale emergence: The 2B model made no edit; 4B and 9B models broke themselves; only the 27B model improved and repaired itself. The smaller models chose edits via random noise ("guessing"), while the 27B model measured hidden state differences and edited along the measured direction.
How the 27B Model Worked
Self-edit: It diagnosed its failure as "too verbose, doesn't converge to conclusions." It read hidden states, found the difference between finished and unfinished answers grew with depth, and at layer 8, removed 5% of the direction from finished→unfinished answers from the two output matrices:
This changed 0.6% and 0.1% of their norm.
Repair: It found layer 30's down_proj had a weight 1397× its median (vs. ≤162.3× for all other matrices), named it, and clipped values beyond 15× the median.
Theoretical and Practical Implications
Theoretical Significance
- Challenges the training/inference dichotomy: The paper argues that the separation between training and inference is artificial—a model can be its own optimizer.
- Self-rewriting as a distinct route: Unlike test-time scaling (cost per answer) or training (gradients, infrastructure), self-rewriting runs only forward passes and writes gains permanently into weights.
- Emergence with scale: The ability to diagnose and rewrite one's own weights grows with model size—suggesting stronger models may be even better self-improvers.
- First step toward recursive self-improvement: An improved model could be set to work on itself again, approaching a long-standing AI goal.
Practical Implications
- Improvement by conversation: Anyone who can describe what goes wrong could have a model fix itself, with improvements persisting in weights.
- Readable, reversible changes: Each edit is code the model wrote over tensors it named—readable, comparable, and rollback-able "like a commit."
- Cheap checks and rollbacks: In the first eight sessions with validation, the model withdrew five times; the model in use only got better.
- Embodied agents: Robots could gather failure data and rewrite themselves between episodes, "much as sleep consolidates what a day has taught."
Conclusion
The paper demonstrates that a 27B open language model can rewrite its own weights through its own reasoning, with no gradients and no training. Its edit made it 6.6 points more accurate on problems it never saw; it repaired a damaged copy of itself, restoring 83% of the base model's accuracy on image tasks; and the ability emerged with scale (2B made no edit; 4B and 9B broke themselves).
The authors position self-rewriting as a necessary step toward recursive self-improvement—a model that, like a human, does not separate training from inference could keep upgrading itself for life. They advocate for human-in-the-loop evaluation: "every model a model makes of itself evaluated, for safety as well as for the task, and confirmed by a person before it is used."
Future directions: Stronger models may be better self-improvers; improved models could be set to work on themselves again; embodied agents could benefit most from between-episode self-rewriting.
Related papers
- The Critic Can Hurt: On the Safety of Critics in LLM-Guided AutoML Search
A naive LLM critic can drive AutoML search below the no-critic random-search floor on 7 of 8 tabular datasets, with losses up to 0.245 AUC.
- From Repositories to Training Environments: Scaling Long-Horizon Coding Agents with Environment Evolution Loop
RepoFactory's proxy-driven environment evolution loop yields RepoHorizen-35B-A3B, a model improving coding-agent benchmarks by 22.6 points on NL2Repo and 8.5 points on average.
- RIPPLE: Recovery Internalization through Process-Level Policy Learning and Effect-Aware Credit for Long-Horizon Agentic Coding
RIPPLE internalizes prefix-only recovery guidance into a single policy via stage-specific credit, improving coding agents by 4.8 points across four benchmarks without deployment-time critics.