# Self-Improvement Can Self-Regress: The Rise-and-Collapse Failure Mode of LLM Self-Training

> LLM self-training for code shows a rise-then-collapse failure mode where pass@1 peaks then crashes within one campaign, and no intervention fully prevents it.

- **Source:** [arXiv](https://arxiv.org/abs/2606.21090)
- **Published:** 2026-08-18
- **Permalink:** https://picx.dev/p/e8QzQK
- **Whiteboard:** https://picx.dev/p/e8QzQK/image

## Summary

# Summary of "Self-Improvement Can Self-Regress: The Rise-and-Collapse Failure Mode of LLM Self-Training"

## Summary (Overview)

- **Core phenomenon**: The paper documents a "rise-then-collapse" failure mode in REINFORCE post-training for code generation, where models improve rapidly on their target metric (pass@1) and then catastrophically collapse *within the same training campaign*—even on a fixed distribution, with no task switching.
- **Scale-dependent intervention effectiveness**: The paper evaluates three intervention levels—between-campaign memory (CARE), within-campaign early-stopping (ES), and algorithm-level variance reduction (GRPO)—and finds their relative value is regime-dependent: CARE helps at 3B (9.5% vs 4.9% end pass@1), ES sets a higher 7B ceiling (22.2%), and GRPO matches ES at 7B (20.7%) without any orchestration.
- **Two-timescale failure decomposition**: Self-improving RL fails at two distinct timescales—within-campaign cliff (phase-transition score ≈ 0.78, collapse onset near step 17/20, zero usable post-onset latency) and between-campaign carryover—and different interventions target different cells of this decomposition.
- **GRPO raises the floor but not the cliff**: Per-step trajectory diagnostics show GRPO and REINFORCE leave the same ≈ 17 pt within-campaign peak-to-end gap; GRPO's gain comes from improved between-campaign carryover, not within-campaign stabilization.
- **Key negative results**: Parameter-level regularization (EWC/KL) does not prevent collapse and can be counterproductive; CARE on top of GRPO adds no measurable gain; even GRPO+ES cannot eliminate single-campaign collapse.

## Introduction and Theoretical Foundation

### Background and Motivation

The paper addresses the proliferation of self-improving AI systems—iterative DPO, self-play, self-reward training, STaR, and autonomous research agents—which all aim to make models better by learning from their own outputs. The promise is a virtuous cycle: the model improves, generates better data, and improves further.

However, the authors document that **self-improvement is not a free lunch**. They observe a concrete failure mode in REINFORCE post-training for code: when training Qwen-2.5-7B on competitive programming tasks, pass@1 rises from 25% to 81% within the first 50 gradient steps, then collapses to near-zero by step 200. The model improves rapidly, then catastrophically forgets how to generate correct code.

### What the Collapse Is (and Is Not)

The collapse is **not** primarily caused by task switching. It occurs within a nominally fixed competitive-programming training distribution, with no change of dataset or objective. The authors interpret it as **within-task policy over-optimization**: early REINFORCE sharpens useful behaviors already present in the pretrained model, but continued optimization narrows the policy around brittle reward-correlated patterns, reducing solution diversity and overwriting broad code-generation priors.

### Core Theoretical Insight

The paper formalizes improvement strategies as capability tradeoff operators rather than scalar operators:

$$s: \mathbf{c} \mapsto \mathbf{c} + \delta_{s}(\mathbf{c}, \mathrm{ctx}) \tag{1}$$

where $\mathbf{c} = (c_1, \ldots, c_K) \in [0,1]^K$ is a K-dimensional capability vector and $\delta_s$ is the strategy's capability effect, which depends on the current capability state and context.

A self-improving system that remembers only "this strategy improved the score" is discarding most of the information it needs to improve safely and cumulatively. The paper formalizes **hidden regression**: a strategy $s$ causes hidden regression on a protected capability $j \in \mathcal{P} \subseteq [K]$ if $\delta_s^{k^*} > 0$ but $\delta_s^{j} < -\epsilon$ for the target capability $k^*$ and threshold $\epsilon$.

### The Three Intervention Levels

| Level | Method | When | What signal it uses |
|-------|--------|------|-------------------|
| Between campaign | CARE | after each campaign | memory + end/peak vector |
| Within campaign | ES | during each campaign | per-step pass@1 trajectory |
| Algorithm level | GRPO | inside each update | group-relative reward |

## Methodology

### Testbed

- **Models**: Qwen-2.5-3B-Instruct and Qwen-2.5-7B-Instruct (with a single-seed Gemma-3-4B pilot for cross-family check)
- **Task**: Competitive programming code generation with binary CodeGrader reward
- **Training**: REINFORCE (vanilla, group-size 16, no advantage normalization) or GRPO (use_score_centering=True, num_alter_tokens=4)
- **Protocol**: Three campaign-length settings:
  1. Diagnostic 200-step run (single seed, to characterize trajectory shape)
  2. Module-ablation campaigns (50 steps × 3 sequential campaigns × 5 seeds)
  3. Headline long-chain protocol (20 steps × 10 sequential campaigns)

### CARE Architecture

CARE (Capability-Aware Research Experience) consists of three modules:

1. **Capability-Effect Memory M**: Records structured entries $\{s, \mathrm{ctx}, \delta_s, \text{boundary}, \text{confidence}\}$; multiple observations aggregated as a linear Gaussian posterior over $\delta_s(x)$ conditioned on context features $\phi(x)$.

2. **Self-Improvement Transfer Gate G**: Given candidate strategy $s$ and current context $x$, selects one of four actions {reuse, adapt, pilot, reject} to maximize expected target improvement subject to a per-protected-capability regression-probability constraint:
$$\max E[\delta_s^{k^*}] \quad \text{s.t.} \quad Pr[\delta_s^j < -\epsilon] \leq \alpha, \forall j \in \mathcal{P}$$

3. **Regression-Aware Belief Revision R**: When observed capability delta deviates from posterior prediction by more than a Mahalanobis-distance threshold, the system refines boundary predicates, updates the Gaussian posterior with inflated noise, and tightens the gate's regression-probability threshold $\alpha$.

### Baselines

- **Naive (no KL)**: Standard REINFORCE with no regularization
- **EWC (KL=0.05)**: Strong KL penalty to reference policy
- **Adaptive KL (0.01→0.02→0.03)**: Progressively increasing KL schedule
- **Random HPO**: Random search over $(lr, kl) \in \{5e-7, 1e-6, 2e-6\} \times \{0, 0.01, 0.05\}$
- **LLM-agent HPO**: Gemini proposes next (lr, kl) given scalar pass@1 history
- **Best-checkpoint oracle**: Peak checkpoint selection, no gate

## Empirical Validation / Results

### Parameter-Level Regularization Fails

**Table 1: Parameter-level regularization fails to prevent collapse in REINFORCE self-improvement** (Start/End = pass@1 at beginning/end of each 50-step campaign)

| Method | C1 Start | C1 Peak | C1 End | C2 Start | C2 Peak | C2 End | C3 Start | C3 Peak | C3 End |
|--------|----------|---------|--------|----------|---------|--------|----------|---------|--------|
| Naive (no KL) | 0.25 | 1.00 | 0.06 | 0.63 | 0.96 | 0.75 | 0.69 | 1.00 | 0.22 |
| EWC (KL=0.05) | 0.25 | 1.00 | 0.07 | 0.56 | 1.00 | 0.10 | 0.75 | 0.94 | 0.08 |
| Adaptive KL | 0.31 | 1.00 | 0.00 | 0.63 | 1.00 | 0.06 | 0.63 | 1.00 | 0.00 |

Key findings: All methods collapse within every campaign; KL regularization is counterproductive (anchors to degraded references); naive occasionally recovers but cannot sustain gains.

### Checkpoint Selection Matters Locally

**Table 2: Checkpoint-selection experiment** (single-step continuation)

| Init Checkpoint | Start pass@1 | End pass@1 | Avg pass@1 | Outcome |
|-----------------|--------------|------------|------------|---------|
| Step 50 (peak) | 0.63 | 0.81 | 0.27 | Maintains & improves |
| Step 100 (degrading) | 0.31 | 0.77 | 0.31 | Partial recovery |
| Step 150 (collapsed) | 0.41 | 0.00 | 0.24 | Cannot recover |

### Headline Long-Chain Results

**Table 6: Long-chain comparison** (10 sequential 20-step campaigns, python)

| Scale | Method | End pass@1 (%) | n |
|-------|--------|----------------|---|
| 3B | Qwen2.5-3B + Naive (end-ckpt) | 4.9 [2.1, 9.5] | 5 |
| 3B | Qwen2.5-3B + CARE v2 (gate + revision) | 9.5 [6.3, 12.7] | 5 |
| 7B | Qwen2.5-7B + Naive (end-ckpt) | 11.8 [5.2, 18.3] | 5 |
| 7B | Qwen2.5-7B + CARE v2 (gate + revision) | 13.8 [2.8, 27.3] | 5 |
| 7B (ref.) | Qwen2.5-7B + Best-ckpt oracle | 25.9 [25.9, 25.9] | 1 |

**Key result**: At 3B, CARE beats naive with paired bootstrap 95% CI of per-seed difference [+0.4, +8.9] (excludes zero, positive on 4/5 seeds). At 7B, CARE reaches parity (overlapping CIs).

### Oracle Upper Bounds and Deployed ES

**Table 9: Oracle upper bounds and deployed ES condition** (Qwen-2.5-7B)

| Method | Achieved (%) | Trace oracle (%) | Steps | Hindsight (%) |
|--------|--------------|------------------|-------|---------------|
| Naive (full budget) | 11.8 [5.2, 18.3] (n=5) | 34.2 [32.5, 36.3] (n=5) | 50 | 37.8 [36.6, 38.9] (n=5) |
| CARE v2 (gate-adjusted) | 13.8 [2.8, 27.3] (n=5) | 35.0 [32.6, 37.4] (n=5) | 55 | 47.9 [40.0, 57.6] (n=5) |
| Deployed ES (T=peak+3) | 22.2 [14.1, 28.0] (n=3) | - | 51 | 49.0 [37.0, 62.5] (n=3) |

### GRPO Results

**Table 10: GRPO raises the end-of-chain floor on Qwen-2.5-7B but does not eliminate the within-campaign cliff**

| RL update | Orchestration | End pass@1 (%) | n |
|-----------|---------------|----------------|---|
| REINFORCE | none (A0) | 11.8 [5.2, 18.3] | 5 |
| REINFORCE | CARE (A3) | 13.8 [2.8, 27.3] | 5 |
| REINFORCE | ES | 22.2 [14.1, 28.0] | 3 |
| GRPO | none (A0) | 20.7 [15.7, 25.1] | 5 |
| GRPO | CARE (A3) | 20.4 [10.9, 28.6] | 5 |
| GRPO | ES | 17.0 [0.0, 28.1] | 3 |

### Trajectory Diagnostics

**Table 11: Per-campaign trajectory diagnostics, Qwen-2.5-7B, naive A0**

| Per-campaign metric (7B, A0) | REINFORCE (W4) | GRPO (W17) |
|------------------------------|----------------|------------|
| n campaigns | 47 | 50 |
| mean peak pass@1 | 0.343 | 0.369 |
| mean end pass@1 | 0.166 | 0.204 |
| mean gap (peak - end) | 0.176 | 0.165 |
| collapse rate (gap > 0.2) | 36% | 38% |
| collapse rate (gap > 0.3) | 11% | 6% |
| collapse rate (gap > 0.5) | 0% | 0% |

**Critical finding**: GRPO does not close the within-campaign peak-to-end gap (≈ 17 pt under both rules). GRPO's gain comes from better between-campaign carryover: REINFORCE's c10 mean end drifts to 0.15 while GRPO's stays at 0.21.

### Failure-Mode Diagnosis

**Table 16: Failure-mode diagnosis on Wave 4 campaigns**

| Method | Phase-transition score | Collapse-onset step | Post-collapse latency |
|--------|----------------------|---------------------|----------------------|
| Naive (A0) | 0.78 [0.71, 0.85] (n=47) | 16.7 [16.6, 16.9] (n=47) | 0.0 [0.0, 0.0] (n=47) |
| Full CARE v2 (A3) | 0.68 [0.58, 0.77] (n=47) | 13.7 [12.6, 14.9] (n=47) | 0.0 [0.0, 0.0] (n=47) |

Three structural findings explain the scale-dependence:
1. **Collapse is phase-transition-like, not smooth**: ≈ 78% of the total peak-to-end drop happens in a single gradient step
2. **The end/peak signal is post-hoc**: collapse onset at step ≈ 17/20 with zero remaining post-onset latency
3. **Local checkpoint choice is high leverage, but chain-level reuse compounds errors**

## Theoretical and Practical Implications

### Where Each Intervention Level Helps

The paper's central contribution is a **two-timescale decomposition** of self-improving RL failure:

- **Within-campaign cliff**: Rise-then-collapse within a single campaign (phase-transition score ≈ 0.78, onset near step 17/20)
- **Between-campaign carryover**: Each campaign's end checkpoint either carries forward gain or drifts down

The three intervention levels target different cells:
- **CARE** (between-campaign memory): Helps fragile carryover at 3B where naive REINFORCE is fragile
- **ES** (within-campaign stopping): Recovers within-campaign peaks under REINFORCE at 7B
- **GRPO** (algorithm-level): Improves carryover at 7B but leaves the within-campaign cliff approximately intact

### Practical Guidance

For practitioners:
- **Rich-signal 7B regime**: Switch the update rule first (GRPO) and add within-campaign stopping
- **Fragile-signal 3B regime**: Between-campaign memory is the lever that still pays

### Compute Efficiency

CARE achieves higher end pass@1 (13.8% vs 11.8%) while consuming 14.5% less compute (141 vs 165 gradient steps), yielding +38% higher per-100-step efficiency (9.80 vs 7.11).

## Conclusion

### Main Takeaways

1. **Rise-then-collapse is a robust, reproducible phenomenon**: It occurs across scales (3B, 7B), seeds, and even model families (single-seed Gemma-3-4B pilot shows peak 32.8%, end 0%), driven by within-task policy over-optimization rather than task switching.

2. **The intervention level matters, and its value is scale-dependent**: CARE nearly doubles end pass@1 at 3B (9.5% vs 4.9%), ES sets a higher 7B ceiling (22.2%), and GRPO matches ES at 7B (20.7%) without any orchestration.

3. **GRPO raises the floor but not the cliff**: The within-campaign peak-to-end gap remains ≈ 17 pt under both REINFORCE and GRPO; GRPO's gain comes from better between-campaign carryover.

4. **Single-campaign collapse remains unresolved**: Even GRPO+ES cannot prevent occasional late-chain catastrophes (1/3 seeds in W18 collapsed in the final campaign).

5. **As a slogan**: "GRPO raises the floor, ES targets the cliff, and CARE helps fragile carryover."

### Future Directions

- **Online collapse predictor**: Replace CARE's end-of-campaign gate with a within-campaign predictor that fires on leading indicators (entropy decay, sample-diversity drop, gradient-norm trajectory) before the cliff
- **K-dimensional deployed gate**: Extend the deployed gate from scalar (K=1) to a true multi-dimensional capability posterior
- **Cross-family and cross-algorithm replication**: PPO, DPO on math or dialogue tasks
- **Fully autonomous self-improvement**: Replace the external meta-reasoner with the improving model itself
- **Single-campaign restart-or-rollback policy**: Address the unresolved failure mode where a single late-chain campaign collapses despite both algorithm-level and within-campaign control

---

_Markdown view of https://picx.dev/p/e8QzQK, served by PicX — AI-generated visual whiteboard summaries of research papers._