# Reasoning Again with a Quarter of the Experts: Retry First, Then Train

> Pruning, not quantization, causes reasoning loops in a 25%-retention 122B MoE model, and low-rank corrections plus restored experts recover 9.2 percentage points over a weight-free baseline.

- **Source:** [arXiv](https://arxiv.org/abs/2611.20419)
- **Published:** 2026-11-21
- **Permalink:** https://picx.dev/p/xEtJnp
- **Whiteboard:** https://picx.dev/p/xEtJnp/image

## Summary

## Summary (Overview)

- **Problem**: Removing three quarters of routed experts from a 122B-parameter reasoning mixture-of-experts (MoE) model makes it fit in a 12 GiB file but causes catastrophic reasoning loops—the pruned model enters verbatim repetition loops hitting the token limit on 44/120 development problems, while the unpruned model at the same 2-bit quantization loops on none.
- **Key attribution**: Through a controlled comparison with byte-identical kept experts, the paper isolates pruning (not 2-bit quantization) as the cause of the looping failure.
- **Method**: The paper introduces **Method B**, which adds rank-8 trained low-rank corrections to kept experts plus restores two experts per layer, deployed as a load-time adapter that leaves quantized bytes untouched.
- **Results**: In a pre-registered comparison on 130 untouched problems, B solves 116 vs. 104 for the weight-free baseline (+9.2 pp, 95% CI [+3.1, +15.4]), with 15 problems gained and 3 lost.
- **Cost**: B costs 1.7× wall time compared to the baseline; on regression checks it shows no gain (HumanEval+ +0.6 pp, MBPP+ −1.9 pp, within a 2 pp tolerance).

---

## Introduction and Theoretical Foundation

Expert pruning removes whole experts from MoE language models to reduce memory footprint. While pruning to half the experts can be nearly lossless on code generation (Lasby et al., 2025), deeper pruning crosses memory tiers and introduces failure modes. This paper studies one such failure: verbatim reasoning loops in a pruned reasoning model.

The paper frames the research question around three escalating repair costs:

1. **Sampling penalty (S)** — a decoding-time intervention, no training
2. **Stop-and-redraw (retry)** — a client-side rule that detects loops and resamples, no training
3. **Weight recovery (A and B)** — trained corrections that must beat the weight-free baselines

The theoretical foundation builds on several prior results:
- Reconstruction-based updates of retained experts improve LiveCodeBench after pruning down to 37.5% retention (Markovic-Voronov et al., 2026)
- Low-rank matrices can compensate compressed models while weights stay unchanged (Liu et al., 2024)
- Loops after aggressive compression of reasoning models have been reported for quantization (Alimaskina et al., 2026; Chen et al., 2026)

**Key contributions**:
- Controlled attribution of loops to pruning (not quantization)
- Stop-and-redraw as a strong weight-free baseline
- A weight recovery method (B) that leaves quantized model bytes untouched
- A pre-registered confirmation with frozen decision rules
- Two negative results: a training-free weight edit and the sampling penalty alone

---

## Methodology

### Model and Pruning

- **Model**: Ornith-1.5-122B, 48 layers, hidden width 3072, 256 routed experts per layer (width 1024), 8 experts selected per token, plus one shared expert
- **Pruning**: REAP criterion (Lasby et al., 2025), keeping top 64 experts per layer (25% retention)
- **Quantization**: Pruned model C built by copying kept experts' slices from a public 2-bit file (UD-IQ2_XXS)—no kept weight is requantized
- **Teacher**: Unpruned 4-bit file (UD-Q4_K_XL) used for fitting

### Problem Sets (disjoint)

| Set | n | Description |
|-----|---|-------------|
| Development | 120 | Earlier half of v4–v6 by contest date; all choices made here |
| Old test half | 124 | Later half of v4–v6; used once for weight edit test, then as fitting data |
| Fresh confirmation | 64 | v1–v3 problems, untouched until gate test |
| Fresh final | 130 | v1–v3 problems, untouched until primary comparison |

### Method A: Low-Rank Corrections

For each layer $\ell$ and each kept expert $e$, each of its three matrices $W \in \{\bar{W}_e^{\mathrm{gate}}, W_e^{\mathrm{up}}, W_e^{\mathrm{down}}\}$ becomes $W + B_W A_W$ with rank 8. Layers are fitted in order, minimizing:

$$
\mathcal{L}_{\ell} = \frac{\sum_{t} w_{t} \| y_{t} - f_{t} \|^{2} + \mu \sum_{t} w_{t} \| f_{t} - f_{t}^{0} \|^{2}}{\sum_{t} w_{t} \| y_{t} - f_{t}^{0} \|^{2}},\tag{1}
$$

with $\mu = 0.01$, where:
- $x_t$ = input to expert block at position $t$ (pruned model with corrections from layers $0..\ell-1$ applied)
- $y_t$ = unpruned model's routed experts' output on $x_t$
- $f_t$ = corrected pruned experts' output under pruned routing
- $f_t^0$ = uncorrected pruned experts' output

$B_W$ starts at zero (training starts exactly at C). Adam optimizer, learning rate $10^{-4}$, gradient clipping at 1, 25 passes over fitting documents.

### Method B: A + Two Restored Experts per Layer

- Restores the next two experts per layer from the unpruned 2-bit file with original router rows (66 experts per layer)
- Restored experts train in full precision with straight-through estimator to quantized deployed values
- Kept 64 experts receive rank-8 corrections as in A
- Routed retention rises from 25% to 25.8%; file from 11.62 to 11.88 GiB; adapter adds 1.16 GiB

### Decoding Policy (given to both arms)

- Temperature 0.6, top-p 0.95, top-k 20, 32,768 token limit
- **Sampling penalty (S)**: presence penalty of 1.5 over a 20,480-token window
- **Stop-and-redraw**: stops a reply when its tail repeats (for period $P \leq 6000$ chars, the last max(5P, 200) chars equal the same-length stretch ending P chars earlier, checked every 256 chars); up to 3 draws per problem

### Pre-registration

The finalist (B+S+retry), baseline (S+retry), sampling settings, seeds, fresh panels, and decision rules were frozen with hashes of the code before any request on confirmation/final problems.

---

## Empirical Validation / Results

### Attribution of Loops to Pruning

**Table 1**: Development problems (n = 120), 16K token limit

| Model | Solved | Hit the limit | Limit, looping |
|-------|--------|---------------|----------------|
| Pruned to 25%, 2-bit (C) | 71 | 44 | 44 |
| Unpruned, 2-bit | 107 | 11 | 0 |
| Unpruned, 4-bit | 110 | 8 | 0 |

Pruning raises loop-at-limit rate by **+36.7 pp** [+28.3, +45.8] and accounts for a **+30.0 pp** drop in solved problems [+21.7, +38.3]. Quantization alone adds no loops.

### Weight-Free Remedies

- **Sampling penalty (S)**: presence penalty 1.5 over 20,480-token window; on development problems solves 86 vs. 71 without it, loops drop from 44 to 25
- **Training-free weight edit**: did not transfer—−4.8 pp against C on old test half [−12.9, +3.2]
- **Stop-and-redraw**: lifts penalized pruned model from 89 to 104 of 130 solved on untouched problems

### Pre-registered Confirmation Results

**Table 3**: S+retry against B+S+retry (problems solved after at most three draws)

| Problems | n | S+retry | B+S+retry | Difference | Gained/Lost | McNemar p |
|----------|---|---------|-----------|------------|-------------|-----------|
| Development (reused) | 120 | 88 | 101 | +10.8 [+3.3, +18.3] | 18/5 | 0.011 |
| Confirmation, seed 1 | 64 | 55 | 58 | +4.7 [−1.6, +12.5] | 4/1 | 0.375 |
| Confirmation, seed 2 | 64 | 52 | 57 | +7.8 [−1.6, +17.2] | 7/2 | 0.180 |
| Confirmation, pooled | 64 | — | — | +6.2 [+0.8, +13.3] | — | — |
| **Final (untouched)** | **130** | **104** | **116** | **+9.2 [+3.1, +15.4]** | **15/3** | **0.007** |

The confirmation gate passed; the final comparison meets the frozen "leads" rule (≥2 pp and 95% interval above zero).

### Where the Gain Is

- **Redrawing helps both arms**: baseline goes from 89 to 104 solved; B from 100 to 116 (after three draws)
- **B removes more wrong answers than loops**: after 3 draws, baseline fails 16 problems with finished wrong answers and 9 looping; B fails 8 and 4
- **Concentrated on medium problems**: easy 60→62 of 63; medium 44→54 of 67

### Regression Guards

| Benchmark | B | Baseline | Difference |
|-----------|---|----------|------------|
| HumanEval+ | 145/164 | 144/164 | +0.6 pp [−4.3, +5.5] |
| MBPP+ | 269/378 | 276/378 | −1.9 pp [−5.6, +1.6] |

The MBPP+ margin is the narrowest possible—one more lost problem would have failed the 2 pp tolerance rule.

---

## Theoretical and Practical Implications

### Practical Significance

- **Memory-tier breakthrough**: Pruning to 25% retention makes a 122B model fit in 11.62 GiB, enabling deployment on consumer hardware
- **Weight-free recovery is powerful**: Stop-and-redraw recovers more than the weight method's marginal gain over it—any weight-based approach must beat this baseline
- **Low-rank adapters preserve quantized files**: The approach keeps the quantized model untouched, avoiding requantization drift and enabling load-time adaptation

### Theoretical Implications

- **Loop attribution is precise**: The controlled comparison (byte-identical kept experts) cleanly isolates pruning as the cause of reasoning loops, not quantization
- **Recovery mechanism is unexplained**: B's gain appears on finished wrong answers and medium problems more than on loops—the paper does not explain why
- **Component contributions are confounded**: B bundles low-rank corrections with restored experts; the individual contributions are not separated on untouched problems

### Limitations (as stated by the authors)

- One model, one pruning ratio, one quantization, one benchmark family
- One fit per method; no hyperparameter search
- B bundles two changes (corrections + restored experts)
- Equal draws, not equal cost (B is 1.7× slower)
- No comparison with other recovery methods (e.g., dense EWU, activation steering)
- No mechanistic account of why B helps

---

## Conclusion

At a quarter of its experts, this reasoning MoE fails mostly by looping, and much of that is recoverable without training: a sampling penalty recovers part on development problems, and stopping and redrawing adds more. On top of that baseline, a small trained correction to the kept experts plus two restored experts per layer solved 116 of 130 untouched problems against 104, in a comparison whose rules were fixed in advance.

On two older, shorter benchmarks the method showed no gain, and on one a small loss inside the pre-set tolerance. The result is confirmed for one model.

**Future directions**:
1. A second model to test generalizability
2. A fresh-panel test separating corrections from restored experts
3. A cheaper way to apply the corrections at inference

---

_Markdown view of https://picx.dev/p/xEtJnp, served by PicX — AI-generated visual whiteboard summaries of research papers._
