Summary (Overview)

  • Problem: Removing three quarters of routed experts from a 122B-parameter reasoning mixture-of-experts (MoE) model makes it fit in a 12 GiB file but causes catastrophic reasoning loops—the pruned model enters verbatim repetition loops hitting the token limit on 44/120 development problems, while the unpruned model at the same 2-bit quantization loops on none.
  • Key attribution: Through a controlled comparison with byte-identical kept experts, the paper isolates pruning (not 2-bit quantization) as the cause of the looping failure.
  • Method: The paper introduces Method B, which adds rank-8 trained low-rank corrections to kept experts plus restores two experts per layer, deployed as a load-time adapter that leaves quantized bytes untouched.
  • Results: In a pre-registered comparison on 130 untouched problems, B solves 116 vs. 104 for the weight-free baseline (+9.2 pp, 95% CI [+3.1, +15.4]), with 15 problems gained and 3 lost.
  • Cost: B costs 1.7× wall time compared to the baseline; on regression checks it shows no gain (HumanEval+ +0.6 pp, MBPP+ −1.9 pp, within a 2 pp tolerance).

Introduction and Theoretical Foundation

Expert pruning removes whole experts from MoE language models to reduce memory footprint. While pruning to half the experts can be nearly lossless on code generation (Lasby et al., 2025), deeper pruning crosses memory tiers and introduces failure modes. This paper studies one such failure: verbatim reasoning loops in a pruned reasoning model.

The paper frames the research question around three escalating repair costs:

  1. Sampling penalty (S) — a decoding-time intervention, no training
  2. Stop-and-redraw (retry) — a client-side rule that detects loops and resamples, no training
  3. Weight recovery (A and B) — trained corrections that must beat the weight-free baselines

The theoretical foundation builds on several prior results:

  • Reconstruction-based updates of retained experts improve LiveCodeBench after pruning down to 37.5% retention (Markovic-Voronov et al., 2026)
  • Low-rank matrices can compensate compressed models while weights stay unchanged (Liu et al., 2024)
  • Loops after aggressive compression of reasoning models have been reported for quantization (Alimaskina et al., 2026; Chen et al., 2026)

Key contributions:

  • Controlled attribution of loops to pruning (not quantization)
  • Stop-and-redraw as a strong weight-free baseline
  • A weight recovery method (B) that leaves quantized model bytes untouched
  • A pre-registered confirmation with frozen decision rules
  • Two negative results: a training-free weight edit and the sampling penalty alone

Methodology

Model and Pruning

  • Model: Ornith-1.5-122B, 48 layers, hidden width 3072, 256 routed experts per layer (width 1024), 8 experts selected per token, plus one shared expert
  • Pruning: REAP criterion (Lasby et al., 2025), keeping top 64 experts per layer (25% retention)
  • Quantization: Pruned model C built by copying kept experts' slices from a public 2-bit file (UD-IQ2_XXS)—no kept weight is requantized
  • Teacher: Unpruned 4-bit file (UD-Q4_K_XL) used for fitting

Problem Sets (disjoint)

SetnDescription
Development120Earlier half of v4–v6 by contest date; all choices made here
Old test half124Later half of v4–v6; used once for weight edit test, then as fitting data
Fresh confirmation64v1–v3 problems, untouched until gate test
Fresh final130v1–v3 problems, untouched until primary comparison

Method A: Low-Rank Corrections

For each layer ℓ\ell and each kept expert ee, each of its three matrices W∈{Wˉegate,Weup,Wedown}W \in \{\bar{W}_e^{\mathrm{gate}}, W_e^{\mathrm{up}}, W_e^{\mathrm{down}}\} becomes W+BWAWW + B_W A_W with rank 8. Layers are fitted in order, minimizing:

Lℓ=∑twt∥yt−ft∥2+μ∑twt∥ft−ft0∥2∑twt∥yt−ft0∥2,(1)\mathcal{L}_{\ell} = \frac{\sum_{t} w_{t} \| y_{t} - f_{t} \|^{2} + \mu \sum_{t} w_{t} \| f_{t} - f_{t}^{0} \|^{2}}{\sum_{t} w_{t} \| y_{t} - f_{t}^{0} \|^{2}},\tag{1}

with μ=0.01\mu = 0.01, where:

  • xtx_t = input to expert block at position tt (pruned model with corrections from layers 0..ℓ−10..\ell-1 applied)
  • yty_t = unpruned model's routed experts' output on xtx_t
  • ftf_t = corrected pruned experts' output under pruned routing
  • ft0f_t^0 = uncorrected pruned experts' output

BWB_W starts at zero (training starts exactly at C). Adam optimizer, learning rate 10−410^{-4}, gradient clipping at 1, 25 passes over fitting documents.

Method B: A + Two Restored Experts per Layer

  • Restores the next two experts per layer from the unpruned 2-bit file with original router rows (66 experts per layer)
  • Restored experts train in full precision with straight-through estimator to quantized deployed values
  • Kept 64 experts receive rank-8 corrections as in A
  • Routed retention rises from 25% to 25.8%; file from 11.62 to 11.88 GiB; adapter adds 1.16 GiB

Decoding Policy (given to both arms)

  • Temperature 0.6, top-p 0.95, top-k 20, 32,768 token limit
  • Sampling penalty (S): presence penalty of 1.5 over a 20,480-token window
  • Stop-and-redraw: stops a reply when its tail repeats (for period P≤6000P \leq 6000 chars, the last max(5P, 200) chars equal the same-length stretch ending P chars earlier, checked every 256 chars); up to 3 draws per problem

Pre-registration

The finalist (B+S+retry), baseline (S+retry), sampling settings, seeds, fresh panels, and decision rules were frozen with hashes of the code before any request on confirmation/final problems.


Empirical Validation / Results

Attribution of Loops to Pruning

Table 1: Development problems (n = 120), 16K token limit

ModelSolvedHit the limitLimit, looping
Pruned to 25%, 2-bit (C)714444
Unpruned, 2-bit107110
Unpruned, 4-bit11080

Pruning raises loop-at-limit rate by +36.7 pp [+28.3, +45.8] and accounts for a +30.0 pp drop in solved problems [+21.7, +38.3]. Quantization alone adds no loops.

Weight-Free Remedies

  • Sampling penalty (S): presence penalty 1.5 over 20,480-token window; on development problems solves 86 vs. 71 without it, loops drop from 44 to 25
  • Training-free weight edit: did not transfer—−4.8 pp against C on old test half [−12.9, +3.2]
  • Stop-and-redraw: lifts penalized pruned model from 89 to 104 of 130 solved on untouched problems

Pre-registered Confirmation Results

Table 3: S+retry against B+S+retry (problems solved after at most three draws)

ProblemsnS+retryB+S+retryDifferenceGained/LostMcNemar p
Development (reused)12088101+10.8 [+3.3, +18.3]18/50.011
Confirmation, seed 1645558+4.7 [−1.6, +12.5]4/10.375
Confirmation, seed 2645257+7.8 [−1.6, +17.2]7/20.180
Confirmation, pooled64——+6.2 [+0.8, +13.3]——
Final (untouched)130104116+9.2 [+3.1, +15.4]15/30.007

The confirmation gate passed; the final comparison meets the frozen "leads" rule (≥2 pp and 95% interval above zero).

Where the Gain Is

  • Redrawing helps both arms: baseline goes from 89 to 104 solved; B from 100 to 116 (after three draws)
  • B removes more wrong answers than loops: after 3 draws, baseline fails 16 problems with finished wrong answers and 9 looping; B fails 8 and 4
  • Concentrated on medium problems: easy 60→62 of 63; medium 44→54 of 67

Regression Guards

BenchmarkBBaselineDifference
HumanEval+145/164144/164+0.6 pp [−4.3, +5.5]
MBPP+269/378276/378−1.9 pp [−5.6, +1.6]

The MBPP+ margin is the narrowest possible—one more lost problem would have failed the 2 pp tolerance rule.


Theoretical and Practical Implications

Practical Significance

  • Memory-tier breakthrough: Pruning to 25% retention makes a 122B model fit in 11.62 GiB, enabling deployment on consumer hardware
  • Weight-free recovery is powerful: Stop-and-redraw recovers more than the weight method's marginal gain over it—any weight-based approach must beat this baseline
  • Low-rank adapters preserve quantized files: The approach keeps the quantized model untouched, avoiding requantization drift and enabling load-time adaptation

Theoretical Implications

  • Loop attribution is precise: The controlled comparison (byte-identical kept experts) cleanly isolates pruning as the cause of reasoning loops, not quantization
  • Recovery mechanism is unexplained: B's gain appears on finished wrong answers and medium problems more than on loops—the paper does not explain why
  • Component contributions are confounded: B bundles low-rank corrections with restored experts; the individual contributions are not separated on untouched problems

Limitations (as stated by the authors)

  • One model, one pruning ratio, one quantization, one benchmark family
  • One fit per method; no hyperparameter search
  • B bundles two changes (corrections + restored experts)
  • Equal draws, not equal cost (B is 1.7× slower)
  • No comparison with other recovery methods (e.g., dense EWU, activation steering)
  • No mechanistic account of why B helps

Conclusion

At a quarter of its experts, this reasoning MoE fails mostly by looping, and much of that is recoverable without training: a sampling penalty recovers part on development problems, and stopping and redrawing adds more. On top of that baseline, a small trained correction to the kept experts plus two restored experts per layer solved 116 of 130 untouched problems against 104, in a comparison whose rules were fixed in advance.

On two older, shorter benchmarks the method showed no gain, and on one a small loss inside the pre-set tolerance. The result is confirmed for one model.

Future directions:

  1. A second model to test generalizability
  2. A fresh-panel test separating corrections from restored experts
  3. A cheaper way to apply the corrections at inference

Related papers