# Delayed Effects of Temporary Weight Decay in Muon Pretraining

> Temporary weight decay in Muon-trained transformers can improve final loss despite hurting loss at removal, because persistent weight-norm deficits keep relative step sizes larger after the intervention ends.

- **Source:** [arXiv](https://arxiv.org/abs/2611.17876)
- **Published:** 2026-11-21
- **Permalink:** https://picx.dev/p/ZfMrTl
- **Whiteboard:** https://picx.dev/p/ZfMrTl/image

## Summary

## Summary (Overview)

- **Core finding**: A temporary optimization intervention (weight decay) can increase validation loss when removed yet improve final loss at the end of training—demonstrated in Muon-trained decoder-only Transformers from 162M to 3.4B parameters.
- **Mechanism**: Early weight decay creates a persistent weight-norm deficit that keeps relative step sizes larger after the intervention ends, continuing to influence optimization long after the explicit update rule returns to normal.
- **Evaluation implications**: Temporary schedules should be evaluated at the intended final training budget, not at removal—the sign and ranking of schedules can reverse after the intervention is removed.
- **Scale transfer**: A schedule selected at 162M transfers without retuning to 1.4B, finishing 0.0275 nats better than no decay despite being 0.26 nats worse at removal.
- **Generalizability**: A separate Gram penalty intervention exhibits similar delayed dynamics, suggesting the phenomenon is not unique to multiplicative weight decay.

## Introduction and Theoretical Foundation

The paper addresses a fundamental problem in evaluating temporary optimization interventions: when an intervention ends, the treated and reference runs enter subsequent phases with different parameters and optimizer states, so their loss difference can continue to evolve even with identical explicit update rules.

The central questions are:
1. How should a temporary optimization intervention be evaluated?
2. What aspects of the training state carry its effect after removal?

**Key theoretical connections:**
- Weight decay in Muon-trained models provides a controlled setting since turning it off cleanly separates direct shrinkage from the training state left behind.
- Muon's matrix updates have approximately controlled magnitude, so reducing weight norms increases the update size relative to current weights—creating a persistent effect via relative step sizes.

The temporary weight decay update is formalized as:

$$
W_{t+1} = (1 - \eta_t \lambda_t) W_t - \eta_t U_t, \quad \lambda_t = \lambda \mathbf{1}_{\{s \leq t < e\}},\tag{1}
$$

where $U_t$ is the Muon update before multiplication by the learning rate $\eta$, and $[s, e)$ is the intervention window.

## Methodology

**Experimental setup:**
- Decoder-only Transformers with 162M, 405M, 1.4B, and 3.4B parameters
- Muon optimizer (orthogonalizes momentum updates) with squared-ReLU ($\text{ReLU}^2$) MLPs
- Context length 1,024; 1,024 sequences per update
- Learning rate 0.01 for first 90% of training, linear decay over final 10%
- Compute-optimal budget: ~20 tokens per parameter
- Main experiments use FineWeb100B; mechanism experiments use FineWeb-Edu

**Key metrics:**
- Loss difference: $\Delta_t = \mathcal{L}_{\text{intervention}}(t) - \mathcal{L}_{\text{reference}}(t)$ (negative values indicate improvement)
- Total log-shrinkage for comparing schedules: $D = -\sum_t \log(1 - \eta_t \lambda_t)\tag{2}$
- Relative step size: $\alpha_t = \frac{\|P_{W_t}^{\perp}\Delta W_t\|_F}{\|W_t\|_F} = \eta_t \frac{\|P_{W_t}^{\perp}U_t\|_F}{\|W_t\|_F},\tag{3}$ where $P_W^{\perp}X = X - \langle W, X\rangle_F W/\|W\|_F^2$

**Experimental manipulations:**
- Varying decay window start time (0%, 20%, 60% of training)
- Varying window duration (320 to 3,564 updates)
- Varying decay coefficient (grid search)
- Fixed-norm replays using Hyperball variant
- Norm-reset experiments
- Gram penalty intervention (off-diagonal penalty on $W^\top W$)

## Empirical Validation / Results

**Delayed crossover phenomenon (Figure 1):** In a 1.4B model, temporary decay ($\lambda = 0.15$, first 44% of training) is 0.26 nats worse than no decay at removal, overtakes it only after 73% of training, and finishes 0.0275 nats better.

**Start time changes sign:** A 10% window starting at initialization beats both references at all scales, while the same window starting at 60% is worse than both.

**Evaluation horizon changes ranking:** At 405M, comparing at a common final checkpoint vs. equal post-removal training reverses the ranking of early schedules.

**Duration effects beyond total shrinkage:** At 162M with matched total log-shrinkage $D \approx 2.1$, final improvement more than doubles as window lengthens from 700 to 3,564 updates.

**Scale transfer results (Table 1):**

| Scale | Updates | Early schedule (λ, window) | Early loss | No decay | Best const. | vs. best ref. |
|-------|---------|---------------------------|------------|----------|-------------|---------------|
| 162M | 3,200 | 0.3, 10% | 3.3710 | 3.3860 | 3.3768 (0.05) | -0.0058 |
| 405M | 8,100 | 0.3, 10% | 3.0287 | 3.0537 | 3.0462 (0.02) | -0.0175 |
| 1.4B | 28,000 | 0.2, 10% | 2.6506 | 2.6754 | 2.6743 (0.003) | -0.0237 |
| 1.4B | 28,000 | 0.15, 44% (transferred) | 2.6479 | 2.6754 | 2.6743 (0.003) | -0.0264 |
| 3.4B | 66,000 | 0.2, 10% | 2.4720 | 2.4832 | 2.4843 (0.001) | -0.0112 |

**Mechanism experiments:**
- Early decay leaves norms at 12–16% of no-decay values at removal, persisting at 60–66% at final checkpoint
- Fixed-norm replays reproduce performance ordering: early-decay trajectory improves final loss by 0.022–0.023 nats vs. no-decay trajectory
- Trajectory swaps show most replay advantage (0.0181 of 0.0221 nats) comes from altered steps after decay removal

**Approximate norm-dynamics recurrence:**

$$
\frac{1}{a_{t+1}^2} - \frac{1}{a_t^2} \approx 1 + 2\kappa_t, \quad \kappa_t = \frac{\langle W_t, \Delta W_t\rangle_F}{\|\Delta W_t\|_F^2},\tag{4}
$$

This recurrence explains the slow convergence: fixed offsets in $1/a_t^2$ become progressively smaller fractions of the growing total.

**Gram penalty results:** A qualitatively different intervention (off-diagonal Gram penalty) shows the same timing dependence: 10% windows starting at 0%, 20%, and 60% change final loss by -0.0147, -0.0080, and +0.0209 nats respectively.

## Theoretical and Practical Implications

**Methodological implication:** Temporary schedules should be evaluated at the training budget at which the model will be used, not only when the intervention ends. Removal-time loss can be a poor predictor of eventual effect.

**Mechanistic insight:** The paper identifies weight norms as a persistent state variable through which removed interventions continue to influence optimization. The relationship between norms and relative step sizes (inverse proportionality) provides a concrete mechanism for delayed effects.

**Limitations acknowledged:**
- Larger relative steps are not universally better (continuing decay keeps steps larger yet yields smaller improvements)
- The norm account explains *how* delayed influence works, not *why* particular trajectories are beneficial
- Norms are not established as the only persistent state
- Results are specific to tested regimes; early decay does not improve every optimizer setting

## Conclusion

The paper demonstrates that temporary optimization schedules can have delayed effects long after removal, with both sign and ranking of schedules capable of reversing at later checkpoints. The mechanism involves persistent weight-norm deficits that keep relative step sizes larger after the intervention ends, with an approximate recurrence explaining the slow convergence of these differences.

The practical takeaway is clear: **temporary schedules should be evaluated at the intended final training budget**, not only at removal. Future work remains open on whether similar delayed dynamics arise broadly across optimizers and temporary interventions, and on understanding why particular relative-step trajectories are beneficial for final loss.

---

_Markdown view of https://picx.dev/p/ZfMrTl, served by PicX — AI-generated visual whiteboard summaries of research papers._
