Summary (Overview)
- Core finding: A temporary optimization intervention (weight decay) can increase validation loss when removed yet improve final loss at the end of training—demonstrated in Muon-trained decoder-only Transformers from 162M to 3.4B parameters.
- Mechanism: Early weight decay creates a persistent weight-norm deficit that keeps relative step sizes larger after the intervention ends, continuing to influence optimization long after the explicit update rule returns to normal.
- Evaluation implications: Temporary schedules should be evaluated at the intended final training budget, not at removal—the sign and ranking of schedules can reverse after the intervention is removed.
- Scale transfer: A schedule selected at 162M transfers without retuning to 1.4B, finishing 0.0275 nats better than no decay despite being 0.26 nats worse at removal.
- Generalizability: A separate Gram penalty intervention exhibits similar delayed dynamics, suggesting the phenomenon is not unique to multiplicative weight decay.
Introduction and Theoretical Foundation
The paper addresses a fundamental problem in evaluating temporary optimization interventions: when an intervention ends, the treated and reference runs enter subsequent phases with different parameters and optimizer states, so their loss difference can continue to evolve even with identical explicit update rules.
The central questions are:
- How should a temporary optimization intervention be evaluated?
- What aspects of the training state carry its effect after removal?
Key theoretical connections:
- Weight decay in Muon-trained models provides a controlled setting since turning it off cleanly separates direct shrinkage from the training state left behind.
- Muon's matrix updates have approximately controlled magnitude, so reducing weight norms increases the update size relative to current weights—creating a persistent effect via relative step sizes.
The temporary weight decay update is formalized as:
where is the Muon update before multiplication by the learning rate , and is the intervention window.
Methodology
Experimental setup:
- Decoder-only Transformers with 162M, 405M, 1.4B, and 3.4B parameters
- Muon optimizer (orthogonalizes momentum updates) with squared-ReLU () MLPs
- Context length 1,024; 1,024 sequences per update
- Learning rate 0.01 for first 90% of training, linear decay over final 10%
- Compute-optimal budget: ~20 tokens per parameter
- Main experiments use FineWeb100B; mechanism experiments use FineWeb-Edu
Key metrics:
- Loss difference: (negative values indicate improvement)
- Total log-shrinkage for comparing schedules: D = -\sum_t \log(1 - \eta_t \lambda_t)\tag{2}
- Relative step size: \alpha_t = \frac{\|P_{W_t}^{\perp}\Delta W_t\|_F}{\|W_t\|_F} = \eta_t \frac{\|P_{W_t}^{\perp}U_t\|_F}{\|W_t\|_F},\tag{3} where
Experimental manipulations:
- Varying decay window start time (0%, 20%, 60% of training)
- Varying window duration (320 to 3,564 updates)
- Varying decay coefficient (grid search)
- Fixed-norm replays using Hyperball variant
- Norm-reset experiments
- Gram penalty intervention (off-diagonal penalty on )
Empirical Validation / Results
Delayed crossover phenomenon (Figure 1): In a 1.4B model, temporary decay (, first 44% of training) is 0.26 nats worse than no decay at removal, overtakes it only after 73% of training, and finishes 0.0275 nats better.
Start time changes sign: A 10% window starting at initialization beats both references at all scales, while the same window starting at 60% is worse than both.
Evaluation horizon changes ranking: At 405M, comparing at a common final checkpoint vs. equal post-removal training reverses the ranking of early schedules.
Duration effects beyond total shrinkage: At 162M with matched total log-shrinkage , final improvement more than doubles as window lengthens from 700 to 3,564 updates.
Scale transfer results (Table 1):
| Scale | Updates | Early schedule (λ, window) | Early loss | No decay | Best const. | vs. best ref. |
|---|---|---|---|---|---|---|
| 162M | 3,200 | 0.3, 10% | 3.3710 | 3.3860 | 3.3768 (0.05) | -0.0058 |
| 405M | 8,100 | 0.3, 10% | 3.0287 | 3.0537 | 3.0462 (0.02) | -0.0175 |
| 1.4B | 28,000 | 0.2, 10% | 2.6506 | 2.6754 | 2.6743 (0.003) | -0.0237 |
| 1.4B | 28,000 | 0.15, 44% (transferred) | 2.6479 | 2.6754 | 2.6743 (0.003) | -0.0264 |
| 3.4B | 66,000 | 0.2, 10% | 2.4720 | 2.4832 | 2.4843 (0.001) | -0.0112 |
Mechanism experiments:
- Early decay leaves norms at 12–16% of no-decay values at removal, persisting at 60–66% at final checkpoint
- Fixed-norm replays reproduce performance ordering: early-decay trajectory improves final loss by 0.022–0.023 nats vs. no-decay trajectory
- Trajectory swaps show most replay advantage (0.0181 of 0.0221 nats) comes from altered steps after decay removal
Approximate norm-dynamics recurrence:
This recurrence explains the slow convergence: fixed offsets in become progressively smaller fractions of the growing total.
Gram penalty results: A qualitatively different intervention (off-diagonal Gram penalty) shows the same timing dependence: 10% windows starting at 0%, 20%, and 60% change final loss by -0.0147, -0.0080, and +0.0209 nats respectively.
Theoretical and Practical Implications
Methodological implication: Temporary schedules should be evaluated at the training budget at which the model will be used, not only when the intervention ends. Removal-time loss can be a poor predictor of eventual effect.
Mechanistic insight: The paper identifies weight norms as a persistent state variable through which removed interventions continue to influence optimization. The relationship between norms and relative step sizes (inverse proportionality) provides a concrete mechanism for delayed effects.
Limitations acknowledged:
- Larger relative steps are not universally better (continuing decay keeps steps larger yet yields smaller improvements)
- The norm account explains how delayed influence works, not why particular trajectories are beneficial
- Norms are not established as the only persistent state
- Results are specific to tested regimes; early decay does not improve every optimizer setting
Conclusion
The paper demonstrates that temporary optimization schedules can have delayed effects long after removal, with both sign and ranking of schedules capable of reversing at later checkpoints. The mechanism involves persistent weight-norm deficits that keep relative step sizes larger after the intervention ends, with an approximate recurrence explaining the slow convergence of these differences.
The practical takeaway is clear: temporary schedules should be evaluated at the intended final training budget, not only at removal. Future work remains open on whether similar delayed dynamics arise broadly across optimizers and temporary interventions, and on understanding why particular relative-step trajectories are beneficial for final loss.
Related papers
- Understanding the Role of Reasoning Data in Pretraining
Optimal pretraining mixtures of atomic facts and reasoning traces favor more atomic data as reasoning chains lengthen, with interior mixtures outperforming pure extremes.
- Pretraining Data Statistics Shape the Phases of Learning Entity Comparison in Language Models
Pretraining data statistics causally determine three sequential learning phases—frequency heuristic, positional bias, then correct algorithm—with critical data density thresholds governing whether models ever escape heuristics.
- Equal Steps, Unequal Curvature: Where Muon's Update Goes Wrong
Muon's equal step lengths across singular directions over-step high-curvature top directions, and splitting QKV orthogonalization plus a spectral tilt improves loss by 0.046 nats.