# Optimizer Memory Schedules for Outscaling the Overtraining Axis

> Optimizer rankings and hyperparameters shift with training horizon, and ADANA's scheduled memory outscales AdamW, matching theory with a 1.15–1.20 exponent.

- **Source:** [arXiv](https://arxiv.org/abs/2609.04577)
- **Published:** 2026-09-26
- **Permalink:** https://picx.dev/p/5eLVmD
- **Whiteboard:** https://picx.dev/p/5eLVmD/image

## Summary

# Optimizer Memory Schedules for Outscaling the Overtraining Axis

## Summary (Overview)

- **Key finding**: Optimizer rankings and optimal hyperparameters change substantially with training horizon, establishing the overtraining (OT) axis as an essential dimension for optimizer evaluation and design.

- **ADANA outscales AdamW**: With log-time weight decay and momentum cooldown, ADANA achieves an equivalent-OT exponent of 1.15–1.20, closely matching the $2 - \kappa = 1.15$ prediction from DANA theory on power-law random features (for $\kappa = 0.85$).

- **Muon and SOAP provide roughly constant gains**: Matrix-preconditioned optimizers offer approximately constant 1.4×–1.7× token-multiplier advantages over AdamW across most of the measured OT range (1×–256×), though SOAP may gain further at the highest OT factors.

- **ADANA's trajectory**: ADANA begins substantially behind Muon and SOAP (0.59×–0.69× token multiplier) but surpasses Muon and becomes competitive with SOAP at the highest OT factors tested.

- **Hyperparameter scaling rules**: The preferred learning rate schedule can reverse across the OT axis, the best weight decay coefficient scales approximately as $\sqrt{f}$, and longer horizons generally favor longer fixed memory.

## Introduction and Theoretical Foundation

### Background and Motivation

Language models exhibit empirical scaling laws relating loss to model size, training data, and compute (Kaplan et al., 2020). The Chinchilla scaling law finds that model size and training tokens should grow together, corresponding to approximately 20 training tokens per parameter (Hoffmann et al., 2022). However, for widely deployed language models, recurring inference costs far exceed the one-time pretraining cost, favoring training smaller models on far more tokens—a practice known as **overtraining** (Sardana et al., 2024; Gadre et al., 2024; Bian et al., 2025).

The **overtraining factor** $f$ expresses the number of training tokens as a multiple of the compute-optimal value:

$$f = T / (20P)$$

where $T$ is the number of training tokens and $P$ is the nominal parameter count.

### Memory as a Theoretical Framework

The paper introduces a unified framework for understanding optimizer memory through timescales measured in optimizer updates.

**Momentum memory**: For a momentum state with constant coefficient $\beta$:

$$m_t = \beta m_{t-1} + (1-\beta) g_t = (1-\beta) \sum_{s=1}^{t} \beta^{t-s} g_s \tag{1}$$

The half-life and effective memory window are:

$$H_{1/2} = \frac{\log(1/2)}{\log \beta}, \qquad M = \frac{1}{1-\beta} \tag{2}$$

**Weight decay as memory**: Under decoupled weight decay:

$$\theta_{t+1} = (1-\alpha_t)\theta_t - \eta_t d_t \tag{3}$$

When $\alpha_t = \alpha > 0$ is constant:

$$\theta_{t+1} = (1-\alpha)\theta_t + \alpha\left(-\frac{\eta_t}{\alpha} d_t\right) \tag{4}$$

making the parameter iterate an exponential moving average of scaled optimizer updates with memory window approximately $1/\alpha$ updates.

### Optimizers Studied

- **AdamW**: Coordinatewise adaptive preconditioning with fixed scalar momentum
- **Muon and SOAP**: Matrix-level preconditioning with fixed memory
- **DANA/ADANA**: Scheduled momentum whose memory grows throughout training; ADANA incorporates this into an Adam-style adaptive optimizer

## Methodology

### Experimental Setup

- **Models**: 51M, 124M, and 253M parameter transformers
- **Overtraining factors**: Up to 256× (51M), 32× (124M), and 8× (253M)
- **Configuration**: Sequence length 2,048, global batch size of 256 sequences
- **Hyperparameter sweeps**: Independent base learning rate sweep at every setting; lowest final validation loss reported

### Key Hyperparameter Choices

- **Weight decay scaling**: $c_{\text{uniform}}(f) = 8\sqrt{f}$ or $c_{\text{log}}(f) = 2\sqrt{f}$, consistent with the square-root scaling predicted by Bergsma et al. (2025a)
- **Learning rate schedules**: Linear decay preferred at 1× OT; cosine decay preferred at 8× and 32× OT
- **Fixed memory coefficients**: $\beta_2 = 0.98$ for AdamW, matrix momentum $\beta = 0.98$ for Muon, $\beta = 0.98$ for SOAP
- **ADANA settings**: $\kappa = 0.85$, $\delta = 8$, $g_3 = 8$ (fixed based on transferability of data distribution properties)

### Momentum Cooldown Formulation

The paper proposes a novel **momentum cooldown** rule. The learning rate decay timescale is:

$$\tau_{\eta,t} = \frac{\eta_t}{|\eta_{t+1} - \eta_t|} \tag{8}$$

The original memory timescale for momentum coefficient $\beta_t$:

$$\tau_{\text{mom},t} = \frac{1}{1-\beta_t} \tag{9}$$

Momentum cooldown replaces this with:

$$\tau_{\text{CD},t} = \max(1, \min\{\tau_{\text{mom},t}, \tau_{\eta,t}\}), \qquad \beta_{\text{CD},t} = 1 - \frac{1}{\tau_{\text{CD},t}} \tag{10}$$

### Weight Decay Schedules

**Uniform**: $\lambda_t^{\text{uniform}} = \frac{c(f)}{S}$ (constant across updates) (Eq. 6)

**Log-time**: $\lambda_t^{\log} = \frac{a(f)}{\tau + t}$ (decreases over training) (Eq. 7)

where $S$ is the total number of optimizer updates and $\tau = 0.1 S_{1\times}$.

### Metrics

- **Equivalent OT factor**: The OT factor the baseline requires to match the indicated optimizer's loss
- **Token multiplier**: Ratio of tokens required by baseline to those used by the indicated optimizer; values above one favor the indicated optimizer
- **Outscaling**: When token multiplier increases with OT factor

## Empirical Validation / Results

### 1. Optimizer Rankings Across Training Horizons

Figure 1 and 2 show clear crossovers in optimizer rankings as training extends beyond the Chinchilla-optimal horizon:

- Muon and SOAP lead at short horizons
- ADANA closes its initial gap as OT factor increases
- SOAP overtakes Muon at the longest measured horizons for 51M and 124M models

### 2. Horizon-Tuned Fixed Memory

**Longer horizons favor longer memory** (Figure 3):

| Optimizer | Optimal M at 1× OT | Optimal M at 128× OT |
|-----------|-------------------|---------------------|
| AdamW | 20 | 1280 |
| Muon | 20 | 640 |
| SOAP | 40 | 1280 |

Key findings:
- Optimal base learning rate decreases as memory grows
- Per-horizon memory tuning improves final validation loss by up to 3.6 mpt at 128× OT
- ADANA's advantage persists even against AdamW with horizon-tuned memory

### 3. Log-Time Weight Decay Effects

The preferred weight decay schedule depends on both optimizer and training horizon:

- **ADANA**: Increasingly benefits from log-time weight decay as OT grows
- **SOAP**: Modestly benefits
- **AdamW**: Little consistent effect
- **Muon**: Harms at high OT factors

### 4. Momentum Cooldown Results

- Little consistent effect on fixed-memory AdamW
- Substantially improves ADANA at long horizons
- Gains compound when combined with log-time weight decay
- ADANA token multiplier appears approximately straight on log–log axes

### 5. Outscaling Analysis

**ADANA vs. AdamW** (Figure 7): Fitted equivalent-OT exponents range from **1.15 to 1.20** across all three model sizes and both AdamW baselines (uniform and log-time weight decay), close to the PLRF prediction $2 - \kappa = 1.15$ for $\kappa = 0.85$.

**Muon and SOAP vs. AdamW** (Figure 8):

| Optimizer | Token Multiplier Range | Behavior |
|-----------|----------------------|----------|
| Muon | 1.4×–1.7× | Approximately constant |
| SOAP | 1.3×–1.7× | Constant, rises to ~1.9× at 128× OT |

**ADANA vs. Matrix-preconditioned optimizers**:

| OT Factor | ADANA vs. Muon | ADANA vs. SOAP |
|-----------|---------------|----------------|
| 2× | 0.59×–0.63× | 0.63×–0.69× |
| Highest OT | Surpasses Muon | Competitive with SOAP |

## Theoretical and Practical Implications

### Theoretical Implications

1. **DANA theory validation**: The empirical equivalent-OT exponent of 1.15–1.20 closely matching the $2 - \kappa$ prediction provides evidence that power-law random features (PLRF) theory transfers to adaptive optimizers and transformers.

2. **Complementary roles**: The results suggest preconditioning (matrix-level) and scheduled memory provide complementary benefits—preconditioning gives constant efficiency multipliers, while scheduled memory improves scaling exponents.

3. **Floor–decay tradeoff**: Analysis shows ADANA's fitted token-decay exponent advantage remains positive even where high-token limits favor AdamW, identifying the mechanism as improved token-decay exponent rather than lower loss limit.

### Practical Implications

1. **Hyperparameter scaling rules**: Practitioners training beyond compute-optimal horizons need to scale:
   - Weight decay coefficient as $\sqrt{f}$
   - Memory windows longer with increasing OT factor
   - Learning rates lower as memory grows

2. **Optimizer selection depends on horizon**: The best optimizer at Chinchilla-optimal training may not be best for overtrained deployment models.

3. **Momentum cooldown**: A simple, effective intervention for scheduled-memory optimizers that bounds memory during terminal learning rate decay.

## Conclusion

### Main Takeaways

1. **Training horizon is an essential experimental axis** for optimizer evaluation—single-horizon comparisons can miss crossover behavior and reverse rankings.

2. **Scheduled memory is a promising direction**: ADANA's relative efficiency improves with additional training, unlike fixed-memory optimizers.

3. **Preconditioning and memory scheduling are complementary**: Matrix preconditioning provides constant gains; scheduled memory provides growing gains.

### Limitations and Future Directions

- **Model scale**: Results span 51M–253M parameters; extension to larger models needed
- **Batch size sensitivity**: ADANA begins losing efficiency at smaller batch sizes than AdamW at 8× OT; batch dependence requires fully controlled study
- **Architecture transfer**: Preliminary experiments suggest scheduled memory can work under another preconditioner ordering (reversing adaptive preconditioning and scheduled momentum)
- **Higher OT factors**: Whether ADANA's outscaling continues beyond 256× OT remains uncertain

### Final Statement

> "Treating training horizon as an essential experimental axis reveals optimizer behavior hidden by single-horizon comparisons and suggests that scheduling optimizer memory is a promising route to methods whose relative efficiency improves with additional training."

---

_Markdown view of https://picx.dev/p/5eLVmD, served by PicX — AI-generated visual whiteboard summaries of research papers._