Summary (Overview)
- Core Problem: FP8 attention training for LLMs fails due to a "stale delta" issue—a saved-output shortcut in the backward pass becomes invalid when forward and backward operands are separately rounded, violating the softmax gradient's zero-row-sum invariant.
- Key Contribution: The paper proposes Delta-Matching, a method that derives the backward correction from unquantized probabilities and FP32 products of FP8 backward operands, restoring the zero-row-sum invariant under stated numerical assumptions.
- Empirical Finding: Stale-delta degradation is scale-dependent—modest at 569M parameters but substantial at 1.67B and 5.29B parameters, suggesting accumulated optimization error that small models and short runs can conceal.
- Architecture Coverage: Delta-Matching works across GDN/GQA, GDN/MLA, KDA/GQA, and Mamba-2/GQA hybrids, as well as during 8K-to-64K context extension, matching BF16/FP32 mixed-precision training loss and downstream performance.
- Practical Impact: Enables native block-scaled FP8 in all seven attention-core matmuls (two forward, five backward) without architectural changes, smaller global batches, or auxiliary forward outputs.
Introduction and Theoretical Foundation
Background and Motivation
FP8 training promises roughly 2× BF16's peak Tensor Core throughput on Hopper hardware (NVIDIA, 2023). While existing systems successfully quantize training-state storage (FP8-LM, COAT) and linear-layer matmuls (DeepSeek-V3, Ling-1T), reliable attention training remains unresolved across architectures, scales, tokens per step, and training durations.
The paper identifies a fundamental issue in FlashAttention's backward pass: the saved-output correction (Dao, 2024) assumes exact arithmetic consistency between forward and backward passes. When FP8 quantization is applied, this shortcut breaks, producing what the authors term "stale delta."
The Zero-Row-Sum Invariant
The softmax backward pass computes:
where is broadcast across row . Since in exact arithmetic:
This invariant expresses softmax shift invariance—adding a constant to a score row leaves probabilities unchanged. It also implies across tokens in each channel for keys entering .
Violation Under Quantization
Under FP8 quantization, the stale-delta row sum becomes:
Two distinct error sources contribute: probability quantization error (from vs. ) and gradient quantization error (from vs. ). These errors produce stale delta when they don't cancel.
Methodology
Delta-Matching Formulation
Building on the direct backward correction concept (Qiu & Yao, 2026), Delta-Matching uses FP32 probabilities before quantization and FP32 products of FP8 backward operands. The matched delta is:
This restores the invariant:
The key insight is that merely matching output-gradient precision (consistent ) is insufficient—it removes the gradient-quantization term but leaves the probability-quantization mismatch:
Experimental Setup
- Base model: 1.67B-parameter GDN/GQA hybrid (18 Gated DeltaNet layers + 6 GQA layers), pretrained on 30B Nemotron-CC tokens at 8K context
- Attention configuration: 16 query heads, 4 KV heads, dimension 128
- Evaluation: BF16 evaluation with FlashAttention-2; validation CE, RULER tasks (4K/8K), extraction recall (SWDE/FDA/SQuAD), TriviaQA, MMLU, and 9 commonsense benchmarks (CS9)
- Precision scopes: P1 (BF16 attention, FP8 FFNs), P4 (full FP8 including attention and projections), P0/P2 (BF16 FFNs and projections)
- Baselines: cuDNN/TE FP8, naive FP8 (stale delta), consistent dO, and BF16/FP32 mixed-precision reference
Empirical Validation / Results
Main Results (1.67B GDN/GQA, Default Configuration)
| Method | Validation CE ↓ | RULER-4K | RULER-8K | Extraction | TriviaQA | MMLU | CS9 |
|---|---|---|---|---|---|---|---|
| BF16/FP32 MP | |||||||
| cuDNN/TE FP8 | |||||||
| Naive FP8 (Stale Delta) | |||||||
| Consistent dO | |||||||
| Delta-Matching |
Scale Effects
| Parameters | Method | Validation CE ↓ | RULER-8K | CS9 |
|---|---|---|---|---|
| 0.569B | BF16/FP32 MP | |||
| 0.569B | Naive FP8 | |||
| 0.569B | Delta-Matching | |||
| 5.289B | BF16/FP32 MP | |||
| 5.289B | Naive FP8 | |||
| 5.289B | Delta-Matching |
Architecture Results
- GDN/MLA: Stale delta raises validation CE by 0.5090; Delta-Matching stays within 0.0006 of reference
- KDA/GQA: Stale delta raises validation CE by 0.4230; Delta-Matching stays within 0.0006 of reference
- Mamba-2/GQA: Stale delta shows no substantial gap at 1.67B/30B tokens (CE 1.4191 vs. 1.4152), but layer probes reveal severe gain corruption masked by compensation elsewhere
Context Extension (8K→64K)
During continued pretraining at lower learning rate (), stale delta's loss gap exceeds 0.01 only after ~19,000 optimizer steps (20B tokens). Delta-Matching tracks BF16/FP32 on RULER at 64K (45.6% vs. 46.0%) and outperforms both FP8 baselines on all recall metrics.
Training Dynamics Analysis
Stale delta causes:
- Row sharpening/flattening: Biases logit updates toward sharpening when and flattening when
- Channel gain amplification: Selected channels show consistent upward pressure on key RMSNorm gains, with a few channels accounting for >99% of in affected layers
- Selective amplification compounding: Degradation accelerates with scale (5.29B shows steeper gain growth than 1.67B over the same steps)
Theoretical and Practical Implications
Theoretical Contributions
- Identifies a novel failure mechanism in FP8 attention training: the stale-delta problem arising from forward-backward numerical inconsistency
- Provides a rigorous proof that Delta-Matching restores the zero-row-sum invariant under stated numerical assumptions (normalized P, identical effective in both passes, no accumulation/output-storage/reduction rounding)
- Demonstrates that consistent output-gradient precision alone is insufficient—the probability-quantization mismatch persists and degrades training
Practical Implications
- Enables fully native 8-bit LLM training: All seven attention-core matmuls can use FP8 without architectural changes, smaller global batches, or auxiliary forward outputs
- Performance parity: Delta-Matching matches BF16/FP32 mixed-precision training loss and downstream performance across tested configurations
- Hardware efficiency: The kernel achieves lower combined forward/backward latency on H100 than Hopper-optimized BF16 FlashAttention-3
- Mitigation limitations: QK normalization, NoPE, larger head dimensions, and lower-learning-rate context extension delay but don't eliminate stale-delta degradation
Conclusion
The paper establishes that forward-backward numerical consistency is essential for reliable 8-bit training. Stale delta's violation of the zero-row-sum invariant produces accumulated optimization error that:
- Is concealed by small models (569M) and short runs
- Emerges at larger scales (1.67B, 5.29B) and longer durations
- Persists across hybrid architectures (GDN/GQA, GDN/MLA, KDA/GQA)
- Can be masked by architectures like Mamba-2/GQA despite internal corruption
Delta-Matching resolves these issues by deriving the correction from unquantized probabilities and FP32 products of FP8 backward operands, enabling native FP8 in all attention-core matmuls while matching BF16/FP32 training loss and downstream performance. Future work (Appendix A.4) addresses remaining limitations including broader architecture coverage and longer training horizons.
Related papers
- Do Halting Heads Really Save Compute in Looped Language Models?
Halting heads in looped language models save far less compute than reported because mixture-trained heads rarely stop tokens and batching overhead eliminates most remaining gains.
- Do LLM Agents Still Need to Be Told How to Self-Improve?
Newer LLM generations spontaneously construct their own self-improvement algorithms from primitive tools alone, transferring learned strategies across tasks and models without prescribed learning loops.
- RIPPLE: Recovery Internalization through Process-Level Policy Learning and Effect-Aware Credit for Long-Horizon Agentic Coding
RIPPLE internalizes prefix-only recovery guidance into a single policy via stage-specific credit, improving coding agents by 4.8 points across four benchmarks without deployment-time critics.