Summary (Overview)

  • Core Problem: FP8 attention training for LLMs fails due to a "stale delta" issue—a saved-output shortcut in the backward pass becomes invalid when forward and backward operands are separately rounded, violating the softmax gradient's zero-row-sum invariant.
  • Key Contribution: The paper proposes Delta-Matching, a method that derives the backward correction from unquantized probabilities and FP32 products of FP8 backward operands, restoring the zero-row-sum invariant under stated numerical assumptions.
  • Empirical Finding: Stale-delta degradation is scale-dependent—modest at 569M parameters but substantial at 1.67B and 5.29B parameters, suggesting accumulated optimization error that small models and short runs can conceal.
  • Architecture Coverage: Delta-Matching works across GDN/GQA, GDN/MLA, KDA/GQA, and Mamba-2/GQA hybrids, as well as during 8K-to-64K context extension, matching BF16/FP32 mixed-precision training loss and downstream performance.
  • Practical Impact: Enables native block-scaled FP8 in all seven attention-core matmuls (two forward, five backward) without architectural changes, smaller global batches, or auxiliary forward outputs.

Introduction and Theoretical Foundation

Background and Motivation

FP8 training promises roughly 2× BF16's peak Tensor Core throughput on Hopper hardware (NVIDIA, 2023). While existing systems successfully quantize training-state storage (FP8-LM, COAT) and linear-layer matmuls (DeepSeek-V3, Ling-1T), reliable attention training remains unresolved across architectures, scales, tokens per step, and training durations.

The paper identifies a fundamental issue in FlashAttention's backward pass: the saved-output correction δi=⟨dOi,Oi⟩\delta_i = \langle dO_i, O_i\rangle (Dao, 2024) assumes exact arithmetic consistency between forward and backward passes. When FP8 quantization is applied, this shortcut breaks, producing what the authors term "stale delta."

The Zero-Row-Sum Invariant

The softmax backward pass computes:

dS=P⊙(dP−δ)dS = P \odot (dP - \delta)

where δi=∑jPijdPij\delta_i = \sum_j P_{ij} dP_{ij} is broadcast across row ii. Since ∑jPij=1\sum_j P_{ij} = 1 in exact arithmetic:

∑jdSij=∑jPij(dPij−δi)=δi−δi∑jPij=0.(1)\sum_j dS_{ij} = \sum_j P_{ij}(dP_{ij} - \delta_i) = \delta_i - \delta_i \sum_j P_{ij} = 0. \tag{1}

This invariant expresses softmax shift invariance—adding a constant to a score row leaves probabilities unchanged. It also implies ∑jdKjc=0\sum_j dK_{jc} = 0 across tokens in each channel cc for keys entering S=QK⊤/dS = QK^\top/\sqrt{d}.

Violation Under Quantization

Under FP8 quantization, the stale-delta row sum becomes:

∑jdSijstale=∑kdO^ik(∑j(Pij−P^ij⏟probabilityquantization error)V^jk)+∑k(dO^ik−dOik⏟gradientquantization error)(∑jP^ijV^jk).(2)\sum_{j}dS_{ij}^{\text{stale}} = \sum_{k}\widehat{dO}_{ik}\Big(\sum_{j}(\underbrace{P_{ij} - \widehat{P}_{ij}}_{\substack{\text{probability}\\ \text{quantization error}}})\widehat{V}_{jk}\Big) + \sum_{k}(\underbrace{\widehat{dO}_{ik} - dO_{ik}}_{\substack{\text{gradient}\\ \text{quantization error}}})\Big(\sum_{j}\widehat{P}_{ij}\widehat{V}_{jk}\Big). \tag{2}

Two distinct error sources contribute: probability quantization error (from PP vs. P^\widehat{P}) and gradient quantization error (from dOdO vs. dO^\widehat{dO}). These errors produce stale delta when they don't cancel.

Methodology

Delta-Matching Formulation

Building on the direct backward correction concept (Qiu & Yao, 2026), Delta-Matching uses FP32 probabilities before quantization and FP32 products of FP8 backward operands. The matched delta is:

δimatched=∑jPijdP^ij.(4)\delta_{i}^{\text{matched}} = \sum_{j} P_{ij} \widehat{dP}_{ij}. \tag{4}

This restores the invariant:

∑jdSijmatched=∑jPij(dP^ij−δimatched)=δimatched(1−∑jPij)=0.(5)\sum_{j} dS_{ij}^{\text{matched}} = \sum_{j} P_{ij}(\widehat{dP}_{ij} - \delta_{i}^{\text{matched}}) = \delta_{i}^{\text{matched}}\Big(1 - \sum_{j} P_{ij}\Big) = 0. \tag{5}

The key insight is that merely matching output-gradient precision (consistent dOdO) is insufficient—it removes the gradient-quantization term but leaves the probability-quantization mismatch:

∑jPij(dP^ij−⟨dO^i,O^i⟩)=∑kdO^ik(∑j(Pij−P^ij⏟probabilityquantization error)V^jk).(3)\sum_{j}P_{ij}\bigl(\widehat{dP}_{ij} - \langle \widehat{dO}_{i},\widehat{O}_{i}\rangle \bigr) = \sum_{k}\widehat{dO}_{ik}\Bigl(\sum_{j}(\underbrace{P_{ij} - \widehat{P}_{ij}}_{\substack{\text{probability}\\ \text{quantization error}}})\widehat{V}_{jk}\Bigr). \tag{3}

Experimental Setup

  • Base model: 1.67B-parameter GDN/GQA hybrid (18 Gated DeltaNet layers + 6 GQA layers), pretrained on 30B Nemotron-CC tokens at 8K context
  • Attention configuration: 16 query heads, 4 KV heads, dimension 128
  • Evaluation: BF16 evaluation with FlashAttention-2; validation CE, RULER tasks (4K/8K), extraction recall (SWDE/FDA/SQuAD), TriviaQA, MMLU, and 9 commonsense benchmarks (CS9)
  • Precision scopes: P1 (BF16 attention, FP8 FFNs), P4 (full FP8 including attention and projections), P0/P2 (BF16 FFNs and projections)
  • Baselines: cuDNN/TE FP8, naive FP8 (stale delta), consistent dO, and BF16/FP32 mixed-precision reference

Empirical Validation / Results

Main Results (1.67B GDN/GQA, Default Configuration)

MethodValidation CE ↓RULER-4KRULER-8KExtractionTriviaQAMMLUCS9
BF16/FP32 MP1.4178±0.00041.4178 \pm 0.000461.561.552.352.365.765.715.515.535.435.458.858.8
cuDNN/TE FP81.6105±0.06001.6105 \pm 0.060038.838.831.031.057.457.47.07.028.528.553.253.2
Naive FP8 (Stale Delta)1.8970±0.02961.8970 \pm 0.029624.824.819.819.844.744.72.72.726.326.347.047.0
Consistent dO1.4667±0.01751.4667 \pm 0.017554.454.443.243.263.663.612.912.933.033.057.357.3
Delta-Matching1.4162±0.00061.4162 \pm 0.000661.461.447.847.868.068.015.615.634.034.059.059.0

Scale Effects

ParametersMethodValidation CE ↓RULER-8KCS9
0.569BBF16/FP32 MP1.52331.523337.837.854.754.7
0.569BNaive FP81.53991.539940.640.653.853.8
0.569BDelta-Matching1.52281.522838.238.254.654.6
5.289BBF16/FP32 MP1.33491.334953.553.563.163.1
5.289BNaive FP81.93271.932716.316.345.445.4
5.289BDelta-Matching1.33741.337458.658.662.062.0

Architecture Results

  • GDN/MLA: Stale delta raises validation CE by 0.5090; Delta-Matching stays within 0.0006 of reference
  • KDA/GQA: Stale delta raises validation CE by 0.4230; Delta-Matching stays within 0.0006 of reference
  • Mamba-2/GQA: Stale delta shows no substantial gap at 1.67B/30B tokens (CE 1.4191 vs. 1.4152), but layer probes reveal severe gain corruption masked by compensation elsewhere

Context Extension (8K→64K)

During continued pretraining at lower learning rate (8×10−48 \times 10^{-4}), stale delta's loss gap exceeds 0.01 only after ~19,000 optimizer steps (20B tokens). Delta-Matching tracks BF16/FP32 on RULER at 64K (45.6% vs. 46.0%) and outperforms both FP8 baselines on all recall metrics.

Training Dynamics Analysis

Stale delta causes:

  1. Row sharpening/flattening: Biases logit updates toward sharpening when εi<0\varepsilon_i < 0 and flattening when εi>0\varepsilon_i > 0
  2. Channel gain amplification: Selected channels show consistent upward pressure on key RMSNorm gains, with a few channels accounting for >99% of ∑c(gq,cgk,c)2\sum_c (g_{q,c}g_{k,c})^2 in affected layers
  3. Selective amplification compounding: Degradation accelerates with scale (5.29B shows steeper gain growth than 1.67B over the same steps)

Theoretical and Practical Implications

Theoretical Contributions

  • Identifies a novel failure mechanism in FP8 attention training: the stale-delta problem arising from forward-backward numerical inconsistency
  • Provides a rigorous proof that Delta-Matching restores the zero-row-sum invariant under stated numerical assumptions (normalized P, identical effective V^\widehat{V} in both passes, no accumulation/output-storage/reduction rounding)
  • Demonstrates that consistent output-gradient precision alone is insufficient—the probability-quantization mismatch persists and degrades training

Practical Implications

  • Enables fully native 8-bit LLM training: All seven attention-core matmuls can use FP8 without architectural changes, smaller global batches, or auxiliary forward outputs
  • Performance parity: Delta-Matching matches BF16/FP32 mixed-precision training loss and downstream performance across tested configurations
  • Hardware efficiency: The kernel achieves lower combined forward/backward latency on H100 than Hopper-optimized BF16 FlashAttention-3
  • Mitigation limitations: QK normalization, NoPE, larger head dimensions, and lower-learning-rate context extension delay but don't eliminate stale-delta degradation

Conclusion

The paper establishes that forward-backward numerical consistency is essential for reliable 8-bit training. Stale delta's violation of the zero-row-sum invariant produces accumulated optimization error that:

  • Is concealed by small models (569M) and short runs
  • Emerges at larger scales (1.67B, 5.29B) and longer durations
  • Persists across hybrid architectures (GDN/GQA, GDN/MLA, KDA/GQA)
  • Can be masked by architectures like Mamba-2/GQA despite internal corruption

Delta-Matching resolves these issues by deriving the correction from unquantized probabilities and FP32 products of FP8 backward operands, enabling native FP8 in all attention-core matmuls while matching BF16/FP32 training loss and downstream performance. Future work (Appendix A.4) addresses remaining limitations including broader architecture coverage and longer training horizons.

Related papers