# Delta-Matching: Closing the Final Gap of Native 8-bit Training for LLMs

> Delta-Matching restores the zero-row-sum invariant in FP8 attention training, matching BF16/FP32 performance by fixing the stale-delta failure that degrades larger models.

- **Source:** [arXiv](https://arxiv.org/abs/2609.37852)
- **Published:** 2026-11-21
- **Permalink:** https://picx.dev/p/CCloeF
- **Whiteboard:** https://picx.dev/p/CCloeF/image

## Summary

## Summary (Overview)

- **Core Problem**: FP8 attention training for LLMs fails due to a "stale delta" issue—a saved-output shortcut in the backward pass becomes invalid when forward and backward operands are separately rounded, violating the softmax gradient's zero-row-sum invariant.
- **Key Contribution**: The paper proposes **Delta-Matching**, a method that derives the backward correction from unquantized probabilities and FP32 products of FP8 backward operands, restoring the zero-row-sum invariant under stated numerical assumptions.
- **Empirical Finding**: Stale-delta degradation is scale-dependent—modest at 569M parameters but substantial at 1.67B and 5.29B parameters, suggesting accumulated optimization error that small models and short runs can conceal.
- **Architecture Coverage**: Delta-Matching works across GDN/GQA, GDN/MLA, KDA/GQA, and Mamba-2/GQA hybrids, as well as during 8K-to-64K context extension, matching BF16/FP32 mixed-precision training loss and downstream performance.
- **Practical Impact**: Enables native block-scaled FP8 in all seven attention-core matmuls (two forward, five backward) without architectural changes, smaller global batches, or auxiliary forward outputs.

## Introduction and Theoretical Foundation

### Background and Motivation

FP8 training promises roughly **2× BF16's peak Tensor Core throughput** on Hopper hardware (NVIDIA, 2023). While existing systems successfully quantize training-state storage (FP8-LM, COAT) and linear-layer matmuls (DeepSeek-V3, Ling-1T), **reliable attention training remains unresolved** across architectures, scales, tokens per step, and training durations.

The paper identifies a fundamental issue in FlashAttention's backward pass: the saved-output correction $\delta_i = \langle dO_i, O_i\rangle$ (Dao, 2024) assumes exact arithmetic consistency between forward and backward passes. When FP8 quantization is applied, this shortcut breaks, producing what the authors term **"stale delta."**

### The Zero-Row-Sum Invariant

The softmax backward pass computes:

$$dS = P \odot (dP - \delta)$$

where $\delta_i = \sum_j P_{ij} dP_{ij}$ is broadcast across row $i$. Since $\sum_j P_{ij} = 1$ in exact arithmetic:

$$\sum_j dS_{ij} = \sum_j P_{ij}(dP_{ij} - \delta_i) = \delta_i - \delta_i \sum_j P_{ij} = 0. \tag{1}$$

This invariant expresses **softmax shift invariance**—adding a constant to a score row leaves probabilities unchanged. It also implies $\sum_j dK_{jc} = 0$ across tokens in each channel $c$ for keys entering $S = QK^\top/\sqrt{d}$.

### Violation Under Quantization

Under FP8 quantization, the stale-delta row sum becomes:

$$\sum_{j}dS_{ij}^{\text{stale}} = \sum_{k}\widehat{dO}_{ik}\Big(\sum_{j}(\underbrace{P_{ij} - \widehat{P}_{ij}}_{\substack{\text{probability}\\ \text{quantization error}}})\widehat{V}_{jk}\Big) + \sum_{k}(\underbrace{\widehat{dO}_{ik} - dO_{ik}}_{\substack{\text{gradient}\\ \text{quantization error}}})\Big(\sum_{j}\widehat{P}_{ij}\widehat{V}_{jk}\Big). \tag{2}$$

Two distinct error sources contribute: **probability quantization error** (from $P$ vs. $\widehat{P}$) and **gradient quantization error** (from $dO$ vs. $\widehat{dO}$). These errors produce stale delta when they don't cancel.

## Methodology

### Delta-Matching Formulation

Building on the direct backward correction concept (Qiu & Yao, 2026), Delta-Matching uses **FP32 probabilities before quantization** and **FP32 products of FP8 backward operands**. The matched delta is:

$$\delta_{i}^{\text{matched}} = \sum_{j} P_{ij} \widehat{dP}_{ij}. \tag{4}$$

This restores the invariant:

$$\sum_{j} dS_{ij}^{\text{matched}} = \sum_{j} P_{ij}(\widehat{dP}_{ij} - \delta_{i}^{\text{matched}}) = \delta_{i}^{\text{matched}}\Big(1 - \sum_{j} P_{ij}\Big) = 0. \tag{5}$$

The key insight is that merely matching output-gradient precision (consistent $dO$) is insufficient—it removes the gradient-quantization term but leaves the probability-quantization mismatch:

$$\sum_{j}P_{ij}\bigl(\widehat{dP}_{ij} - \langle \widehat{dO}_{i},\widehat{O}_{i}\rangle \bigr) = \sum_{k}\widehat{dO}_{ik}\Bigl(\sum_{j}(\underbrace{P_{ij} - \widehat{P}_{ij}}_{\substack{\text{probability}\\ \text{quantization error}}})\widehat{V}_{jk}\Bigr). \tag{3}$$

### Experimental Setup

- **Base model**: 1.67B-parameter GDN/GQA hybrid (18 Gated DeltaNet layers + 6 GQA layers), pretrained on 30B Nemotron-CC tokens at 8K context
- **Attention configuration**: 16 query heads, 4 KV heads, dimension 128
- **Evaluation**: BF16 evaluation with FlashAttention-2; validation CE, RULER tasks (4K/8K), extraction recall (SWDE/FDA/SQuAD), TriviaQA, MMLU, and 9 commonsense benchmarks (CS9)
- **Precision scopes**: P1 (BF16 attention, FP8 FFNs), P4 (full FP8 including attention and projections), P0/P2 (BF16 FFNs and projections)
- **Baselines**: cuDNN/TE FP8, naive FP8 (stale delta), consistent dO, and BF16/FP32 mixed-precision reference

## Empirical Validation / Results

### Main Results (1.67B GDN/GQA, Default Configuration)

| Method | Validation CE ↓ | RULER-4K | RULER-8K | Extraction | TriviaQA | MMLU | CS9 |
|---|---|---|---|---|---|---|---|
| BF16/FP32 MP | $1.4178 \pm 0.0004$ | $61.5$ | $52.3$ | $65.7$ | $15.5$ | $35.4$ | $58.8$ |
| cuDNN/TE FP8 | $1.6105 \pm 0.0600$ | $38.8$ | $31.0$ | $57.4$ | $7.0$ | $28.5$ | $53.2$ |
| Naive FP8 (Stale Delta) | $1.8970 \pm 0.0296$ | $24.8$ | $19.8$ | $44.7$ | $2.7$ | $26.3$ | $47.0$ |
| Consistent dO | $1.4667 \pm 0.0175$ | $54.4$ | $43.2$ | $63.6$ | $12.9$ | $33.0$ | $57.3$ |
| **Delta-Matching** | **$1.4162 \pm 0.0006$** | **$61.4$** | **$47.8$** | **$68.0$** | **$15.6$** | **$34.0$** | **$59.0$** |

### Scale Effects

| Parameters | Method | Validation CE ↓ | RULER-8K | CS9 |
|---|---|---|---|---|
| 0.569B | BF16/FP32 MP | $1.5233$ | $37.8$ | $54.7$ |
| 0.569B | Naive FP8 | $1.5399$ | $40.6$ | $53.8$ |
| 0.569B | **Delta-Matching** | **$1.5228$** | $38.2$ | **$54.6$** |
| 5.289B | BF16/FP32 MP | $1.3349$ | $53.5$ | $63.1$ |
| 5.289B | Naive FP8 | $1.9327$ | $16.3$ | $45.4$ |
| 5.289B | **Delta-Matching** | **$1.3374$** | **$58.6$** | **$62.0$** |

### Architecture Results

- **GDN/MLA**: Stale delta raises validation CE by 0.5090; Delta-Matching stays within 0.0006 of reference
- **KDA/GQA**: Stale delta raises validation CE by 0.4230; Delta-Matching stays within 0.0006 of reference
- **Mamba-2/GQA**: Stale delta shows no substantial gap at 1.67B/30B tokens (CE 1.4191 vs. 1.4152), but layer probes reveal severe gain corruption masked by compensation elsewhere

### Context Extension (8K→64K)

During continued pretraining at lower learning rate ($8 \times 10^{-4}$), stale delta's loss gap exceeds 0.01 only after ~19,000 optimizer steps (20B tokens). Delta-Matching tracks BF16/FP32 on RULER at 64K (45.6% vs. 46.0%) and outperforms both FP8 baselines on all recall metrics.

### Training Dynamics Analysis

Stale delta causes:
1. **Row sharpening/flattening**: Biases logit updates toward sharpening when $\varepsilon_i < 0$ and flattening when $\varepsilon_i > 0$
2. **Channel gain amplification**: Selected channels show consistent upward pressure on key RMSNorm gains, with a few channels accounting for >99% of $\sum_c (g_{q,c}g_{k,c})^2$ in affected layers
3. **Selective amplification compounding**: Degradation accelerates with scale (5.29B shows steeper gain growth than 1.67B over the same steps)

## Theoretical and Practical Implications

### Theoretical Contributions

- **Identifies a novel failure mechanism** in FP8 attention training: the stale-delta problem arising from forward-backward numerical inconsistency
- **Provides a rigorous proof** that Delta-Matching restores the zero-row-sum invariant under stated numerical assumptions (normalized P, identical effective $\widehat{V}$ in both passes, no accumulation/output-storage/reduction rounding)
- **Demonstrates that consistent output-gradient precision alone is insufficient**—the probability-quantization mismatch persists and degrades training

### Practical Implications

- **Enables fully native 8-bit LLM training**: All seven attention-core matmuls can use FP8 without architectural changes, smaller global batches, or auxiliary forward outputs
- **Performance parity**: Delta-Matching matches BF16/FP32 mixed-precision training loss and downstream performance across tested configurations
- **Hardware efficiency**: The kernel achieves lower combined forward/backward latency on H100 than Hopper-optimized BF16 FlashAttention-3
- **Mitigation limitations**: QK normalization, NoPE, larger head dimensions, and lower-learning-rate context extension delay but don't eliminate stale-delta degradation

## Conclusion

The paper establishes that **forward-backward numerical consistency is essential for reliable 8-bit training**. Stale delta's violation of the zero-row-sum invariant produces accumulated optimization error that:
- Is concealed by small models (569M) and short runs
- Emerges at larger scales (1.67B, 5.29B) and longer durations
- Persists across hybrid architectures (GDN/GQA, GDN/MLA, KDA/GQA)
- Can be masked by architectures like Mamba-2/GQA despite internal corruption

**Delta-Matching** resolves these issues by deriving the correction from unquantized probabilities and FP32 products of FP8 backward operands, enabling native FP8 in all attention-core matmuls while matching BF16/FP32 training loss and downstream performance. Future work (Appendix A.4) addresses remaining limitations including broader architecture coverage and longer training horizons.

---

_Markdown view of https://picx.dev/p/CCloeF, served by PicX — AI-generated visual whiteboard summaries of research papers._
