# Why Gated DeltaNet Survives 4-Bit Quantization: NVFP4 W4A4 for the Recurrent Half of a Hybrid 27B LLM

> NVFP4 W4A4 quantization matches BF16 accuracy on a hybrid 27B LLM because Gated DeltaNet's gating and delta-rule recurrence make it more robust to quantization than attention layers.

- **Source:** [arXiv](https://arxiv.org/abs/2609.04098)
- **Published:** 2026-09-05
- **Permalink:** https://picx.dev/p/Lj0mZI
- **Whiteboard:** https://picx.dev/p/Lj0mZI/image

## Summary

# Summary of "Why Gated DeltaNet Survives 4-Bit Quantization: NVFP4 W4A4 for the Recurrent Half of a Hybrid 27B LLM"

## Overview

- **Core claim**: A fully W4A4 NVFP4 quantized version of Qwen/*wen3.8-27B* (a hybrid model with 48 Gated DeltaNet layers and 16 attention layers) matches BF16 accuracy across all tested tasks, despite community consensus that the recurrent GDN block requires higher precision.
- **Key contribution**: The authors build "Minima", quantizing all 496 linear layers to NVFP4 W4A4, and demonstrate that the GDN block's gating structure and delta-rule recurrence make it *more* robust to quantization than attention layers, not less.
- **Mechanistic explanation**: Four-part chain showing that NVFP4 block scaling localizes outliers, gate nonlinearities compress noise, the delta rule actively erases state error, and the end-to-end quantization gap shrinks with context length.
- **Practical findings**: Identifies and repairs a global-scale mismatch in fused GEMM serving, shows calibrated FP8 KV-cache scales are performance-free, and provides a complete recipe for hybrid-model quantization.
- **Result**: Minima achieves 17.53 GiB weights (vs. 50.13 GiB BF16), 2.9× smaller, with 1,154 tok/s decode (vs. 621 BF16) and 4.03s TTFT at 32K (vs. 6.90s BF16).

## Introduction and Theoretical Foundation

### Background

The paper addresses two converging trends:
1. **W4A4 inference**: NVFP4 format uses E2M1 4-bit values with one E4M3 scale per 16-element block (set to blockmax/6) and one FP32 scale per tensor, enabling native 4-bit tensor core GEMMs.
2. **Hybrid architectures**: Models like Qwen3.8-27B use Gated Delta# DeltaNet (GDN) layers as the dominant mixer, with only 16 of 64 layers using full attention.

### Gated DeltaNet Mechanics

The GDN layer's recurrence is defined as:

$$S_t = \alpha_t S_{t-1} + \betaimport { beta_t k_t (v_t - S_{t-1}^{\top} k_t)^{\top}, \quad
quad o_t = S_t^{\top} (q_t / \sqrt{K})$$

where $\alpha_t$ is a per-token forget gate, $\beta_t$ scales a correction term, and the write replaces what the state### state currently predicts for key $k_t$ rather than accumulating blindly.

The gates are parameterized in log-space:

:
$$g_t = -\exp(A_{\log}) \cdot \text{softplus}(a_t + \# dt\_bias)$$

$$\package com.example

This log-space parameterization is crucial for training stability but,/**
 * Main application entry point
 * @param args command line arguments
 */
fun main(args: Array<String>) {
    println("Hello, World!")
}

---

_Markdown view of https://picx.dev/p/Lj0mZI, served by PicX — AI-generated visual whiteboard summaries of research papers._