Summary (Overview)

• OrbitQuant is a calibration-free weight-activation quantizer for diffusion transformers (DiTs) that replaces per-timestep range calibration with a single distributional codebook applied in a shared, rotated, normalized basis.
• It uses a randomized permuted block-Hadamard (RPBH) rotation to map activations into a fixed, known marginal distribution $f_d \approx \mathcal{N}(0,1/d)$ regardless of timestep, prompt, or guidance branch, enabling a single Lloyd–Max codebook per input dimension to be applied across all layers and denoising steps.
• The rotation is folded into weight rows offline and cancels in the linear layer matrix product; only a forward RPBH rotation on activations is needed at runtime, adding negligible overhead.
• State-of-the-art PTQ results are achieved on image DiTs (FLUX.1, Z-Image-Turbo) and video DiTs (Wan 2.1, CogVideoX) at low bit-widths, including the first practical PTQ at W2A4 where prior methods collapse.
• The same recipe transfers from image to video without per-modality tuning and sets new records on GenEval and VBench benchmarks.

Introduction and Theoretical Foundation

Background and Motivation

Diffusion transformers (DiTs) have become the backbone of state-of-the-art image and video generation, but their iterative denoising and growing parameter counts make inference expensive. Post-training quantization (PTQ) of both weights and activations is a natural remedy, but existing methods rely on calibration data to capture activation statistics that shift across timesteps, prompts, and classifier-free-guidance (CFG) branches. This forces re-calibration for every new checkpoint, resolution, or modality.

Theoretical Basis

Two key ingredients are inherited from TurboQuant (a prior KV-cache quantizer):

1. Haar-random orthogonal rotation: For a normalized vector $\tilde{\mathbf{x}} = \mathbf{x} / \|\mathbf{x}\|_2$, applying a Haar-random rotation $\mathbf{\Phi}_d \in \mathbb{R}^{d \times d}$ makes each coordinate follow the fixed marginal (for $d \ge 64$, approximated by $\mathcal{N}(0, 1/d)$): $$f_d(t) = \frac{\Gamma(d/2)}{\sqrt{\pi}\,\Gamma((d-1)/2)}\,(1-t^2)^{(d-3)/2}, \quad t \in [-1, 1]$$
2. Lloyd–Max codebook: An MSE-optimal scalar quantizer precomputed offline from $f_d$, with $2^b$ centroids $\mathcal{C}^{(d,b)}$, applied coordinate-wise as: $$\hat{q}_b^{(d)}(t) = \arg\min_{c \in \mathcal{C}^{(d,b)}} |t - c|$$

Key Insight

OrbitQuant extends this idea to both weights and activations inside each linear layer, using a shared rotation that cancels in the product. This eliminates the need for calibration: one codebook per dimension serves all timesteps, prompts, layers, and both image and video DiTs.

Methodology

Overview

OrbitQuant replaces per-input range calibration with a distributional quantizer applied in one shared, rotated, normalized basis. The rotation is absorbed into weights offline and applied forward on activations online. The pipeline uses the randomized permuted block-Hadamard (RPBH) transform for efficient implementation.

Offline Weight Quantization

For each linear layer with weight matrix $\mathbf{W} \in \mathbb{R}^{m \times d}$ and the shared rotation $\mathbf{\Pi}_d$ (orthogonal, dimension $d$):

1. Rotate weights: $\mathbf{W}' = \mathbf{W} \mathbf{\Pi}_d^\top$
2. Split each row into norm and direction: $r_i' = \|\mathbf{w}_i'\|_2,\; \tilde{\mathbf{w}}_i' = \mathbf{w}_i' / r_i'$
3. Quantize direction with Lloyd–Max codebook and reattach norms: $$\hat{\mathbf{W}}' = \mathrm{diag}(\mathbf{r}') \cdot \hat{Q}_{b_w}^{(d)}(\tilde{\mathbf{W}}')$$
4. Store row-norms $\mathbf{r}'$ in BF16 (negligible overhead) and replace $\mathbf{W}$ with $\hat{\mathbf{W}}'$ in the model.

Online Activation Quantization

At inference, for each incoming activation $\mathbf{x} \in \mathbb{R}^d$:

1. Rotate: $\mathbf{x}' = \mathbf{\Pi}_d \mathbf{x}$ (applied as $\mathbf{x}\mathbf{\Pi}_d^\top$ for token batches)
2. Normalize: $s = \|\mathbf{x}'\|_2,\; \tilde{\mathbf{x}}' = \mathbf{x}' / (s + \varepsilon)$
3. Quantize direction and rescale: $\hat{\mathbf{x}}' = s \cdot \hat{Q}_{b_a}^{(d)}(\tilde{\mathbf{x}}')$

The rotations cancel in the product: $\hat{\mathbf{W}}' \hat{\mathbf{x}}' \approx \mathbf{W}\mathbf{\Pi}_d^\top \mathbf{\Pi}_d \mathbf{x} = \mathbf{W} \mathbf{x}$, with no inverse rotation needed.

Randomized Permuted Block-Hadamard (RPBH) Rotation

The rotation $\mathbf{\Pi}_d$ is implemented as:

$$\mathbf{\Pi}_d = \mathrm{blkdiag}(\mathbf{H}_h \mathbf{D}_1, \ldots, \mathbf{H}_h \mathbf{D}_{d/h}) \cdot \mathbf{P}_\pi$$

where $\mathbf{H}_h$ is a $h \times h$ Walsh–Hadamard matrix, $\mathbf{D}_i$ are Rademacher sign diagonals, and $\mathbf{P}_\pi$ is a uniform random permutation matrix. This admits an $\mathcal{O}(d \log h)$ transform via Fast Walsh–Hadamard Transform and constructs on any dimension $d$ (block size $h$ is largest power of two dividing $d$).

The random permutation $\mathbf{P}_\pi$ is crucial: it spreads potential outliers across blocks, keeping the marginal close to $f_d$ even at low bit-width. Proposition 1 (in paper) gives a probabilistic variance concentration bound. A fixed permutation (drawn once per dimension) suffices; no calibration or data-dependent selection is needed.

Data-Agnostic Codebook

Because RPBH ensures each rotated coordinate of a normalized vector follows approximately the same fixed marginal $f_d$, the same Lloyd–Max codebook $\mathcal{C}^{(d,b)}$ works for:

• All timesteps and prompts
• All layers of dimension $d$
• Both weight rows and activation tokens
• Both image and video DiTs

This is what makes OrbitQuant completely calibration-free: no model evaluation or calibration data is required at quantizer construction time.

Empirical Validation / Results

Image Generation: GenEval (Table 1)

Table 1: GenEval scores on three image DiTs at W4A4 and W2A4.

At W4A4, OrbitQuant is essentially lossless, exceeding FP16 on Overall on FLUX.1-schnell and Z-Image-Turbo, and setting state-of-the-art among PTQ methods. At W2A4, it is the only method that produces usable images – baselines collapse to near-zero scores.

Video Generation: VBench (Table 2)

Table 2: VBench scores on Wan 2.1-1.3B and CogVideoX-2B at W4A6 and W4A4.

OrbitQuant achieves highest Overall Consistency under both bit-widths, and leads on most per-dimension scores, using the identical recipe as for image DiTs – no tuning per modality.

Qualitative Results (Figure 4)

• At W3A3, OrbitQuant stays close to BF16 on FLUX.1-dev, FLUX.1-schnell, and Z-Image-Turbo, while QuaRot and ViDiT-Q show artifacts or collapse.
• At W4A4 on Wan 14B, OrbitQuant preserves scene layout and temporal consistency across 81 frames; other methods drift in color and structure.

Latency and Memory Analysis (Figure 5)

• On FLUX.1-dev (H100, 1024², 50 steps) and Wan 2.1-1.3B, OrbitQuant has the lowest latency overhead among weight-and-activation PTQ methods (SmoothQuant: 1.09× slower, QuaRot: 1.28×, ViDiT-Q: 1.40×).
• Peak memory matches unquantized model on image; slightly higher on video due to index/gather tensors but still below ViDiT-Q.
• The efficient RPBH transform (vs. dense Haar) reduces activation rotation cost by ~26×.

Rotation Ablation (Table 3)

RPBH achieves the best quality at low bit-width (W2A4) among all rotations, with latency comparable to structured transforms. Removing the random permutation (Block-RHT) degrades performance, confirming its importance.

AdaLN Bit-Width Ablation (Figure 6)

Quantizing AdaLN modulation weights to INT4 (instead of BF16) nearly matches full-precision GenEval performance, while boosting compression from 2.21× to 4× on FLUX architecture. Further quantization to W2 degrades FLUX models, so OrbitQuant keeps AdaLN at INT4.

Theoretical and Practical Implications

Theoretical Significance

• Provides a formal justification for using random rotations (not just data-dependent permutations) to enable distributional codebooks for DiT quantization, with Proposition 1 offering a probabilistic variance bound.
• Shifts the paradigm from per-input range calibration to distributional codebook design, eliminating the need for model evaluation at quantizer construction.
• The shared-rotation design (folding into weights) is computationally elegant – rotations cancel in the product, requiring no inverse transform at inference.

Practical Implications

• Calibration-free operation: no need to collect prompts, timesteps, or CFG branches for each new model or resolution. This dramatically simplifies deployment pipelines.
• Cross-modality transfer: the same recipe works for both image and video DiTs without per-modality tuning, reducing engineering effort.
• Usable at extreme bit-widths: W2A4 is achieved for the first time in PTQ of DiTs, enabling significant memory and compute savings while maintaining generation quality.
• Low runtime overhead: the fixed codebook lookup and efficient RPBH rotation (O(d log h)) add minimal latency, making deployment practical.

Conclusion

OrbitQuant presents a calibration-free, data-agnostic quantization framework for diffusion transformers that replaces per-timestep range calibration with a single distributional codebook applied in a shared rotated basis. The key innovations are:

1. RPBH rotation with a uniform random permutation that keeps rotated coordinates close to a known fixed marginal, enabling a Lloyd–Max codebook that works for all inputs.
2. Weight-activation sharing: the same rotation is absorbed into weights offline and applied to activations online, canceling in the product and requiring only a forward transform at runtime.
3. Empirical success: state-of-the-art PTQ results at W4A4 and W2A4 on GenEval and VBench, with the only method maintaining generation quality at 2-bit weights. The same recipe transfers from image to video without per-modality tuning.

Future Directions

• Extending the approach to other architectures (e.g., U-Net-based diffusion models, autoregressive transformers).
• Hardware implementation of the RPBH transform for real-time inference.
• Combining with adaptive bit-allocation or mixed-precision schemes for further compression.
• Exploring the theoretical limits of distributional codebooks under even lower bit-widths (e.g., binary or ternary quantization).