# OrbitQuant: Data-Agnostic Quantization for Image and Video Diffusion Transformers

> OrbitQuant achieves calibration-free W2A4 quantization of diffusion transformers via a shared RPBH rotation and Lloyd–Max codebook, outperforming prior PTQ.

- **Source:** [arXiv](https://arxiv.org/abs/2607.02461)
- **Published:** 2026-07-07
- **Permalink:** https://picx.dev/p/B0mkuh
- **Whiteboard:** https://picx.dev/p/B0mkuh/image

## Summary

## Summary (Overview)

- **OrbitQuant** is a calibration-free weight-activation quantizer for diffusion transformers (DiTs) that replaces per-timestep range calibration with a single distributional codebook applied in a shared, rotated, normalized basis.
- It uses a **randomized permuted block-Hadamard (RPBH)** rotation to map activations into a fixed, known marginal distribution $f_d \approx \mathcal{N}(0,1/d)$ regardless of timestep, prompt, or guidance branch, enabling a single Lloyd–Max codebook per input dimension to be applied across all layers and denoising steps.
- The rotation is **folded into weight rows offline** and cancels in the linear layer matrix product; only a forward RPBH rotation on activations is needed at runtime, adding negligible overhead.
- **State-of-the-art PTQ results** are achieved on image DiTs (FLUX.1, Z-Image-Turbo) and video DiTs (Wan 2.1, CogVideoX) at low bit-widths, including the first practical PTQ at **W2A4** where prior methods collapse.
- The same recipe **transfers from image to video without per-modality tuning** and sets new records on GenEval and VBench benchmarks.

## Introduction and Theoretical Foundation

### Background and Motivation

Diffusion transformers (DiTs) have become the backbone of state-of-the-art image and video generation, but their iterative denoising and growing parameter counts make inference expensive. Post-training quantization (PTQ) of both weights and activations is a natural remedy, but existing methods rely on calibration data to capture activation statistics that **shift across timesteps, prompts, and classifier-free-guidance (CFG) branches**. This forces re-calibration for every new checkpoint, resolution, or modality.

### Theoretical Basis

Two key ingredients are inherited from TurboQuant (a prior KV-cache quantizer):

1. **Haar-random orthogonal rotation**: For a normalized vector $\tilde{\mathbf{x}} = \mathbf{x} / \|\mathbf{x}\|_2$, applying a Haar-random rotation $\mathbf{\Phi}_d \in \mathbb{R}^{d \times d}$ makes each coordinate follow the fixed marginal (for $d \ge 64$, approximated by $\mathcal{N}(0, 1/d)$):
   $$ f_d(t) = \frac{\Gamma(d/2)}{\sqrt{\pi}\,\Gamma((d-1)/2)}\,(1-t^2)^{(d-3)/2}, \quad t \in [-1, 1] $$
2. **Lloyd–Max codebook**: An MSE-optimal scalar quantizer precomputed offline from $f_d$, with $2^b$ centroids $\mathcal{C}^{(d,b)}$, applied coordinate-wise as:
   $$ \hat{q}_b^{(d)}(t) = \arg\min_{c \in \mathcal{C}^{(d,b)}} |t - c| $$

### Key Insight

OrbitQuant extends this idea to **both weights and activations inside each linear layer**, using a **shared rotation** that cancels in the product. This eliminates the need for calibration: one codebook per dimension serves all timesteps, prompts, layers, and both image and video DiTs.

## Methodology

### Overview

OrbitQuant replaces per-input range calibration with a distributional quantizer applied in one shared, rotated, normalized basis. The rotation is absorbed into weights offline and applied forward on activations online. The pipeline uses the **randomized permuted block-Hadamard (RPBH)** transform for efficient implementation.

### Offline Weight Quantization

For each linear layer with weight matrix $\mathbf{W} \in \mathbb{R}^{m \times d}$ and the shared rotation $\mathbf{\Pi}_d$ (orthogonal, dimension $d$):

1. Rotate weights: $\mathbf{W}' = \mathbf{W} \mathbf{\Pi}_d^\top$
2. Split each row into norm and direction: $r_i' = \|\mathbf{w}_i'\|_2,\; \tilde{\mathbf{w}}_i' = \mathbf{w}_i' / r_i'$
3. Quantize direction with Lloyd–Max codebook and reattach norms:
   $$ \hat{\mathbf{W}}' = \mathrm{diag}(\mathbf{r}') \cdot \hat{Q}_{b_w}^{(d)}(\tilde{\mathbf{W}}') $$
4. Store row-norms $\mathbf{r}'$ in BF16 (negligible overhead) and replace $\mathbf{W}$ with $\hat{\mathbf{W}}'$ in the model.

### Online Activation Quantization

At inference, for each incoming activation $\mathbf{x} \in \mathbb{R}^d$:

1. Rotate: $\mathbf{x}' = \mathbf{\Pi}_d \mathbf{x}$ (applied as $\mathbf{x}\mathbf{\Pi}_d^\top$ for token batches)
2. Normalize: $s = \|\mathbf{x}'\|_2,\; \tilde{\mathbf{x}}' = \mathbf{x}' / (s + \varepsilon)$
3. Quantize direction and rescale: $\hat{\mathbf{x}}' = s \cdot \hat{Q}_{b_a}^{(d)}(\tilde{\mathbf{x}}')$

The rotations cancel in the product: $\hat{\mathbf{W}}' \hat{\mathbf{x}}' \approx \mathbf{W}\mathbf{\Pi}_d^\top \mathbf{\Pi}_d \mathbf{x} = \mathbf{W} \mathbf{x}$, with no inverse rotation needed.

### Randomized Permuted Block-Hadamard (RPBH) Rotation

The rotation $\mathbf{\Pi}_d$ is implemented as:
$$ \mathbf{\Pi}_d = \mathrm{blkdiag}(\mathbf{H}_h \mathbf{D}_1, \ldots, \mathbf{H}_h \mathbf{D}_{d/h}) \cdot \mathbf{P}_\pi $$
where $\mathbf{H}_h$ is a $h \times h$ Walsh–Hadamard matrix, $\mathbf{D}_i$ are Rademacher sign diagonals, and $\mathbf{P}_\pi$ is a **uniform random permutation matrix**. This admits an $\mathcal{O}(d \log h)$ transform via Fast Walsh–Hadamard Transform and constructs on any dimension $d$ (block size $h$ is largest power of two dividing $d$).

The random permutation $\mathbf{P}_\pi$ is crucial: it spreads potential outliers across blocks, keeping the marginal close to $f_d$ even at low bit-width. **Proposition 1** (in paper) gives a probabilistic variance concentration bound. A fixed permutation (drawn once per dimension) suffices; no calibration or data-dependent selection is needed.

### Data-Agnostic Codebook

Because RPBH ensures each rotated coordinate of a normalized vector follows approximately the same fixed marginal $f_d$, the same Lloyd–Max codebook $\mathcal{C}^{(d,b)}$ works for:
- All timesteps and prompts
- All layers of dimension $d$
- Both weight rows and activation tokens
- Both image and video DiTs

This is what makes OrbitQuant **completely calibration-free**: no model evaluation or calibration data is required at quantizer construction time.

## Empirical Validation / Results

### Image Generation: GenEval (Table 1)

Table 1: GenEval scores on three image DiTs at W4A4 and W2A4.
| Model | Method | Bit | Single Object ↑ | Two Object ↑ | Counting ↑ | Colors ↑ | Position ↑ | Color Attribution ↑ | Overall ↑ |
|-------|--------|-----|-----------------|--------------|------------|----------|------------|---------------------|-----------|
| FLUX.1-schnell | FP16 | 16/16 | 0.997 | 0.884 | 0.600 | 0.742 | 0.275 | 0.488 | 0.664 |
| | AdaTSQ [52] | W4A4 | 0.997 | 0.894 | 0.622 | 0.793 | 0.278 | 0.498 | 0.680 |
| | **OrbitQuant** | **W4A4** | 0.991 | 0.881 | **0.706** | **0.803** | **0.323** | **0.512** | **0.703** |
| | **OrbitQuant** | **W2A4** | 0.972 | 0.697 | 0.575 | 0.766 | 0.198 | 0.420 | **0.604** |
| | QuaRot† [2] | W2A4 | 0.006 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.001 |
| FLUX.1-dev | FP16 | 16/16 | 0.984 | 0.823 | 0.769 | 0.771 | 0.203 | 0.450 | 0.667 |
| | **OrbitQuant** | **W4A4** | 0.988 | 0.768 | 0.691 | 0.755 | 0.178 | 0.420 | **0.633** |
| | **OrbitQuant** | **W2A4** | 0.956 | 0.424 | 0.481 | 0.678 | 0.110 | 0.203 | **0.475** |
| Z-Image-Turbo | FP16 | 16/16 | 1.000 | 0.907 | 0.709 | 0.859 | 0.468 | 0.583 | 0.754 |
| | **OrbitQuant** | **W4A4** | 0.997 | 0.889 | 0.781 | 0.888 | 0.450 | 0.598 | **0.767** |
| | **OrbitQuant** | **W2A4** | 0.703 | 0.194 | 0.275 | 0.500 | 0.128 | 0.113 | **0.319** |

At W4A4, OrbitQuant is essentially lossless, exceeding FP16 on Overall on FLUX.1-schnell and Z-Image-Turbo, and setting state-of-the-art among PTQ methods. At W2A4, it is the **only method that produces usable images** – baselines collapse to near-zero scores.

### Video Generation: VBench (Table 2)

Table 2: VBench scores on Wan 2.1-1.3B and CogVideoX-2B at W4A6 and W4A4.
| Model | Method | Bit | Imaging Quality ↑ | Aesthetic Quality ↑ | Motion Smoothness ↑ | Dynamic Degree ↑ | Background Consistency ↑ | Subject Consistency ↑ | Scene ↑ | Overall Consistency ↑ |
|-------|--------|-----|-------------------|---------------------|---------------------|------------------|--------------------------|------------------------|---------|-----------------------|
| Wan 2.1-1.3B | Full Prec. | 16/16 | 64.30 | 58.21 | 97.37 | 70.28 | 95.94 | 93.84 | 28.05 | 24.67 |
| | **OrbitQuant** | **W4A6** | **61.25** | **56.08** | **97.76** | **59.78** | 95.51 | **94.23** | **24.88** | **24.35** |
| | **OrbitQuant** | **W4A4** | **58.58** | **53.41** | **97.42** | 53.89 | 95.30 | **92.98** | **18.81** | **23.86** |
| CogVideoX-2B | Full Prec. | 16/16 | 59.15 | 54.49 | 97.43 | 67.78 | 94.79 | 92.82 | 36.24 | 25.06 |
| | **OrbitQuant** | **W4A6** | 55.59 | **54.42** | **97.02** | **57.50** | **94.78** | **92.56** | 32.51 | **24.55** |
| | **OrbitQuant** | **W4A4** | **52.62** | **51.66** | 96.99 | **42.78** | 94.50 | 91.65 | **28.53** | **23.86** |

OrbitQuant achieves highest Overall Consistency under both bit-widths, and leads on most per-dimension scores, using the **identical recipe as for image DiTs** – no tuning per modality.

### Qualitative Results (Figure 4)

- At **W3A3**, OrbitQuant stays close to BF16 on FLUX.1-dev, FLUX.1-schnell, and Z-Image-Turbo, while QuaRot and ViDiT-Q show artifacts or collapse.
- At **W4A4 on Wan 14B**, OrbitQuant preserves scene layout and temporal consistency across 81 frames; other methods drift in color and structure.

### Latency and Memory Analysis (Figure 5)

- On FLUX.1-dev (H100, 1024², 50 steps) and Wan 2.1-1.3B, OrbitQuant has the **lowest latency overhead** among weight-and-activation PTQ methods (SmoothQuant: 1.09× slower, QuaRot: 1.28×, ViDiT-Q: 1.40×).
- Peak memory matches unquantized model on image; slightly higher on video due to index/gather tensors but still below ViDiT-Q.
- The efficient RPBH transform (vs. dense Haar) reduces activation rotation cost by ~26×.

### Rotation Ablation (Table 3)

| Rotation | W4A4 | W3A3 | W2A4 | Latency (s) |
|----------|------|------|------|-------------|
| Haar (dense) | 0.696 | 0.669 | 0.591 | 11.65 |
| Full RHT | 0.691 | 0.672 | 0.587 | 0.452 |
| Block-RHT (no perm) | 0.678 | 0.642 | 0.558 | 0.381 |
| **RPBH (ours)** | **0.690** | **0.674** | **0.595** | **0.451** |

RPBH achieves the best quality at low bit-width (W2A4) among all rotations, with latency comparable to structured transforms. Removing the random permutation (Block-RHT) degrades performance, confirming its importance.

### AdaLN Bit-Width Ablation (Figure 6)

Quantizing AdaLN modulation weights to INT4 (instead of BF16) nearly matches full-precision GenEval performance, while boosting compression from 2.21× to 4× on FLUX architecture. Further quantization to W2 degrades FLUX models, so OrbitQuant keeps AdaLN at INT4.

## Theoretical and Practical Implications

### Theoretical Significance

- Provides a **formal justification** for using random rotations (not just data-dependent permutations) to enable distributional codebooks for DiT quantization, with Proposition 1 offering a probabilistic variance bound.
- Shifts the paradigm from **per-input range calibration** to **distributional codebook design**, eliminating the need for model evaluation at quantizer construction.
- The shared-rotation design (folding into weights) is computationally elegant – rotations cancel in the product, requiring no inverse transform at inference.

### Practical Implications

- **Calibration-free** operation: no need to collect prompts, timesteps, or CFG branches for each new model or resolution. This dramatically simplifies deployment pipelines.
- **Cross-modality transfer**: the same recipe works for both image and video DiTs without per-modality tuning, reducing engineering effort.
- **Usable at extreme bit-widths**: W2A4 is achieved for the first time in PTQ of DiTs, enabling significant memory and compute savings while maintaining generation quality.
- **Low runtime overhead**: the fixed codebook lookup and efficient RPBH rotation (O(d log h)) add minimal latency, making deployment practical.

## Conclusion

OrbitQuant presents a calibration-free, data-agnostic quantization framework for diffusion transformers that replaces per-timestep range calibration with a single distributional codebook applied in a shared rotated basis. The key innovations are:

1. **RPBH rotation** with a uniform random permutation that keeps rotated coordinates close to a known fixed marginal, enabling a Lloyd–Max codebook that works for all inputs.
2. **Weight-activation sharing**: the same rotation is absorbed into weights offline and applied to activations online, canceling in the product and requiring only a forward transform at runtime.
3. **Empirical success**: state-of-the-art PTQ results at W4A4 and W2A4 on GenEval and VBench, with the only method maintaining generation quality at 2-bit weights. The same recipe transfers from image to video without per-modality tuning.

### Future Directions

- Extending the approach to other architectures (e.g., U-Net-based diffusion models, autoregressive transformers).
- Hardware implementation of the RPBH transform for real-time inference.
- Combining with adaptive bit-allocation or mixed-precision schemes for further compression.
- Exploring the theoretical limits of distributional codebooks under even lower bit-widths (e.g., binary or ternary quantization).

---

_Markdown view of https://picx.dev/p/B0mkuh, served by PicX — AI-generated visual whiteboard summaries of research papers._