# Complex KDA: Understanding and Enhancing the Expressivity of Kimi Delta Attention

> CKDA extends KDA with signed gates and beta up to 2, enabling complex eigenvalues and rotations that match DeltaProduct expressivity with a single delta-rule update per token.

- **Source:** [arXiv](https://arxiv.org/abs/2609.24797)
- **Published:** 2026-09-26
- **Permalink:** https://picx.dev/p/q0NCYx
- **Whiteboard:** https://picx.dev/p/q0NCYx/image

## Summary

## Summary (Overview)

- **Core contribution**: The paper introduces **Complex KDA (CKDA)**, an extension of Kimi Delta Attention (KDA) that combines two existing range extensions—signed channel-wise gates $\alpha_t \in [-1, 1]^n$ and delta-rule coefficient $\beta_t \in [0, 2]$—to enable 2D rotations and complex eigenvalues within a single diagonal-plus-rank-one (DPR1) transition.
- **Key theoretical result**: Every orthogonal diagonal-plus-rank-one matrix is exactly a CKDA transition matrix (Theorem 1), and a single CKDA layer can track every finite group isomorphic to a subgroup of SO(3), including $S_3$, $S_4$, and $A_5$ (Theorem 3).
- **Expressivity improvement**: CKDA achieves state-tracking expressivity comparable to DeltaProduct₂ but with a single delta-rule update per token, using one fewer layer than (Gated) DeltaNet for many state-tracking results.
- **Empirical findings**: CKDA demonstrates the strongest length extrapolation among tested KDA range settings on $S_3$, $S_4$, and periodic audio continuation; at 1.3B parameters, it performs on par with KDA while outperforming Transformers and other linear RNNs.
- **Efficiency**: The implementation retains approximately 96–97% of standard KDA's throughput, requiring only minor modifications to existing KDA recurrence kernels.

---

## Introduction and Theoretical Foundation

### Background

Linear recurrent neural networks (RNNs) offer efficient sequence modeling with linear scaling in sequence length and fixed-size recurrent states. Their expressivity depends critically on the structure of their state-transition matrices:

- **Diagonal transitions** (Mamba-1/2, GLA, mLSTM) support fast computation
- **Non-diagonal transitions** using the delta-rule introduce a rank-one correction that mixes information across state coordinates

The state-transition matrix for gated delta-rule models takes the form:

$$\pmb{A}_t = (\pmb{I} - \beta_t \pmb{k}_t \pmb{k}_t^\top) \operatorname{Diag}(\pmb{\alpha}_t)$$

Two main variants exist:
- **Gated DeltaNet (GDN)**: uses a scalar gate $\pmb{\alpha}_t = \alpha_t \mathbf{1}$
- **Kimi Delta Attention (KDA)**: allows a separate gate value per channel

### Motivation: From Symmetry to Rotation

The paper identifies a fundamental mechanism: **signed channel-wise gates combined with Householder reflections produce rotations**.

**Symmetry analysis**: A KDA transition $A = H_k D$ (product of two symmetric matrices) is symmetric iff the factors commute:

$$\left(\boldsymbol{H}_{\boldsymbol{k}} \boldsymbol{D} - \boldsymbol{D} \boldsymbol{H}_{\boldsymbol{k}}\right)_{ij} = \beta k_i k_j (\alpha_i - \alpha_j) = 0 \quad \text{for all } i, j$$

- **Scalar gate** (GDN): $\alpha_i = \alpha$ always satisfies this condition → real spectrum only
- **Channel-wise gate** (KDA): noncommutation possible when $\beta \neq 0$ and gate values differ

However, **strictly positive gates still yield real spectra** (the matrix is similar to a symmetric matrix). Only **signed gates** enable complex eigenvalues.

**Key 2D construction**: With $\beta = 2$, $k = (\cos\theta, \sin\theta)^\top$, and $D_\alpha = \operatorname{Diag}(\alpha, 1)$:

$$\boldsymbol{A}_{\alpha,\theta} = \left(\begin{array}{cc} -\alpha \cos 2\theta & -\sin 2\theta \\ -\alpha \sin 2\theta & \cos 2\theta \end{array}\right)$$

The discriminant is $\Delta = (1-\alpha)^2 \cos^2(2\theta) + 4\alpha$. For $\alpha < 0$, complex-conjugate eigenvalues exist, and at $\alpha = -1$, the transition is a composition of two reflections—a rotation by $2\theta$.

---

## Methodology

### CKDA Definition

CKDA is defined as KDA with:
- $\pmb{\alpha}_t \in [-1, 1]^n$ (signed channel-wise gates)
- $\beta_t \in [0, 2]$ (extended delta-rule coefficient)

Implementation uses $\beta_t = 2\sigma(b_t)$ and signed gates $r_{t,i} = 2\sigma(a_{t,i}) - 1$.

### Theoretical Framework

**Theorem 1 (DPR1 and CKDA)**: Every orthogonal DPR1 matrix $\pmb{A} = \pmb{D} + \pmb{uv}^\top \in \mathbb{R}^{n \times n}$ can be written as:

$$\pmb{A} = (\pmb{I} - 2\pmb{k}\pmb{k}^\top)\mathcal{S}, \quad \|\pmb{k}\|_2 = 1, \quad \mathcal{S} = \operatorname{Diag}(s_i), \quad s_i \in \{-1, +1\}$$

**Proposition 2 (Sign-magnitude decomposition)**: A CKDA transform decomposes as $\pmb{A} = \tilde{A}\operatorname{Diag}(|\pmb{\alpha}|)$ where $\tilde{A} = (I - \beta kk^\top)S$. The restricted block on $\mathcal{U} = \operatorname{span}\{k_-, k_+\}$ is:

$$\boldsymbol{B}(\beta, \theta) = \left(\begin{array}{cc} \beta \cos^2\theta - 1 & -\beta \sin\theta \cos\theta \\ \beta \sin\theta \cos\theta & 1 - \beta \sin^2\theta \end{array}\right)$$

This block is a rotation by $2\theta$ when $\beta = 2$, with non-real eigenvalues exactly when $\beta^2 \cos^2(2\theta) < 4(\beta - 1)$.

### Key Conditions for Complex Eigenvalues

Complex eigenvalues require:
1. At least two gate coordinates with opposite signs
2. Key vector spanning coordinates with opposite signs
3. $\beta > 1$

### Implementation

Three implementation variants:
1. **Compiled PyTorch**: absorbs cumulative signs into keys/queries without changing kernels
2. **Triton kernels**: fuse signs into normalization and backward pass
3. **TileLang hybrid**: combines Triton changes with TileLang backward kernels

---

## Empirical Validation / Results

### Expressivity Results (Table 1 Summary)

| Property | CKDA | GDN | Gated DeltaProduct | RWKV-7/GDN-2 |
|---|---|---|---|---|
| Complex pairs | ≤ 1 (Thm. 8) | 0 | ≤ ⌊k/2⌋ | 0 |
| $S_2$ (parity) | ✓ 1 layer | ✓ 1 layer | ✓ 1 layer (k≥1) | ✓ 1 layer |
| $\mathbb{Z}_n, D_n$ (n>2) | ✓ 1 layer | ✓ 2 layers | ✓ 1 layer (k≥2) | ✓ 2 layers |
| $S_4, A_5$ | ✓ 1 layer | ✓ 4 layers | ✓ 1 layer (k≥2) | ✓ 4 layers |
| $S_n$ (n≥5) | ✓ 3 layers | ✓ 4 layers | ✓ 3 layers | ✓ 4 layers |

### State-Tracking Results

- **CKDA extrapolates well on $S_3$ and $S_4$** while other KDA settings fail (Figure 6)
- Learned CKDA heads recover the theoretical mechanism: $\beta \approx 2$, nearly sign-valued gates, complex-conjugate eigenvalues near the unit circle (Figure 7)
- $A_5$ requires theory-initialized training (optimization obstacle, not representational)

### Audio Continuation

- CKDA with both range extensions accurately continues periodic waveforms beyond training horizon
- At length 264 (vs. max training length 136): **38.1 dB SNR** for CKDA vs. **2.8 dB** for causal Transformer

### Language Modeling (1.3B parameters, 100B tokens, FineWeb-Edu)

| Model | Wiki. ppl ↓ | LMB. ppl ↓ | Avg. accuracy (%) ↑ |
|---|---|---|---|
| KDA | 16.81 | 11.68 | 52.28 |
| CKDA (ours) | 15.78 | 10.08 | 54.06 |
| KDA + attn 3:1 (ours) | 15.04 | 9.93 | 53.92 |
| CKDA + attn 3:1 (ours) | 15.34 | 9.90 | 54.37 |
| Mamba-3 (MIMO) | 16.45 | 11.66 | 52.39 |
| Gated DeltaNet-2 | 15.90 | 11.41 | 53.11 |

### Kernel Throughput

CKDA retains approximately **96–97% of standard KDA throughput** on H100 GPUs.

---

## Theoretical and Practical Implications

### Theoretical Implications

1. **Characterization of DPR1 expressivity**: CKDA captures the entire orthogonal DPR1 family, showing that the non-symmetric $uv^\top$ structure in models like RWKV-7 adds no orthogonal transitions beyond what CKDA achieves.

2. **Spectral restrictions**: Every nonexpansive DPR1 matrix has at most one non-real conjugate eigenvalue pair on the unit circle (Theorem 9). This is a fundamental limitation of the architecture family.

3. **State-tracking hierarchy**: CKDA requires one fewer layer than (Gated) DeltaNet for many state-tracking results (3 vs. 4 layers for general regular languages and WFAs), matching DeltaProduct's expressivity with a single rank-one correction per token.

4. **Spectral obstruction theorem**: Non-expansive transitions with at most one complex eigenvalue pair cannot track $S_5$ in one layer (Theorem 4), establishing a fundamental limitation shared by CKDA and DeltaProduct with $k \leq 3$.

### Practical Implications

1. **Efficient expressivity gain**: CKDA achieves DeltaProduct₂-level expressivity without the computational cost of two delta-rule updates per token.

2. **Length extrapolation**: Signed gates and extended β enable better extrapolation on group word problems and periodic waveform continuation—useful for tasks requiring long-range periodic structure.

3. **Language modeling**: CKDA matches or slightly exceeds KDA baselines while providing additional expressivity, with no significant throughput penalty.

4. **Learned mechanisms**: Analysis of trained models confirms that language models actively learn to use negative gates and extended β ranges, particularly in early layers, suggesting these mechanisms serve functional roles in sequence processing.

---

## Conclusion

### Main Takeaways

- **CKDA** combines signed channel-wise gates ($\alpha \in [-1, 1]^n$) with extended delta-rule coefficient ($\beta \in [0, 2]$) to enable planar rotations and complex eigenvalues within a single non-expansive DPR1 transition.
- **Every orthogonal DPR1 matrix** is exactly a CKDA transition, establishing CKDA's completeness within this important matrix family.
- **State-tracking expressivity** matches DeltaProduct while using one fewer layer than (Gated) DeltaNet for many tasks.
- **Empirical validation** confirms the theoretical mechanism in both state tracking and language modeling, with CKDA achieving the strongest length extrapolation among KDA variants.

### Limitations

- Rank-one structure limits transitions to at most one persistent complex-conjugate eigenvalue pair
- Expressivity results do not imply learnability ($A_5$ requires theory-initialized training)
- General WFA construction requires $\beta > 2$, sacrificing guaranteed non-expansiveness
- No inherent computational advantage over (Gated) DeltaProduct₂

### Future Work

- Combine CKDA with separate gates (as in GDN-2) controlling additive terms and state forgetting
- Mechanistic analysis of how learned gates and β ranges function in language models
- Investigation of the functional roles of negative gates and extended β in trained models

---

_Markdown view of https://picx.dev/p/q0NCYx, served by PicX — AI-generated visual whiteboard summaries of research papers._