Summary (Overview)

• Core contribution: The paper introduces Complex KDA (CKDA), an extension of Kimi Delta Attention (KDA) that combines two existing range extensions—signed channel-wise gates $\alpha_t \in [-1, 1]^n$ and delta-rule coefficient $\beta_t \in [0, 2]$—to enable 2D rotations and complex eigenvalues within a single diagonal-plus-rank-one (DPR1) transition.
• Key theoretical result: Every orthogonal diagonal-plus-rank-one matrix is exactly a CKDA transition matrix (Theorem 1), and a single CKDA layer can track every finite group isomorphic to a subgroup of SO(3), including $S_3$, $S_4$, and $A_5$ (Theorem 3).
• Expressivity improvement: CKDA achieves state-tracking expressivity comparable to DeltaProduct₂ but with a single delta-rule update per token, using one fewer layer than (Gated) DeltaNet for many state-tracking results.
• Empirical findings: CKDA demonstrates the strongest length extrapolation among tested KDA range settings on $S_3$, $S_4$, and periodic audio continuation; at 1.3B parameters, it performs on par with KDA while outperforming Transformers and other linear RNNs.
• Efficiency: The implementation retains approximately 96–97% of standard KDA's throughput, requiring only minor modifications to existing KDA recurrence kernels.

---

Introduction and Theoretical Foundation

Background

Linear recurrent neural networks (RNNs) offer efficient sequence modeling with linear scaling in sequence length and fixed-size recurrent states. Their expressivity depends critically on the structure of their state-transition matrices:

• Diagonal transitions (Mamba-1/2, GLA, mLSTM) support fast computation
• Non-diagonal transitions using the delta-rule introduce a rank-one correction that mixes information across state coordinates

The state-transition matrix for gated delta-rule models takes the form:

$$\pmb{A}_t = (\pmb{I} - \beta_t \pmb{k}_t \pmb{k}_t^\top) \operatorname{Diag}(\pmb{\alpha}_t)$$

Two main variants exist:

• Gated DeltaNet (GDN): uses a scalar gate $\pmb{\alpha}_t = \alpha_t \mathbf{1}$
• Kimi Delta Attention (KDA): allows a separate gate value per channel

Motivation: From Symmetry to Rotation

The paper identifies a fundamental mechanism: signed channel-wise gates combined with Householder reflections produce rotations.

Symmetry analysis: A KDA transition $A = H_k D$ (product of two symmetric matrices) is symmetric iff the factors commute:

$$\left(\boldsymbol{H}_{\boldsymbol{k}} \boldsymbol{D} - \boldsymbol{D} \boldsymbol{H}_{\boldsymbol{k}}\right)_{ij} = \beta k_i k_j (\alpha_i - \alpha_j) = 0 \quad \text{for all } i, j$$

• Scalar gate (GDN): $\alpha_i = \alpha$ always satisfies this condition → real spectrum only
• Channel-wise gate (KDA): noncommutation possible when $\beta \neq 0$ and gate values differ

However, strictly positive gates still yield real spectra (the matrix is similar to a symmetric matrix). Only signed gates enable complex eigenvalues.

Key 2D construction: With $\beta = 2$, $k = (\cos\theta, \sin\theta)^\top$, and $D_\alpha = \operatorname{Diag}(\alpha, 1)$:

$$\boldsymbol{A}_{\alpha,\theta} = \left(\begin{array}{cc} -\alpha \cos 2\theta & -\sin 2\theta \\ -\alpha \sin 2\theta & \cos 2\theta \end{array}\right)$$

The discriminant is $\Delta = (1-\alpha)^2 \cos^2(2\theta) + 4\alpha$. For $\alpha < 0$, complex-conjugate eigenvalues exist, and at $\alpha = -1$, the transition is a composition of two reflections—a rotation by $2\theta$.

---

Methodology

CKDA Definition

CKDA is defined as KDA with:

• $\pmb{\alpha}_t \in [-1, 1]^n$ (signed channel-wise gates)
• $\beta_t \in [0, 2]$ (extended delta-rule coefficient)

Implementation uses $\beta_t = 2\sigma(b_t)$ and signed gates $r_{t,i} = 2\sigma(a_{t,i}) - 1$.

Theoretical Framework

Theorem 1 (DPR1 and CKDA): Every orthogonal DPR1 matrix $\pmb{A} = \pmb{D} + \pmb{uv}^\top \in \mathbb{R}^{n \times n}$ can be written as:

$$\pmb{A} = (\pmb{I} - 2\pmb{k}\pmb{k}^\top)\mathcal{S}, \quad \|\pmb{k}\|_2 = 1, \quad \mathcal{S} = \operatorname{Diag}(s_i), \quad s_i \in \{-1, +1\}$$

Proposition 2 (Sign-magnitude decomposition): A CKDA transform decomposes as $\pmb{A} = \tilde{A}\operatorname{Diag}(|\pmb{\alpha}|)$ where $\tilde{A} = (I - \beta kk^\top)S$. The restricted block on $\mathcal{U} = \operatorname{span}\{k_-, k_+\}$ is:

$$\boldsymbol{B}(\beta, \theta) = \left(\begin{array}{cc} \beta \cos^2\theta - 1 & -\beta \sin\theta \cos\theta \\ \beta \sin\theta \cos\theta & 1 - \beta \sin^2\theta \end{array}\right)$$

This block is a rotation by $2\theta$ when $\beta = 2$, with non-real eigenvalues exactly when $\beta^2 \cos^2(2\theta) < 4(\beta - 1)$.

Key Conditions for Complex Eigenvalues

Complex eigenvalues require:

1. At least two gate coordinates with opposite signs
2. Key vector spanning coordinates with opposite signs
3. $\beta > 1$

Implementation

Three implementation variants:

1. Compiled PyTorch: absorbs cumulative signs into keys/queries without changing kernels
2. Triton kernels: fuse signs into normalization and backward pass
3. TileLang hybrid: combines Triton changes with TileLang backward kernels

---

Empirical Validation / Results

Expressivity Results (Table 1 Summary)

State-Tracking Results

• CKDA extrapolates well on $S_3$ and $S_4$ while other KDA settings fail (Figure 6)
• Learned CKDA heads recover the theoretical mechanism: $\beta \approx 2$, nearly sign-valued gates, complex-conjugate eigenvalues near the unit circle (Figure 7)
• $A_5$ requires theory-initialized training (optimization obstacle, not representational)

Audio Continuation

• CKDA with both range extensions accurately continues periodic waveforms beyond training horizon
• At length 264 (vs. max training length 136): 38.1 dB SNR for CKDA vs. 2.8 dB for causal Transformer

Language Modeling (1.3B parameters, 100B tokens, FineWeb-Edu)

Kernel Throughput

CKDA retains approximately 96–97% of standard KDA throughput on H100 GPUs.

---

Theoretical and Practical Implications

Theoretical Implications

1. Characterization of DPR1 expressivity: CKDA captures the entire orthogonal DPR1 family, showing that the non-symmetric $uv^\top$ structure in models like RWKV-7 adds no orthogonal transitions beyond what CKDA achieves.

2. Spectral restrictions: Every nonexpansive DPR1 matrix has at most one non-real conjugate eigenvalue pair on the unit circle (Theorem 9). This is a fundamental limitation of the architecture family.

3. State-tracking hierarchy: CKDA requires one fewer layer than (Gated) DeltaNet for many state-tracking results (3 vs. 4 layers for general regular languages and WFAs), matching DeltaProduct's expressivity with a single rank-one correction per token.

4. Spectral obstruction theorem: Non-expansive transitions with at most one complex eigenvalue pair cannot track $S_5$ in one layer (Theorem 4), establishing a fundamental limitation shared by CKDA and DeltaProduct with $k \leq 3$.

Practical Implications

1. Efficient expressivity gain: CKDA achieves DeltaProduct₂-level expressivity without the computational cost of two delta-rule updates per token.

2. Length extrapolation: Signed gates and extended β enable better extrapolation on group word problems and periodic waveform continuation—useful for tasks requiring long-range periodic structure.

3. Language modeling: CKDA matches or slightly exceeds KDA baselines while providing additional expressivity, with no significant throughput penalty.

4. Learned mechanisms: Analysis of trained models confirms that language models actively learn to use negative gates and extended β ranges, particularly in early layers, suggesting these mechanisms serve functional roles in sequence processing.

---

Conclusion

Main Takeaways

• CKDA combines signed channel-wise gates ($\alpha \in [-1, 1]^n$) with extended delta-rule coefficient ($\beta \in [0, 2]$) to enable planar rotations and complex eigenvalues within a single non-expansive DPR1 transition.
• Every orthogonal DPR1 matrix is exactly a CKDA transition, establishing CKDA's completeness within this important matrix family.
• State-tracking expressivity matches DeltaProduct while using one fewer layer than (Gated) DeltaNet for many tasks.
• Empirical validation confirms the theoretical mechanism in both state tracking and language modeling, with CKDA achieving the strongest length extrapolation among KDA variants.

Limitations

• Rank-one structure limits transitions to at most one persistent complex-conjugate eigenvalue pair
• Expressivity results do not imply learnability ($A_5$ requires theory-initialized training)
• General WFA construction requires $\beta > 2$, sacrificing guaranteed non-expansiveness
• No inherent computational advantage over (Gated) DeltaProduct₂

Future Work

• Combine CKDA with separate gates (as in GDN-2) controlling additive terms and state forgetting
• Mechanistic analysis of how learned gates and β ranges function in language models
• Investigation of the functional roles of negative gates and extended β in trained models