# Taylor-Calibrate: Principled Initialization for Hybrid Linear Attention Distillation

> Taylor-Calibrate initializes Gated DeltaNet hybrid layers from teacher attention statistics, cutting distillation tokens 4.9x-9.2x while matching transformer quality.

- **Source:** [arXiv](https://arxiv.org/abs/2606.16429)
- **Published:** 2026-09-12
- **Permalink:** https://picx.dev/p/iXX1sC
- **Whiteboard:** https://picx.dev/p/iXX1sC/image

## Summary

## Summary (Overview)

- **Core Problem**: Converting pretrained Transformers to hybrid linear attention models (specifically Gated DeltaNet, GDN) by simply copying teacher Q/K/V/O projections leaves the new recurrent decay, write, and output-gating dynamics uninitialized, resulting in poor starting quality and slow recovery during distillation.
- **Proposed Solution**: **Taylor-Calibrate**, a lightweight two-stage initialization method that (1) extracts Taylor-guided statistics from teacher attention maps to calibrate value scale, decay timescale, write gate, and output gate, and (2) performs a brief per-layer alignment step to match each converted GDN layer to the teacher output.
- **Key Results**: Taylor-Calibrate achieves **$4.9\times$–$9.2\times$ fewer training tokens** than naive conversion to reach teacher-like quality, with consistent improvements in zero-shot perplexity (Avg) and RULER scores across multiple teacher models (Qwen3-8B, Llama-3.2-3B, Qwen2.5-7B).
- **Insight**: Initialization quality mainly affects *recovery speed* rather than the final converged quality; long-context retrieval (RULER) requires longer training budgets and is governed by a slower timescale than short-context perplexity.
- **Broader Message**: Linear-attention transfer needs to initialize *dynamical parameters*, not just copy projection weights.

---

## Introduction and Theoretical Foundation

### Background
Transformers remain the default for LLMs, but long-context decoding suffers from quadratic attention cost and linearly growing KV cache. Hybrid models (e.g., Jamba, Samba, Hymba, MiniMax-01) replace some softmax layers with cheaper recurrent/linear modules to improve throughput and memory trade-offs. However, pure recurrent models struggle on retrieval-heavy tasks, motivating hybrid designs that retain some softmax layers.

### Motivation
Recent pipelines convert pretrained Transformers into hybrid students by copying teacher projections. The central issue: **hybrid transfer is not only a projection-copying problem**. Copying $Q/K/V/O$ weights preserves useful projections but does not specify:
- Recurrent timescales (decay)
- Write gates
- Output gates

For GDN, the recurrent output is additionally modulated by a learned output gate before the final projection. If these new parameters are left random, the converted student can have very poor zero-shot quality.

### Theoretical Foundation: Taylor View of Softmax Attention
The method is motivated by expanding the softmax numerator around a neutral logit regime, following the view that softmax attention admits linear-attention and recurrent approximations:

$$
\text{softmax}(QK^\top) \approx \text{linear approximation around neutral logits}
$$

Truncating this expansion motivates a fixed-size linear-recurrent approximation of softmax attention, making teacher attention statistics (value amplitude, average look-back distance, attention concentration) operational proxies for GDN's value scale, decay timescale, and write strength.

### Gated DeltaNet (GDN)
GDN replaces softmax attention with a recurrent state update:

$$
\text{state update: } s_t = \beta_t s_{t-1} + \alpha_t v_t
$$

where $\beta_t$ is the decay, $\alpha_t$ is the write gate, and the output is further modulated by an output gate $g_{\text{proj}}$. GDN provides constant-memory inference but places substantial burden on decay and gating parameters.

---

## Methodology

### Stage 1: Taylor-Guided Analytical Calibration
For each converted layer $\ell \in S_{\text{gdn}}$, the method reads statistics from teacher attention maps:

| Parameter/statistic | Standard GDN transfer | Taylor-Calibrate initialization |
|---|---|---|
| $d_h, e_h, c_h, \sigma_h^\star$ | Not used | Teacher attention statistics |
| $W_Q, W_K, W_V, W_O$ | Copied | Copied |

**Value scale** — Teacher output amplitude calibrates $W_V$: the value-side OLS factor $\sigma_h^\star$ is computed per head by matching the student value-path output to the teacher output in least squares (Appendix A.5).

**Decay timescale** — The distance statistic $d_h$ is the attention-weighted average look-back length. If a head should preserve half its information after roughly $d_h$ tokens, the target decay magnitude is:

$$
|g_h| = \frac{\ln 2}{d_h}
$$

**Write gate** — Attention entropy is converted into a write-gate target via a concentration score $c_h$; higher entropy (more diffuse attention) leads to weaker writes.

**Output gate** — Initialized small enough (RMS-matched) to avoid dominating the layer output, keeping the recurrent branch in a low-contribution, high-gradient regime.

### Stage 2: Layer-Local Alignment
A short per-layer distillation step matches each converted GDN layer output to the teacher layer output:

$$
\min_{\theta_{\text{GDN}}} \| z_S^{(\ell)}(x; \theta_{\text{GDN}}) - z_T^{(\ell)}(x) \|^2
$$

This is performed on calibration data before global distillation. It allows transferred projections to adapt to recurrent dynamics and updates gate *directions* rather than just scales. Notably, successful alignment does not dramatically increase the magnitude of $g_{\text{proj}}$; instead, it rotates parameters into directions producing teacher-like outputs.

---

## Empirical Validation / Results

### Setup
- **Teachers**: Qwen3-8B, Llama-3.2-3B, Qwen2.5-7B
- **Conversion**: Replace a fraction (25%, 50%, 75%) of softmax layers with GDN, following RADLADS-style projection transfer
- **Baselines**: Baseline (random recurrent params), Zero-Gate, Small-Gate, Taylor-Only (Stage 1 only), Taylor-Calibrate (Stages 1 + 2)
- **Metrics**: Avg PPL (short-context), RULER (long-context retrieval)

### Zero-Shot Results (Table 2)

Key trends:
- **Aggressive conversion is hard**: At 25–50% retained attention, all methods have very low RULER; Baseline Avg is near chance-level.
- **Output-gate stabilization helps most when enough softmax remains**: At 75% retention, Zero-Gate and Small-Gate recover much of the short-context Avg.
- **Full Taylor-Calibrate most consistently improves Avg**, especially at 50% and 75% retention. RULER gains are more model- and budget-dependent.

### Recovery Dynamics (Table 3)

| Setting | Baseline | Taylor-Only | Alignment-Only | Taylor-Calibrate |
|---|---|---|---|---|
| PPL (initial) | 37337.3 | ~22470 | 2015.9 | **424.1** |
| Avg (best) | — | — | — | **32.1** |

Taylor calibration is most useful as a **better starting point for local alignment**, not a standalone replacement. The largest benefit appears early in recovery, before Stage 2 has enough tokens to repair poor initialization.

### Training Token Savings
- **$4.9\times$–$9.2\times$ fewer tokens** to reach teacher-like quality compared to naive conversion
- For Llama-3.2-3B, the baseline recovers reasonably after Stage 1, but Taylor-Calibrate still gives consistent short-context gains
- By 700M tokens, the Avg gap narrows as distillation restores short-context behavior for both initializations

### Long-Context Recovery
RULER recovery is governed by a **slower timescale** than PPL or short-context Avg. Restoring retrieval behavior requires SFTMix v0.2 and more than 20k optimization steps, beyond the short recovery schedule. Taylor-Calibrate improves the starting point and early recovery, but full long-context restoration remains a slower process.

---

## Theoretical and Practical Implications

### Theoretical Implications
1. **Initialization as architecture transfer**: The results reframe hybrid conversion as an initialization problem for recurrent dynamics, not just projection copying. The Taylor view of softmax attention provides a principled bridge between teacher attention statistics and GDN's decay/gating structure.
2. **Layer-local alignment matters**: Analytical calibration alone (Taylor-Only) is insufficient; the short alignment step is critical for rotating gate directions into teacher-like configurations.
3. **Recovery speed vs. final quality**: Initialization primarily accelerates recovery rather than changing the converged optimum, suggesting the loss landscape is well-behaved but the initialization determines the path.

### Practical Implications
1. **Token efficiency**: Taylor-Calibrate reduces distillation token budgets by up to $9.2\times$, directly lowering training compute costs for model conversion.
2. **Hybrid model deployment**: Enables faster conversion of pretrained Transformers into efficient hybrid architectures for long-context inference, reducing KV-cache and quadratic attention costs.
3. **Design guidance**: Output-gate stabilization is a cheap but effective safeguard; full calibration is most valuable when aggressive layer replacement is desired.

---

## Conclusion

**Main takeaways**:
- Taylor-Calibrate is a lightweight, two-stage initialization method for GDN-based hybrid distillation that combines Taylor-derived gate calibration with brief layer-local alignment.
- It significantly improves zero-shot quality and reduces recovery tokens ($4.9\times$–$9.2\times$) across multiple teacher models.
- Initialization quality mainly affects recovery speed; long-context retrieval requires additional training beyond short-context recovery.

**Future directions**:
- Extending calibration to other linear-attention families (e.g., Mamba, RWKV)
- Larger-model support and architecture-specific calibration
- Finer-grained head-wise conversion strategies
- Understanding the slower timescale of long-context retrieval recovery

---

_Markdown view of https://picx.dev/p/iXX1sC, served by PicX — AI-generated visual whiteboard summaries of research papers._