# On-Policy Morphing into Hybrid Attention Models

> FlashMorph2 fixes two off-policy mismatches in FlashMorph's layer selection, improving hybrid attention model conversion by up to +2.42 RULER points with under 4 GPU hours of search.

- **Source:** [arXiv](https://arxiv.org/abs/2611.24809)
- **Published:** 2026-11-21
- **Permalink:** https://picx.dev/p/5jeUyR
- **Whiteboard:** https://picx.dev/p/5jeUyR/image

## Summary

## Summary (Overview)

- **Core Problem**: Converting pretrained Transformers into hybrid attention models (full + linear attention) requires selecting which layers retain full attention under a budget. FlashMorph2 identifies and fixes two "off-policy" mismatches in FlashMorph's layer selection.
- **Key Contribution**: FlashMorph2 introduces an on-policy layer-selection method that (I) evaluates sampled discrete hybrid architectures end-to-end using self-generated hidden states, and (II) jointly optimizes layer-selection scores and model weights under the recovery objective.
- **Main Results**: FlashMorph2 improves average scores over FlashMorph by **+2.42 points on RULER**, **+1.08 on LongBench**, and **+0.76 on recall-intensive tasks**, across Qwen3-0.6B and Qwen3-1.7B backbones with three linear-attention variants.
- **Efficiency**: Layer selection completes in only **0.72–3.57 GPU hours** across 0.6B–8B models, running 8.4–20.8× faster than HALO and 560–854× faster than KL-LS.
- **Identified Limitations in FlashMorph**: (L1) continuous branch interpolation and teacher-forced hidden states differ from discrete hybrid computation; (L2) frozen model weights fail to account for parameter adaptation during subsequent recovery.

---

## Introduction and Theoretical Foundation

### Background

Hybrid attention models retain full attention in only a few layers and use linear attention elsewhere, preserving long-context modeling and precise retrieval while substantially reducing computation and KV-cache memory usage. To avoid prohibitive pretraining costs, existing approaches reuse pretrained LLM weights and replace a subset of full attention layers with linear attention, followed by distillation and continued training.

### The Layer Selection Challenge

A central challenge is determining **which full-attention layers to retain** under a given budget. Since these layers are the only ones that can directly access the entire context, their placement largely determines how much of the original model's capability can be recovered.

### Limitations of FlashMorph

FlashMorph's layer selection suffers from two off-policy mismatches:

- **L1: Off-policy computation** — Continuous gates mix full- and linear-attention outputs computed from teacher hidden states. At deployment, only one branch is retained per layer and hidden states are generated by the hybrid model itself. Complementarities observed under continuous mixing may not transfer to the final discrete architecture, and error accumulation/cross-layer compensation is missed.
- **L2: Off-policy weights** — Weights used for layer selection remain frozen during gate optimization, without adapting to the hybrid architecture. Layer selection cannot account for cross-layer co-adaptation during subsequent recovery.

### Theoretical Foundation

The paper draws on **on-policy learning** principles from reinforcement learning and sequence generation, where supervision is obtained on states induced by the learner's own policy rather than ground-truth (teacher-forced) states. This mitigates exposure bias and distribution mismatch.

---

## Methodology

### Preliminaries: Attention Formulations

**Full attention** (causal masking) computes:

$$
\mathbf{o}_{\mathrm{full},t} = \frac{\sum_{i=1}^{t} \exp\left(\mathbf{q}_t \mathbf{k}_i^{\top} / \sqrt{d}\right) \mathbf{v}_i}{\sum_{j=1}^{t} \exp\left(\mathbf{q}_t \mathbf{k}_j^{\top} / \sqrt{d}\right)} \mathbf{W}_O \tag{1}
$$

This incurs $\mathcal{O}(T^2)$ computational cost and $\mathcal{O}(T)$ KV-cache memory.

**Linear attention** uses a fixed-size recurrent state:

$$
\mathbf{o}_{\mathrm{lin},t} = \mathbf{q}_t \mathbf{S}_t \mathbf{W}_O, \quad \mathbf{S}_t = \mathbf{S}_{t-1} + \mathbf{k}_t^{\top} \mathbf{v}_t \tag{2}
$$

where $\mathbf{S}_t \in \mathbb{R}^{d \times d}$ and $\mathbf{S}_0 = 0$, incurring $\mathcal{O}(T)$ cost and $\mathcal{O}(1)$ cache.

**Hybrid attention** combines both via a binary mask $\mathbf{m} \in \{0,1\}^L$:

$$
\mathbf{o}^{(l)} = m_l \mathbf{o}_{\mathrm{full}}^{(l)} + (1 - m_l) \mathbf{o}_{\mathrm{lin}}^{(l)} \tag{3}
$$

Given a budget $K$, the feasible set is $\mathcal{C}_K = \{\mathcal{T} \subseteq [L] : |\mathcal{T}| = K\}$.

### FlashMorph Pipeline (Four Stages)

1. **Stage 1 — Layerwise alignment**: Linear-attention branches are initialized from pretrained weights and trained to match teacher attention outputs:
   $$\mathcal{L}_{\mathrm{align}} = \frac{1}{L}\sum_{l=1}^{L}\left\| A_{\mathrm{lin}}^{(l)}(\mathbf{h}_{\mathrm{T}}^{(l)}) - A_{\mathrm{full}}^{(l)}(\mathbf{h}_{\mathrm{T}}^{(l)}) \right\|_2^2 \tag{4}$$

2. **Stage 2 — Layer selection**: Continuous gates $\alpha_l$ interpolate between branches with teacher-forced states:
   $$\mathbf{o}_{\mathrm{mix}}^{(l)} = \alpha_l \mathbf{o}_{\mathrm{full,T}}^{(l)} + (1-\alpha_l) \mathbf{o}_{\mathrm{lin,T}}^{(l)} \tag{5}$$
   $$\mathcal{L}_{\mathrm{LS}} = \frac{1}{L}\sum_{l=1}^{L}\left\| \mathbf{o}_{\mathrm{mix}}^{(l)} - \mathbf{o}_{\mathrm{full,T}}^{(l)} \right\|_2^2 + \lambda \sum_{l=1}^{L} \alpha^{(l)} \tag{6}$$

3. **Stage 3 — Logits distillation**: KL divergence from the frozen teacher:
   $$\mathcal{L}_{\mathrm{KD}} = \sum D_{\mathrm{KL}}\left(p_{\mathrm{T}}(\cdot \mid x_{\leq t}) \| p_{\theta}(\cdot \mid x_{\leq t})\right) \tag{7}$$

4. **Stage 4 — Long-context finetuning**: Standard next-token prediction:
   $$\mathcal{L}_{\mathrm{FT}} = -\sum \log p_{\theta}(x_{t+1} \mid x_{\leq t}) \tag{8}$$

### FlashMorph2: On-Policy Layer Selection

#### 1. On-Policy Hybrid Computation

Each layer gets a learnable score $\phi_l$. To explore architectures, Gumbel noise is added and Top-K selection is applied:

$$\mathcal{I} = \operatorname{TopK}(\phi + \mathbf{g}, K), \quad g_l \stackrel{\text{i.i.d.}}{\sim} \operatorname{Gumbel}(0,1) \tag{10}$$

This induces a distribution $q_\phi$ over hybrid architectures in $\mathcal{C}_K$. The model performs an **end-to-end forward pass** through the sampled discrete architecture, allowing earlier branch choices to affect subsequent layer inputs.

#### 2. On-Policy Weight Adaptation

Layer-selection parameters $\phi$ and model weights $\theta$ are **jointly optimized**:

$$\min_{\phi,\theta} \mathcal{J}(\phi,\theta) = \mathbb{E}_{\mathcal{I} \sim q_\phi}\left[\mathcal{L}_{\mathrm{KD}}(\theta; \mathcal{I})\right] \tag{11}$$

The distillation loss is computed on sampled architectures, with gradients for both $\phi$ and $\theta$ obtained through a single backward pass.

#### 3. Budget-Constrained Straight-Through Estimator

To handle non-differentiable discrete Top-K selection, a continuous relaxation preserving the budget constraint is constructed:

$$\tilde{m}_l = \sigma(\tilde{\phi}_l - \tau), \quad \sum_{l=1}^{L} \sigma(\tilde{\phi}_l - \tau) = K \tag{12}$$

where $\tau$ is found via binary search. The gradient straight-through estimator is:

$$\mathbf{m}_{\mathrm{ST}} = \operatorname{sg}\left(\mathbf{m} - \tilde{\mathbf{m}}\right) + \tilde{\mathbf{m}} \tag{13}$$

The biased gradient estimator is:

$$\widehat{\nabla}_{\phi_j} \mathcal{L}_{\mathrm{KD}} = s_j (u_j - \bar{u}), \quad \bar{u} = \frac{\sum_{l=1}^{L} s_l u_l}{\sum_{l=1}^{L} s_l} \tag{14}$$

where $s_l = \tilde{m}_l(1-\tilde{m}_l)$ and $u_l = \partial \mathcal{L}_{\mathrm{KD}}/\partial m_l$.

After optimization, the final architecture is $\mathcal{T}^\star = \operatorname{TopK}(\phi, K)$, and the model is reinitialized with $\theta_0$ for Stages 3–4.

---

## Empirical Validation / Results

### Experimental Setup

- **Backbones**: Qwen3-0.6B and Qwen3-1.7B
- **Linear attention variants**: Lightning Attention, GLA, Gated DeltaNet (GDN)
- **Hybrid ratio**: 3:1 linear-to-full attention by default
- **Baselines**: Uniform, PostNAS, KL-LS, HALO, FlashMorph
- **Evaluation**: RULER (13 tasks), LongBench (14 tasks), 6 recall-intensive tasks, perplexity, commonsense reasoning

### Long-Context Performance (RULER & LongBench)

**Key results from Table 1** (0.6B backbone, GDN variant):

| Method | RULER Avg | LongBench Avg |
|--------|-----------|---------------|
| FlashMorph | 60.29 | 29.15 |
| **FlashMorph2** | **63.65** | **29.77** |

**Key results from Table 1** (1.7B backbone, GDN variant):

| Method | RULER Avg | LongBench Avg |
|--------|-----------|---------------|
| FlashMorph | 73.07 | 32.62 |
| **FlashMorph2** | **74.46** | **33.81** |

**Averaged across all six configurations**: FlashMorph2 raises RULER from 63.26 to 65.68 (+2.42) and LongBench from 29.86 to 30.94 (+1.08) over FlashMorph.

### Recall-Intensive Tasks

**Key results from Table 2** (1.7B backbone, GDN variant):

| Method | FDA | SWDE | SQD | NQ | TQA | DROP | Avg |
|--------|-----|------|-----|-----|------|------|-----|
| FlashMorph | 59.53 | 65.17 | 53.25 | 32.06 | 66.00 | 34.64 | 51.77 |
| **FlashMorph2** | **61.71** | **66.88** | **53.62** | 31.58 | **66.94** | 33.21 | **52.32** |

**Averaged across all configurations**: FlashMorph2 improves from 48.94 to 49.70 (+0.76) over FlashMorph.

### Layer-Selection Efficiency

- FlashMorph2 requires 1.7–2.6× the FLOPs and 1.9–2.4× the GPU hours of FlashMorph
- Completes in **0.72–3.57 GPU hours** across 0.6B–8B models
- HALO requires 8.4–20.8× more GPU hours; KL-LS requires 560–854× more
- Selection overhead is <1% of the overall conversion token budget

### Ablation Study (Table 3)

| Method | RULER | LongBench |
|--------|-------|-----------|
| **FlashMorph2** | **74.46** | **33.81** |
| w/o On-Policy Computation | 44.91 | 28.10 |
| w/o On-Policy Weight | 69.58 | 32.43 |
| w/o Both (FlashMorph) | 73.07 | 32.62 |

**Key findings**:
- Removing on-policy computation causes a **-29.55 point drop on RULER**, as the variant concentrates all full-attention layers in the deepest layers {20, 21, 23–27}, misallocating the budget.
- Removing on-policy weight adaptation lowers RULER by 4.88 points, replacing layers {6, 17, 19, 20} with {0, 1, 2, 26}.
- Both components are **complementary** — removing either degrades performance.

### Hybrid Ratios

- At the tightest budget (6:1 ratio), FlashMorph2 achieves 60.12 on GLA and 64.45 on GDN, outperforming Uniform by +10.65 and +7.86 points.
- At 1:1 ratio, FlashMorph2 reaches 76.93 on GLA (+1.42 over FlashMorph) and 80.69 on GDN (+0.92).

---

## Theoretical and Practical Implications

### Theoretical Contributions

1. **On-policy principle for architecture search**: FlashMorph2 generalizes on-policy learning beyond sequence generation to the domain of neural architecture selection, demonstrating that evaluating architectures under self-generated states (rather than teacher-forced states) yields more reliable importance estimates.

2. **Joint optimization of selection and weights**: The paper shows that layer importance is not intrinsic to a frozen model but co-adapts with weights during recovery training. Optimizing selection scores and weights jointly under the same objective as deployment reflects the true post-adaptation performance landscape.

3. **Budget-constrained STE**: The implicit differentiation of the threshold $\tau$ provides a principled way to propagate gradients through discrete Top-K selection while preserving the exact budget constraint.

### Practical Implications

1. **Efficient Transformer-to-hybrid conversion**: FlashMorph2 makes hybrid model conversion practical for production-scale LLMs by reducing layer selection to under a few GPU hours while improving quality.

2. **Scalability**: Demonstrated across 0.6B–8B models, FlashMorph2's efficiency advantage grows with model scale, making it suitable for larger models.

3. **Variant-agnostic**: Consistent improvements across three different linear-attention variants (Lightning, GLA, GDN) suggest the method generalizes to any linear-attention formulation.

---

## Conclusion

### Main Takeaways

- FlashMorph2 identifies and resolves two off-policy mismatches in FlashMorph's layer selection: teacher-forced continuous mixing (vs. discrete hybrid computation) and frozen weights (vs. post-recovery adapted weights).
- By sampling discrete hybrid architectures, performing end-to-end forward passes with self-generated hidden states, and jointly optimizing selection scores and model weights under the recovery objective, FlashMorph2 consistently outperforms all baselines on long-context and recall-intensive benchmarks.
- The method maintains high efficiency and scalability, completing layer selection in under 4 GPU hours even for 8B models.

### Future Directions

The paper suggests that on-policy principles could be extended to other aspects of model conversion and architecture search. Future work could explore:
- Applying on-policy selection to other architectural decisions (e.g., which layers to quantize, prune, or sparsify)
- Extending to larger model scales and additional attention variants
- Investigating the interaction between layer selection and other recovery training stages

---

_Markdown view of https://picx.dev/p/5jeUyR, served by PicX — AI-generated visual whiteboard summaries of research papers._
