Summary (Overview)

  • Core Problem: Converting pretrained Transformers into hybrid attention models (full + linear attention) requires selecting which layers retain full attention under a budget. FlashMorph2 identifies and fixes two "off-policy" mismatches in FlashMorph's layer selection.
  • Key Contribution: FlashMorph2 introduces an on-policy layer-selection method that (I) evaluates sampled discrete hybrid architectures end-to-end using self-generated hidden states, and (II) jointly optimizes layer-selection scores and model weights under the recovery objective.
  • Main Results: FlashMorph2 improves average scores over FlashMorph by +2.42 points on RULER, +1.08 on LongBench, and +0.76 on recall-intensive tasks, across Qwen3-0.6B and Qwen3-1.7B backbones with three linear-attention variants.
  • Efficiency: Layer selection completes in only 0.72–3.57 GPU hours across 0.6B–8B models, running 8.4–20.8× faster than HALO and 560–854× faster than KL-LS.
  • Identified Limitations in FlashMorph: (L1) continuous branch interpolation and teacher-forced hidden states differ from discrete hybrid computation; (L2) frozen model weights fail to account for parameter adaptation during subsequent recovery.

Introduction and Theoretical Foundation

Background

Hybrid attention models retain full attention in only a few layers and use linear attention elsewhere, preserving long-context modeling and precise retrieval while substantially reducing computation and KV-cache memory usage. To avoid prohibitive pretraining costs, existing approaches reuse pretrained LLM weights and replace a subset of full attention layers with linear attention, followed by distillation and continued training.

The Layer Selection Challenge

A central challenge is determining which full-attention layers to retain under a given budget. Since these layers are the only ones that can directly access the entire context, their placement largely determines how much of the original model's capability can be recovered.

Limitations of FlashMorph

FlashMorph's layer selection suffers from two off-policy mismatches:

  • L1: Off-policy computation — Continuous gates mix full- and linear-attention outputs computed from teacher hidden states. At deployment, only one branch is retained per layer and hidden states are generated by the hybrid model itself. Complementarities observed under continuous mixing may not transfer to the final discrete architecture, and error accumulation/cross-layer compensation is missed.
  • L2: Off-policy weights — Weights used for layer selection remain frozen during gate optimization, without adapting to the hybrid architecture. Layer selection cannot account for cross-layer co-adaptation during subsequent recovery.

Theoretical Foundation

The paper draws on on-policy learning principles from reinforcement learning and sequence generation, where supervision is obtained on states induced by the learner's own policy rather than ground-truth (teacher-forced) states. This mitigates exposure bias and distribution mismatch.


Methodology

Preliminaries: Attention Formulations

Full attention (causal masking) computes:

ofull,t=∑i=1texp⁡(qtki⊤/d)vi∑j=1texp⁡(qtkj⊤/d)WO(1)\mathbf{o}_{\mathrm{full},t} = \frac{\sum_{i=1}^{t} \exp\left(\mathbf{q}_t \mathbf{k}_i^{\top} / \sqrt{d}\right) \mathbf{v}_i}{\sum_{j=1}^{t} \exp\left(\mathbf{q}_t \mathbf{k}_j^{\top} / \sqrt{d}\right)} \mathbf{W}_O \tag{1}

This incurs O(T2)\mathcal{O}(T^2) computational cost and O(T)\mathcal{O}(T) KV-cache memory.

Linear attention uses a fixed-size recurrent state:

olin,t=qtStWO,St=St−1+kt⊤vt(2)\mathbf{o}_{\mathrm{lin},t} = \mathbf{q}_t \mathbf{S}_t \mathbf{W}_O, \quad \mathbf{S}_t = \mathbf{S}_{t-1} + \mathbf{k}_t^{\top} \mathbf{v}_t \tag{2}

where St∈Rd×d\mathbf{S}_t \in \mathbb{R}^{d \times d} and S0=0\mathbf{S}_0 = 0, incurring O(T)\mathcal{O}(T) cost and O(1)\mathcal{O}(1) cache.

Hybrid attention combines both via a binary mask m∈{0,1}L\mathbf{m} \in \{0,1\}^L:

o(l)=mlofull(l)+(1−ml)olin(l)(3)\mathbf{o}^{(l)} = m_l \mathbf{o}_{\mathrm{full}}^{(l)} + (1 - m_l) \mathbf{o}_{\mathrm{lin}}^{(l)} \tag{3}

Given a budget KK, the feasible set is CK={T⊆[L]:∣T∣=K}\mathcal{C}_K = \{\mathcal{T} \subseteq [L] : |\mathcal{T}| = K\}.

FlashMorph Pipeline (Four Stages)

  1. Stage 1 — Layerwise alignment: Linear-attention branches are initialized from pretrained weights and trained to match teacher attention outputs:

    Lalign=1L∑l=1L∥Alin(l)(hT(l))−Afull(l)(hT(l))∥22(4)\mathcal{L}_{\mathrm{align}} = \frac{1}{L}\sum_{l=1}^{L}\left\| A_{\mathrm{lin}}^{(l)}(\mathbf{h}_{\mathrm{T}}^{(l)}) - A_{\mathrm{full}}^{(l)}(\mathbf{h}_{\mathrm{T}}^{(l)}) \right\|_2^2 \tag{4}
  2. Stage 2 — Layer selection: Continuous gates αl\alpha_l interpolate between branches with teacher-forced states:

    omix(l)=αlofull,T(l)+(1−αl)olin,T(l)(5)\mathbf{o}_{\mathrm{mix}}^{(l)} = \alpha_l \mathbf{o}_{\mathrm{full,T}}^{(l)} + (1-\alpha_l) \mathbf{o}_{\mathrm{lin,T}}^{(l)} \tag{5} LLS=1L∑l=1L∥omix(l)−ofull,T(l)∥22+λ∑l=1Lα(l)(6)\mathcal{L}_{\mathrm{LS}} = \frac{1}{L}\sum_{l=1}^{L}\left\| \mathbf{o}_{\mathrm{mix}}^{(l)} - \mathbf{o}_{\mathrm{full,T}}^{(l)} \right\|_2^2 + \lambda \sum_{l=1}^{L} \alpha^{(l)} \tag{6}
  3. Stage 3 — Logits distillation: KL divergence from the frozen teacher:

    LKD=∑DKL(pT(⋅∣x≤t)∥pθ(⋅∣x≤t))(7)\mathcal{L}_{\mathrm{KD}} = \sum D_{\mathrm{KL}}\left(p_{\mathrm{T}}(\cdot \mid x_{\leq t}) \| p_{\theta}(\cdot \mid x_{\leq t})\right) \tag{7}
  4. Stage 4 — Long-context finetuning: Standard next-token prediction:

    LFT=−∑log⁡pθ(xt+1∣x≤t)(8)\mathcal{L}_{\mathrm{FT}} = -\sum \log p_{\theta}(x_{t+1} \mid x_{\leq t}) \tag{8}

FlashMorph2: On-Policy Layer Selection

1. On-Policy Hybrid Computation

Each layer gets a learnable score ϕl\phi_l. To explore architectures, Gumbel noise is added and Top-K selection is applied:

I=TopK⁡(ϕ+g,K),gl∼i.i.d.Gumbel⁡(0,1)(10)\mathcal{I} = \operatorname{TopK}(\phi + \mathbf{g}, K), \quad g_l \stackrel{\text{i.i.d.}}{\sim} \operatorname{Gumbel}(0,1) \tag{10}

This induces a distribution qϕq_\phi over hybrid architectures in CK\mathcal{C}_K. The model performs an end-to-end forward pass through the sampled discrete architecture, allowing earlier branch choices to affect subsequent layer inputs.

2. On-Policy Weight Adaptation

Layer-selection parameters ϕ\phi and model weights θ\theta are jointly optimized:

min⁡ϕ,θJ(ϕ,θ)=EI∼qϕ[LKD(θ;I)](11)\min_{\phi,\theta} \mathcal{J}(\phi,\theta) = \mathbb{E}_{\mathcal{I} \sim q_\phi}\left[\mathcal{L}_{\mathrm{KD}}(\theta; \mathcal{I})\right] \tag{11}

The distillation loss is computed on sampled architectures, with gradients for both ϕ\phi and θ\theta obtained through a single backward pass.

3. Budget-Constrained Straight-Through Estimator

To handle non-differentiable discrete Top-K selection, a continuous relaxation preserving the budget constraint is constructed:

m~l=σ(ϕ~l−τ),∑l=1Lσ(ϕ~l−τ)=K(12)\tilde{m}_l = \sigma(\tilde{\phi}_l - \tau), \quad \sum_{l=1}^{L} \sigma(\tilde{\phi}_l - \tau) = K \tag{12}

where τ\tau is found via binary search. The gradient straight-through estimator is:

mST=sg⁡(m−m~)+m~(13)\mathbf{m}_{\mathrm{ST}} = \operatorname{sg}\left(\mathbf{m} - \tilde{\mathbf{m}}\right) + \tilde{\mathbf{m}} \tag{13}

The biased gradient estimator is:

∇^ϕjLKD=sj(uj−uˉ),uˉ=∑l=1Lslul∑l=1Lsl(14)\widehat{\nabla}_{\phi_j} \mathcal{L}_{\mathrm{KD}} = s_j (u_j - \bar{u}), \quad \bar{u} = \frac{\sum_{l=1}^{L} s_l u_l}{\sum_{l=1}^{L} s_l} \tag{14}

where sl=m~l(1−m~l)s_l = \tilde{m}_l(1-\tilde{m}_l) and ul=∂LKD/∂mlu_l = \partial \mathcal{L}_{\mathrm{KD}}/\partial m_l.

After optimization, the final architecture is T⋆=TopK⁡(ϕ,K)\mathcal{T}^\star = \operatorname{TopK}(\phi, K), and the model is reinitialized with θ0\theta_0 for Stages 3–4.


Empirical Validation / Results

Experimental Setup

  • Backbones: Qwen3-0.6B and Qwen3-1.7B
  • Linear attention variants: Lightning Attention, GLA, Gated DeltaNet (GDN)
  • Hybrid ratio: 3:1 linear-to-full attention by default
  • Baselines: Uniform, PostNAS, KL-LS, HALO, FlashMorph
  • Evaluation: RULER (13 tasks), LongBench (14 tasks), 6 recall-intensive tasks, perplexity, commonsense reasoning

Long-Context Performance (RULER & LongBench)

Key results from Table 1 (0.6B backbone, GDN variant):

MethodRULER AvgLongBench Avg
FlashMorph60.2929.15
FlashMorph263.6529.77

Key results from Table 1 (1.7B backbone, GDN variant):

MethodRULER AvgLongBench Avg
FlashMorph73.0732.62
FlashMorph274.4633.81

Averaged across all six configurations: FlashMorph2 raises RULER from 63.26 to 65.68 (+2.42) and LongBench from 29.86 to 30.94 (+1.08) over FlashMorph.

Recall-Intensive Tasks

Key results from Table 2 (1.7B backbone, GDN variant):

MethodFDASWDESQDNQTQADROPAvg
FlashMorph59.5365.1753.2532.0666.0034.6451.77
FlashMorph261.7166.8853.6231.5866.9433.2152.32

Averaged across all configurations: FlashMorph2 improves from 48.94 to 49.70 (+0.76) over FlashMorph.

Layer-Selection Efficiency

  • FlashMorph2 requires 1.7–2.6× the FLOPs and 1.9–2.4× the GPU hours of FlashMorph
  • Completes in 0.72–3.57 GPU hours across 0.6B–8B models
  • HALO requires 8.4–20.8× more GPU hours; KL-LS requires 560–854× more
  • Selection overhead is <1% of the overall conversion token budget

Ablation Study (Table 3)

MethodRULERLongBench
FlashMorph274.4633.81
w/o On-Policy Computation44.9128.10
w/o On-Policy Weight69.5832.43
w/o Both (FlashMorph)73.0732.62

Key findings:

  • Removing on-policy computation causes a -29.55 point drop on RULER, as the variant concentrates all full-attention layers in the deepest layers {20, 21, 23–27}, misallocating the budget.
  • Removing on-policy weight adaptation lowers RULER by 4.88 points, replacing layers {6, 17, 19, 20} with {0, 1, 2, 26}.
  • Both components are complementary — removing either degrades performance.

Hybrid Ratios

  • At the tightest budget (6:1 ratio), FlashMorph2 achieves 60.12 on GLA and 64.45 on GDN, outperforming Uniform by +10.65 and +7.86 points.
  • At 1:1 ratio, FlashMorph2 reaches 76.93 on GLA (+1.42 over FlashMorph) and 80.69 on GDN (+0.92).

Theoretical and Practical Implications

Theoretical Contributions

  1. On-policy principle for architecture search: FlashMorph2 generalizes on-policy learning beyond sequence generation to the domain of neural architecture selection, demonstrating that evaluating architectures under self-generated states (rather than teacher-forced states) yields more reliable importance estimates.

  2. Joint optimization of selection and weights: The paper shows that layer importance is not intrinsic to a frozen model but co-adapts with weights during recovery training. Optimizing selection scores and weights jointly under the same objective as deployment reflects the true post-adaptation performance landscape.

  3. Budget-constrained STE: The implicit differentiation of the threshold τ\tau provides a principled way to propagate gradients through discrete Top-K selection while preserving the exact budget constraint.

Practical Implications

  1. Efficient Transformer-to-hybrid conversion: FlashMorph2 makes hybrid model conversion practical for production-scale LLMs by reducing layer selection to under a few GPU hours while improving quality.

  2. Scalability: Demonstrated across 0.6B–8B models, FlashMorph2's efficiency advantage grows with model scale, making it suitable for larger models.

  3. Variant-agnostic: Consistent improvements across three different linear-attention variants (Lightning, GLA, GDN) suggest the method generalizes to any linear-attention formulation.


Conclusion

Main Takeaways

  • FlashMorph2 identifies and resolves two off-policy mismatches in FlashMorph's layer selection: teacher-forced continuous mixing (vs. discrete hybrid computation) and frozen weights (vs. post-recovery adapted weights).
  • By sampling discrete hybrid architectures, performing end-to-end forward passes with self-generated hidden states, and jointly optimizing selection scores and model weights under the recovery objective, FlashMorph2 consistently outperforms all baselines on long-context and recall-intensive benchmarks.
  • The method maintains high efficiency and scalability, completing layer selection in under 4 GPU hours even for 8B models.

Future Directions

The paper suggests that on-policy principles could be extended to other aspects of model conversion and architecture search. Future work could explore:

  • Applying on-policy selection to other architectural decisions (e.g., which layers to quantize, prune, or sparsify)
  • Extending to larger model scales and additional attention variants
  • Investigating the interaction between layer selection and other recovery training stages

Related papers