Summary (Overview)
- Core Problem: Converting pretrained Transformers into hybrid attention models (full + linear attention) requires selecting which layers retain full attention under a budget. FlashMorph2 identifies and fixes two "off-policy" mismatches in FlashMorph's layer selection.
- Key Contribution: FlashMorph2 introduces an on-policy layer-selection method that (I) evaluates sampled discrete hybrid architectures end-to-end using self-generated hidden states, and (II) jointly optimizes layer-selection scores and model weights under the recovery objective.
- Main Results: FlashMorph2 improves average scores over FlashMorph by +2.42 points on RULER, +1.08 on LongBench, and +0.76 on recall-intensive tasks, across Qwen3-0.6B and Qwen3-1.7B backbones with three linear-attention variants.
- Efficiency: Layer selection completes in only 0.72–3.57 GPU hours across 0.6B–8B models, running 8.4–20.8× faster than HALO and 560–854× faster than KL-LS.
- Identified Limitations in FlashMorph: (L1) continuous branch interpolation and teacher-forced hidden states differ from discrete hybrid computation; (L2) frozen model weights fail to account for parameter adaptation during subsequent recovery.
Introduction and Theoretical Foundation
Background
Hybrid attention models retain full attention in only a few layers and use linear attention elsewhere, preserving long-context modeling and precise retrieval while substantially reducing computation and KV-cache memory usage. To avoid prohibitive pretraining costs, existing approaches reuse pretrained LLM weights and replace a subset of full attention layers with linear attention, followed by distillation and continued training.
The Layer Selection Challenge
A central challenge is determining which full-attention layers to retain under a given budget. Since these layers are the only ones that can directly access the entire context, their placement largely determines how much of the original model's capability can be recovered.
Limitations of FlashMorph
FlashMorph's layer selection suffers from two off-policy mismatches:
- L1: Off-policy computation — Continuous gates mix full- and linear-attention outputs computed from teacher hidden states. At deployment, only one branch is retained per layer and hidden states are generated by the hybrid model itself. Complementarities observed under continuous mixing may not transfer to the final discrete architecture, and error accumulation/cross-layer compensation is missed.
- L2: Off-policy weights — Weights used for layer selection remain frozen during gate optimization, without adapting to the hybrid architecture. Layer selection cannot account for cross-layer co-adaptation during subsequent recovery.
Theoretical Foundation
The paper draws on on-policy learning principles from reinforcement learning and sequence generation, where supervision is obtained on states induced by the learner's own policy rather than ground-truth (teacher-forced) states. This mitigates exposure bias and distribution mismatch.
Methodology
Preliminaries: Attention Formulations
Full attention (causal masking) computes:
This incurs computational cost and KV-cache memory.
Linear attention uses a fixed-size recurrent state:
where and , incurring cost and cache.
Hybrid attention combines both via a binary mask :
Given a budget , the feasible set is .
FlashMorph Pipeline (Four Stages)
-
Stage 1 — Layerwise alignment: Linear-attention branches are initialized from pretrained weights and trained to match teacher attention outputs:
-
Stage 2 — Layer selection: Continuous gates interpolate between branches with teacher-forced states:
-
Stage 3 — Logits distillation: KL divergence from the frozen teacher:
-
Stage 4 — Long-context finetuning: Standard next-token prediction:
FlashMorph2: On-Policy Layer Selection
1. On-Policy Hybrid Computation
Each layer gets a learnable score . To explore architectures, Gumbel noise is added and Top-K selection is applied:
This induces a distribution over hybrid architectures in . The model performs an end-to-end forward pass through the sampled discrete architecture, allowing earlier branch choices to affect subsequent layer inputs.
2. On-Policy Weight Adaptation
Layer-selection parameters and model weights are jointly optimized:
The distillation loss is computed on sampled architectures, with gradients for both and obtained through a single backward pass.
3. Budget-Constrained Straight-Through Estimator
To handle non-differentiable discrete Top-K selection, a continuous relaxation preserving the budget constraint is constructed:
where is found via binary search. The gradient straight-through estimator is:
The biased gradient estimator is:
where and .
After optimization, the final architecture is , and the model is reinitialized with for Stages 3–4.
Empirical Validation / Results
Experimental Setup
- Backbones: Qwen3-0.6B and Qwen3-1.7B
- Linear attention variants: Lightning Attention, GLA, Gated DeltaNet (GDN)
- Hybrid ratio: 3:1 linear-to-full attention by default
- Baselines: Uniform, PostNAS, KL-LS, HALO, FlashMorph
- Evaluation: RULER (13 tasks), LongBench (14 tasks), 6 recall-intensive tasks, perplexity, commonsense reasoning
Long-Context Performance (RULER & LongBench)
Key results from Table 1 (0.6B backbone, GDN variant):
| Method | RULER Avg | LongBench Avg |
|---|---|---|
| FlashMorph | 60.29 | 29.15 |
| FlashMorph2 | 63.65 | 29.77 |
Key results from Table 1 (1.7B backbone, GDN variant):
| Method | RULER Avg | LongBench Avg |
|---|---|---|
| FlashMorph | 73.07 | 32.62 |
| FlashMorph2 | 74.46 | 33.81 |
Averaged across all six configurations: FlashMorph2 raises RULER from 63.26 to 65.68 (+2.42) and LongBench from 29.86 to 30.94 (+1.08) over FlashMorph.
Recall-Intensive Tasks
Key results from Table 2 (1.7B backbone, GDN variant):
| Method | FDA | SWDE | SQD | NQ | TQA | DROP | Avg |
|---|---|---|---|---|---|---|---|
| FlashMorph | 59.53 | 65.17 | 53.25 | 32.06 | 66.00 | 34.64 | 51.77 |
| FlashMorph2 | 61.71 | 66.88 | 53.62 | 31.58 | 66.94 | 33.21 | 52.32 |
Averaged across all configurations: FlashMorph2 improves from 48.94 to 49.70 (+0.76) over FlashMorph.
Layer-Selection Efficiency
- FlashMorph2 requires 1.7–2.6× the FLOPs and 1.9–2.4× the GPU hours of FlashMorph
- Completes in 0.72–3.57 GPU hours across 0.6B–8B models
- HALO requires 8.4–20.8× more GPU hours; KL-LS requires 560–854× more
- Selection overhead is <1% of the overall conversion token budget
Ablation Study (Table 3)
| Method | RULER | LongBench |
|---|---|---|
| FlashMorph2 | 74.46 | 33.81 |
| w/o On-Policy Computation | 44.91 | 28.10 |
| w/o On-Policy Weight | 69.58 | 32.43 |
| w/o Both (FlashMorph) | 73.07 | 32.62 |
Key findings:
- Removing on-policy computation causes a -29.55 point drop on RULER, as the variant concentrates all full-attention layers in the deepest layers {20, 21, 23–27}, misallocating the budget.
- Removing on-policy weight adaptation lowers RULER by 4.88 points, replacing layers {6, 17, 19, 20} with {0, 1, 2, 26}.
- Both components are complementary — removing either degrades performance.
Hybrid Ratios
- At the tightest budget (6:1 ratio), FlashMorph2 achieves 60.12 on GLA and 64.45 on GDN, outperforming Uniform by +10.65 and +7.86 points.
- At 1:1 ratio, FlashMorph2 reaches 76.93 on GLA (+1.42 over FlashMorph) and 80.69 on GDN (+0.92).
Theoretical and Practical Implications
Theoretical Contributions
-
On-policy principle for architecture search: FlashMorph2 generalizes on-policy learning beyond sequence generation to the domain of neural architecture selection, demonstrating that evaluating architectures under self-generated states (rather than teacher-forced states) yields more reliable importance estimates.
-
Joint optimization of selection and weights: The paper shows that layer importance is not intrinsic to a frozen model but co-adapts with weights during recovery training. Optimizing selection scores and weights jointly under the same objective as deployment reflects the true post-adaptation performance landscape.
-
Budget-constrained STE: The implicit differentiation of the threshold provides a principled way to propagate gradients through discrete Top-K selection while preserving the exact budget constraint.
Practical Implications
-
Efficient Transformer-to-hybrid conversion: FlashMorph2 makes hybrid model conversion practical for production-scale LLMs by reducing layer selection to under a few GPU hours while improving quality.
-
Scalability: Demonstrated across 0.6B–8B models, FlashMorph2's efficiency advantage grows with model scale, making it suitable for larger models.
-
Variant-agnostic: Consistent improvements across three different linear-attention variants (Lightning, GLA, GDN) suggest the method generalizes to any linear-attention formulation.
Conclusion
Main Takeaways
- FlashMorph2 identifies and resolves two off-policy mismatches in FlashMorph's layer selection: teacher-forced continuous mixing (vs. discrete hybrid computation) and frozen weights (vs. post-recovery adapted weights).
- By sampling discrete hybrid architectures, performing end-to-end forward passes with self-generated hidden states, and jointly optimizing selection scores and model weights under the recovery objective, FlashMorph2 consistently outperforms all baselines on long-context and recall-intensive benchmarks.
- The method maintains high efficiency and scalability, completing layer selection in under 4 GPU hours even for 8B models.
Future Directions
The paper suggests that on-policy principles could be extended to other aspects of model conversion and architecture search. Future work could explore:
- Applying on-policy selection to other architectural decisions (e.g., which layers to quantize, prune, or sparsify)
- Extending to larger model scales and additional attention variants
- Investigating the interaction between layer selection and other recovery training stages
Related papers
- The Critic Can Hurt: On the Safety of Critics in LLM-Guided AutoML Search
A naive LLM critic can drive AutoML search below the no-critic random-search floor on 7 of 8 tabular datasets, with losses up to 0.245 AUC.
- Pro$^2$ Agent: Scientific Problem Probing for Autonomous AI R&D
Pro²Agent's explicit Problem State—tracking diagnosed bottlenecks, candidate solutions, and evidence—achieves a record 76% medal rate on MLE-Bench, outperforming code-centric agents.
- Rethinking Fixed-Horizon Latent Reasoning
Adaptive latent inference that stops reasoning when progress stalls improves accuracy by up to 7.2 points while cutting computation by 29%, as completing fixed latent horizons can degrade performance.