HySparse2: Hybrid Sparse Attention with Two-Level KV Sharing

Summary (Overview)

• HySparse2 is a novel hybrid sparse attention architecture designed for long-horizon, multi-turn agentic workloads, combining YOCO-style decoder splitting with HySparse's KV reuse mechanism.
• The architecture introduces two-level KV sharing: (1) KV Bridging at the outer level connects full-attention layers across self-decoder and cross-decoder, and (2) KV Reuse at the inner level shares KV caches and selection indices within hybrid blocks.
• Two key refinements over HySparse: token-level sparsity replaces block-level selection for finer long-context retrieval, and a forced local window replaces the separate SWA branch, enabling complete prefill early-exit after the self-decoder.
• On 80B-A3B MoE models, HySparse2 achieves 2.92× and 5.02× reductions in prefill FLOPs relative to HySparse and Hybrid SWA at 1M tokens, with KV cache reduced to 2.69 GB (vs. 6.72 GB and 12.09 GB respectively).
• HySparse2 improves mean MRCR-v2 and RULER-v2 scores over HySparse by 11.30 and 19.81 percentage points respectively after post-training, while achieving lower AgentPPL and LongPPL at all evaluated lengths up to 256k.

Introduction and Theoretical Foundation

Background and Motivation

Agentic inference combines long contexts with multi-turn interaction, where short generated actions or tool calls return much longer search results, execution traces, or documents. As observations accumulate, agents must retrieve and combine evidence across expanding histories, creating three critical demands:

1. Efficient prefill — tool responses add substantially more tokens than the actions producing them
2. Compact KV-cache storage — growing contexts require minimal cache footprints
3. Accurate long-context retrieval — multi-turn trajectories require precise evidence location

Prior Work and Limitations

HySparse (Gao et al., 2026) addressed long-context efficiency by interleaving full-attention layers with sparse-attention layers, where full-attention layers supply selection indices and KV caches to following sparse layers. However, for agentic inference, HySparse leaves room for improvement in:

• Shortening prefill computation
• Further reducing KV-cache storage
• Improving retrieval precision on multi-turn tasks

Theoretical Foundation

The design builds on several key insights:

YOCO-style architecture (Sun et al., 2024): Divides the backbone into a self-decoder (constructing a global KV cache) and a cross-decoder (sharing that cache), allowing prefill cache construction to exit early.

Cross-layer KV sharing: Reduces both KV storage and prefill computation by constructing cross-decoder KV caches from self-decoder hidden states.

Sparse attention granularity trade-off: Block-level sparsity was previously adopted for kernel efficiency, but agentic workloads with long-horizon multi-turn trajectories require token-level precision for faithful retrieval.

Key Design Principle

Full attention remains important for model quality and serves as an "indexer" — using exact attention scores to provide oracle token selection for subsequent sparse layers. This supports native end-to-end training without a separate indexer module or auxiliary distillation objective.

Methodology

Architecture Overview

HySparse2 follows the YOCO structure:

• Self-decoder: Hybridizes full attention with sliding-window attention (SWA) for local modeling
• Cross-decoder: Hybridizes full attention with sparse attention for global retrieval

KV Bridging (Outer Level)

KV Bridging operates only between full-attention layers in the self-decoder and cross-decoder. For a pair of layers $(i, j)$:

$$\mathbf{K}_j^{\text{cross}} = \operatorname{Proj}_j^{K}\left(\mathbf{H}_i^{\text{self}}\right), \quad \mathbf{V}_j^{\text{cross}} = \operatorname{Proj}_j^{V}\left(\mathbf{H}_i^{\text{self}}\right), \quad \mathbf{Q}_j^{\text{cross}} = \operatorname{Proj}_j^{Q}\left(\mathbf{H}_j^{\text{cross}}\right) \tag{1}$$

Each cross-decoder full-attention layer has its own K/V projections, so layers sharing a hidden state source still construct distinct KV caches. One self-decoder full-attention layer can supply multiple cross-decoder full-attention layers.

KV Reuse (Inner Level)

Token-Level Sparse Selection: Instead of block-level selection, HySparse2 applies top-$k$ selection to individual tokens. For a fixed attention budget, token-level selection can allocate capacity to individual positions across the context rather than spending budget on neighbors of selected blocks.

Removing Separate SWA Branch: A local window of the most recent tokens is forced into the sparse selection — always selected first, followed by highest-scoring tokens outside the window. Both selected sets are read from the full-attention KV cache.

This design enables complete early exit after the self-decoder during prefill. A separate SWA branch would require building a suffix of KV cache from projections of its own hidden states, creating cascading dependencies that grow linearly with depth.

Model Configurations

HySparse2 uses MQA (multi-query attention) for smaller KV cache and better token sparse attention kernel efficiency, while baselines use GQA.

Inference Architecture

Prefill–Decode Disaggregation: The prefill node hosts only the self-decoder and KV Bridging projections — for a 49-layer model, only the first 25 layers, cutting prefill node memory by nearly half. Cross-decoder full-attention KV caches are computed on the prefill node and transferred to decode nodes.

Speculative Decoding: Uses a single MTP layer during pretraining, which could be replaced with a larger DFlash-style drafter conditioned on self-decoder/cross-decoder boundary hidden states during post-training.

Training Setup

• 80B-A3B MoE models: 49 Transformer layers, hidden size 2,048, simplified mHC variant with identity residual mixing matrix
• Pretraining: ~500B tokens at 32k context; post-training: ~100B tokens with agentic data, extended to 256k context
• Muon optimizer with WSD schedule, peak LR $10^{-3}$ (pretraining) and $5 \times 10^{-5}$ (post-training)
• HySparse2 uses partial RoPE (64 rotary dims, base 10,000) for SWA layers; full and sparse attention use NoPE
• Sparse selection: 128 forced local tokens + 1,024 global tokens (token-level)

Empirical Validation / Results

Pretraining Performance

HySparse2 achieves the highest RULER and NoLiMa scores, improving over HySparse by 5.88 and 9.49 points respectively, while remaining broadly comparable on general capabilities.

Agentic and Long-Context Results (Post-Training)

• Mean MRCR-v2: +11.30 points over HySparse, +6.44 over Hybrid SWA
• Mean RULER-v2: +19.81 points over HySparse, +18.65 over Hybrid SWA
• At 256k context: HySparse2 reaches 58.45 RULER-v2 vs. 32.61 (HySparse) and 35.74 (Hybrid SWA)
• Lower AgentPPL and LongPPL at all evaluated context lengths up to 256k

Prefill Efficiency

At 1M tokens:

• Prefill FLOPs: 2.92× reduction vs. HySparse, 5.02× vs. Hybrid SWA
• KV cache: 2.69 GB (HySparse2) vs. 6.72 GB (HySparse) vs. 12.09 GB (Hybrid SWA)

Ablation 1: Token-Level vs. Block-Level Sparsity

Token-level selection improves long-context retrieval and graph reasoning under the same attention budget, with gains present within the 32k training context.

Ablation 2: Local Window in Sparse Layers

Forced SWA remains competitive with Gated SWA while avoiding additional projection parameters and local KV cache, making early-exit prefill feasible.

Ablation 3: KV Bridging

With vs. without bridging (290B-A8B scale, ~1.8T tokens):

KV Bridging preserves quality comparable to baseline while improving LongPPL.

KV Bridging vs. KV Mirror (U-shaped connections): KV Bridging finishes at 87.65 RULER vs. 81.28 for KV Mirror, as full-layer states retain more global information for projection due to denser supervision through full attention.

Theoretical and Practical Implications

Architectural Insights

1. Full attention as oracle indexer: Retaining a small number of full-attention layers serves dual purposes — maintaining model quality and providing exact selection indices for sparse layers without auxiliary distillation.

2. Dependency removal for early exit: By forcing a local window into sparse selection (rather than maintaining a separate SWA branch), HySparse2 eliminates cascading SWA dependencies that would otherwise prevent prefill early-exit. The KV cache in the cross-decoder can be constructed entirely from self-decoder hidden states via projections.

3. Token-level granularity at scale: While block-level selection was previously seen as a practical compromise for kernel efficiency, modern agentic workloads and advances in sparse kernels (Wang et al., 2025) make token-level implementations practical and necessary for multi-turn retrieval precision.

Practical Benefits for Deployment

• Prefill–decode disaggregation: Prefill nodes host only ~half the model weights (self-decoder + KV Bridging projections), reducing memory requirements by nearly 50%.
• KV-cache construction: Only one full-attention layer is needed during prefill for KV-cache construction.
• Early-exit prefill: All cross-decoder KV caches are built from self-decoder hidden states, allowing prefill to exit after the self-decoder.

Trade-offs and Limitations

• Small degradations on some general reasoning tasks (DROP, GSM8K) due to the forced SWA design versus gated SWA
• The accuracy–efficiency trade-off for token-level sparsity kernels and MQA configuration requires careful tuning
• Larger drafter training for speculative decoding and asynchronous execution coordination remain future work

Conclusion

HySparse2 presents a hybrid sparse attention architecture specifically designed for long-horizon, multi-turn agentic workloads. Its two-level KV sharing mechanism enables:

• Faster prefill: 2.92× reduction in prefill FLOPs vs. HySparse, 5.02× vs. Hybrid SWA at 1M tokens
• Smaller KV cache: 2.69 GB vs. 6.72 GB (HySparse) and 12.09 GB (Hybrid SWA) at 1M tokens
• Better long-context quality: Improved MRCR-v2 (+11.30), RULER-v2 (+19.81), and lower perplexity on agent trajectories

Future directions include:

1. Replacing retained full-attention layers with lightweight indexer plus sparse attention in post-training
2. Reducing query heads in full-attention layers while increasing them in sparse-attention layers
3. Training larger DFlash-style drafters for asynchronous speculative decoding
4. Exploring further optimizations to token-level sparse kernels
5. Validating the approach at even larger scales and longer context lengths