Summary (Overview)

• HybridSpec is a throughput-aware speculative decoding system for hybrid GDN–MoE (Gated DeltaNet + Mixture-of-Experts) target models, addressing the critical bottleneck of verification cost in parallel drafting.
• Key insight: Verification time grows 71% from length 4 to 16 for hybrid models versus only 4%–6% for dense models, making verification length selection crucial. HybridSpec selects the verification length on the GPU using draft-side acceptance estimates and measured target latency.
• Novel contributions: (1) GPU-resident length selection with a lightweight routing correction at the MoE router boundary, (2) two-stage verification that pauses after the first MoE router, (3) parallel GDN verification kernels with one-launch accepted-state reconstruction.
• Results: HybridSpec achieves 1.75–1.88× speedup over autoregressive decoding and 1.30–1.33× over vanilla DFlash across three GDN–MoE target models (Qwen3.5-35B-A3B, Qwen3.6-35B-A3B, Qwen3.5-122B-A10B).

Introduction and Theoretical Foundation

Speculative decoding accelerates autoregressive generation by having a drafter propose multiple tokens and a target model verify them in one forward pass. Recent parallel drafters like DFlash generate blocks of draft tokens in a single pass, making drafting cheap and shifting the dominant cost to verification.

The paper focuses on a critical architectural trend: hybrid models that combine Gated DeltaNet (GDN) linear-attention layers with Mixture-of-Experts (MoE) layers. These architectures, used in models like Qwen3.5, Jamba, and MiniMax-01, exhibit a unique challenge:

• GDN cost: Each layer maintains a fixed-size recurrent state, and the default verifier updates and stores the state after every token (requiring rollback capability to the accepted prefix).
• MoE cost: Depends on expert reuse—additional tokens are cheap when reusing active experts but costly when activating new ones. Expert selections are only known after verification runs.

The core problem: Existing length selectors (Sequoia, DSpark, D-Cut, EVICT, EcoSpec, FASER) decide verification length before target execution, unable to leverage actual expert assignments. No prior work uses the target's own expert assignments before the experts execute.

Verification length trade-off: Verifying more draft tokens can accept more of them, but work after the first rejection is wasted. For hybrid models, this trade-off is particularly acute because both GDN and MoE costs grow rapidly with token count.

Methodology

3.1 Throughput Objective

HybridSpec formulates verification length selection as a pooled throughput maximization problem. For rounds with information $\mathcal{I}_1, \ldots, \mathcal{I}_N$, choosing length $M_r$ in round $r$ gives pooled throughput:

$$\frac{\sum_{r=1}^{N} G(M_r \mid \mathcal{I}_r)}{\sum_{r=1}^{N} L(M_r \mid \mathcal{I}_r)}$$

where $G(M \mid \mathcal{I}) = 1 + \sum_{i=1}^{M} p_i(\mathcal{I})$ is the expected emitted tokens and $L(M \mid \mathcal{I})$ is expected latency. By Dinkelbach's theorem, each round maximizes $G(M_r \mid \mathcal{I}_r) - \rho^* L(M_r \mid \mathcal{I}_r)$.

3.2 Two-Stage Verification at the Router Boundary

Verification is split at the router boundary (after the first MoE router, before routed experts execute):

• Stage 0: Processes $\min(w_0 + 1, w_{\text{rem}})$ tokens through Layer-0 GDN + MoE router, using pre-captured CUDA Graphs launched via conditional graph nodes (no CPU round trip).
• Routing correction: At the boundary, observes actual expert assignments and decides whether to keep the extra token based on predicted latency vs. acceptance benefit.

3.3 Length Selection and Routing Correction

The draft-side selector maximizes:

$$w_0 = \arg\max_{w \in \mathcal{W}_0} \left[ 1 + \sum_{i=1}^{w-1} \hat{p}_i - \rho C_{\text{tgt}}(w) \right]$$

The routing correction revises latency estimates using observed Layer-0 expert counts, deciding whether to expand by one token based on:

$$\Delta L_c = C_{\text{tgt}}(w_0+1) - C_{\text{tgt}}(w_0) - s + \eta\left(\bar{n}_{\text{all}}(w_0 \mid n_{\text{new}}) - \bar{n}_{\text{all}}(w_0)\right)$$

3.4 Parallel GDN Verification

The GDN update per token follows:

$$z_i = \beta_i(v_i - \alpha_i S_{i-1} k_i), \quad S_i = \alpha_i S_{i-1} + z_i k_i^{\top}$$

HybridSpec uses the chunkwise formulation to solve for all $w$ updates in parallel:

$$H Z = \operatorname{diag}(\beta)\left(V - \operatorname{diag}\left(e^{g}\right) K S_0^{\top}\right)$$

The commit kernel reconstructs every GDN layer's accepted state in one launch:

$$S_m = e^{g_m} S_0 + Z_{1:m}^{\top} \operatorname{diag}\left(e^{g_m - g_1}, \dots, e^{g_m - g_m}\right) K_{1:m}$$

Empirical Validation / Results

End-to-End Performance (Greedy Decoding)

On three target models across 7 categories (coding, math, reasoning, RAG, summarization, writing, roleplay):

HybridSpec verifies fewer tokens per round (mean length 6.4/6.2 vs. 16 for DFlash) with only 5% lower acceptance length—meaning skipped tokens would mostly have been rejected. Per-round time drops 24%–32% on 35B models.

Ablations (Qwen3.6-35B-A3B)

Length selection contributes the largest gain (1.19–1.25×), while routing correction (1.06×) and GDN kernels (1.05×) add meaningful improvements.

GDN Kernel Microbenchmarks

With all draft tokens accepted (lengths 4–16), HybridSpec kernels achieve 1.54–4.04× speedup over default SGLang kernels, with speedup growing with length. They also save memory: the default verifier reserves 1.88 GB (40% of attention-cache pool) for per-token state storage, while HybridSpec needs no such buffer.

Generalization

On 4 unseen SPEED-Bench categories (humanities, QA, STEM, multilingual), HybridSpec achieves 1.69× mean SR vs. 1.31× for vanilla DFlash without recalibration.

Theoretical and Practical Implications

Theoretical contributions:

• Provable correctness: In exact arithmetic, every selected length preserves greedy outputs (Proposition 1) and the sampling distribution under temperature sampling (Proposition 2).
• Dinkelbach-based optimization: Correctly casts length selection as pooled throughput maximization rather than per-round ratio optimization, which the paper shows produces suboptimal choices.
• Two-stage verification is theoretically sound because the routing correction operates only on observed expert assignments, not modifying acceptance probabilities.

Practical implications:

• Demonstrates that for hybrid GDN–MoE architectures, the verification length should be shorter and adaptive rather than always using full draft blocks.
• The approach of splitting verification at the router boundary and using target-side information (expert assignments) offers a new direction beyond draft-side-only methods like D-Cut and EVICT.
• CUDA Graph conditional nodes enable fully GPU-resident decision-making without CPU synchronization, a general technique applicable beyond this specific setting.

Conclusion

HybridSpec addresses verification cost in speculative decoding for hybrid GDN–MoE targets by:

1. Making throughput-optimal verification length choices on the GPU from acceptance estimates and measured latency
2. Refining decisions at the router boundary using observed expert assignments
3. Providing parallel GDN verification kernels with one-launch accepted-state reconstruction

The system achieves 1.75–1.88× speedup over autoregressive decoding and 1.30–1.33× over vanilla DFlash on three production-scale models. Future directions include extending to batched decoding (mentioned in appendices), temperature sampling, and potentially applying the router-boundary approach to other hybrid architectures.

AI Use Statement: The authors used generative AI tools for writing assistance, coding, and result checking, but took full responsibility for all content.