Summary (Overview)

• On-Demand Attention (ODA) is a local-first decoding method that uses a lightweight "recall head" to decide when to invoke global attention during LLM decoding, keeping pretrained weights frozen.
• The key finding is that decoding states available after local computation in frozen pretrained models already contain signals predictive of the benefit of global attention over local attention.
• ODA recovers most of the performance lost under local attention while substantially reducing the frequency of global attention calls (e.g., ~41% on RULER16K for Qwen3-1.7B) and achieves up to 2.65× decoding throughput over full attention in controlled vLLM measurements.
• The method works across model scales and families (Qwen3, Qwen3.5, Gemma) and architectures (full-attention, hybrid attention), with only the recall head trained using modest data and compute.

Introduction and Theoretical Foundation

Motivation

Reasoning and agentic workloads demand efficient long-context inference. Full-attention decoding reads the growing history at every step, but the benefit of global access varies across prediction positions. The paper asks a key question: before computing global attention for the current step, can we estimate the additional predictive benefit it would provide over local computation and use this estimate to decide whether to access the complete history?

Theoretical Basis

The paper builds on the observation that local computation has a dual role: its output supports the current prediction AND, together with other available states, helps determine whether additional global computation is worthwhile. This contrasts with dynamic attention methods that jointly train the language model and its access policy—ODA instead keeps pretrained weights unchanged and learns to read out the benefit signal from existing model states.

Methodology

Local-First Decoding

At decoding position $t$, the model processes input token $x_t$ to predict $x_{t+1}$. Local and Full are two access modes of the same frozen backbone $F_\phi$. The access sets are:

$$S^F_t = \{1, \ldots, t\}, \quad S^L_t = \{1, \ldots, \min(s, t)\} \cup \{\max(1, t-w+1), \ldots, t\}$$

where $s$ = initial positions (4) and $w$ = recent window width (2048).

Each step first executes Local, then computes a recall score:

$$(h^L_t, \Delta C^L_t) = F^L_\phi(x_t; C_{t-1}), \quad q_t = R_\psi\left(h_{t-1}, E_\phi(x_t), h^L_t\right)$$

If $q_t > \theta$, ODA recomputes with Full; otherwise accepts Local. The default threshold is $\theta = 0$.

Learning to Predict Recall Benefit

The access gain measures the predictive benefit of Full over Local:

$$g_t = \ell^L_t - \ell^F_t = \log p^F_t(x_{t+1}) - \log p^L_t(x_{t+1})$$

With a Full-call penalty $\lambda \geq 0$, the penalized one-step loss is:

$$\ell_t(a_t) = \ell^L_t - a_t(g_t - \lambda)$$

The head learns a regression score using a signed-log transform $T(u) = \text{sign}(u)\log(1 + |u|)$ and Huber regression with transition point 1:

$$\mathcal{L}(\psi) = \frac{1}{|\mathcal{E}|}\sum_{t \in \mathcal{E}} \ell_{\text{Huber}, 1}(q_t - y_t)$$

Computational Savings

For a Full-call rate $\rho$, ODA's average per-step savings relative to Full:

$$\Delta F(n, \rho) = (1-\rho)F_{\text{Full}}(n) - F_{\text{Local}} - F_{\text{Recall}}$$

Efficient Execution

A vLLM runtime uses GPU-side conditional CUDA Graphs, a persistent ring KV workspace for Local, and fused KV commits across layers to reduce synchronization and kernel-launch overhead.

Empirical Validation / Results

Generation Quality (RULER, 4K–64K)

Table 1 (excerpt, Qwen3-1.7B):

ODA preserves quality close to Full across all lengths, with score gaps below 1 point at 64K for all five models.

LongBench Results

On LongBench v1 (ten tasks), ODA's average score is within ~0.5 points of Full for all five models. For example, Qwen3-1.7B: Full = 39.50, ODA = 39.20 (80.12% Full-call rate).

State-Dependent Access Effectiveness

• At a 40% budget, the recall head captures 0.150 nats of net gain per reference position, 2.81× the expected gain of random selection.
• On five RULER16K tasks, ODA scores 88.96 at ~41% Full calls vs. Random's 32.79 at similar call rates.

Runtime Efficiency (Qwen3-1.7B, vLLM)

Table 20 (excerpt):

At 512K input with 12.5% Full calls, ODA achieves 2.65× throughput and uses only 15.69% of Full's FLOPs.

Recall-Head Input Ablation

Table 3 (Qwen3-1.7B, RULER16K):

The complete input set (preceding hidden state, current embedding, current Local state) supports accepting more Local outputs while preserving quality.

Theoretical and Practical Implications

• State-guided computation: The paper provides empirical evidence that pretrained decoding states can serve dual purposes—both token prediction and decisions about accessing distant information—without additional training of the backbone.
• Architecture insights: Hybrid models (Qwen3.5 with Gated DeltaNet layers) require fewer Full calls than full-attention models, suggesting that recurrent states reduce the need for explicit global reads. Larger models within a family also achieve better quality-access trade-offs.
• Practical efficiency: The speedup depends critically on context length and recall frequency; at short contexts (4K), ODA is 13.1% slower than Full, but benefits grow substantially with context length.
• Access value concentration: Learned policies show large early gains followed by smaller increments, indicating that global access provides unequal benefits across positions—most quality recovery occurs at relatively low call rates.

Conclusion

ODA demonstrates that frozen pretrained decoding states contain signals predictive of the benefit of additional global attention. By training only a lightweight recall head, ODA recovers most of the generation quality lost under local attention with fewer global calls while retaining the complete history. The vLLM runtime with GPU-side conditional execution delivers practical decoding speedups in controlled long-context measurements.

Future directions identified by the authors include:

• Learning recall on policy-generated histories rather than reference histories
• Accounting for downstream generation quality and cumulative computational cost
• Improving selection accuracy (the offline Oracle still captures more benefit than the current recall head)