Multi-layer Indexer: Overlapping Selection and Attention in DeepSeek Sparse Attention
Summary (Overview)
- Core contribution: The paper introduces the Multi-layer Indexer (MLI), a drop-in scheduling technique that overlaps the indexer scan with attention computation in DeepSeek Sparse Attention (DSA), eliminating the serial dependency between selection and attention at each layer.
- Key insight: Adjacent DSA layers select nearly identical token sets (Jaccard overlap ~0.42 on DeepSeek-V3.2, ~0.26 on GLM-5), so a layer can attend the selection produced by a previous indexer layer rather than waiting for its own.
- No training required: MLI needs no retraining and no new weights; it changes only which selection each layer attends, leaving all indexer arithmetic and selections unchanged.
- Composition with IndexCache: MLI composes with IndexCache (cross-layer indexer reuse), yielding up to 14.5% faster decode on DeepSeek-V3.2 and 9.8% on GLM-5 at retention 1/2, with quality staying within a few percentage points of IndexCache.
- Free retention halving: At retention , IndexCache+MLI is faster than IndexCache at while attending twice as many fresh selections, reaching 1.32× the unmodified model's throughput at 128K context and retention 1/4.
Introduction and Theoretical Foundation
Background
DeepSeek Sparse Attention (DSA) is the attention mechanism behind DeepSeek-V3.2 and the GLM-5 family. At each layer, a lightning indexer with small heads scores every cached token:
where is the indexer's score for query and cached token , the superscript marks the indexer's own projections, indexes its heads, and are low-rank projections of the layer input with a per-head weight.
The Problem
The layer keeps the set of the highest-scoring tokens and runs sparse attention over those tokens only. This creates a serial dependency: attention cannot begin until the top-k is known. While sparse attention's cost is flat in sequence length , the indexer scans all keys—at 128K context, the scan constitutes roughly two-thirds of the attention-side layer cost.
Key Observation
The cost of the indexer per decode step factors into two independent components:
- How many indexers run (addressed by IndexCache, which keeps indexers on a fraction of layers)
- How long each indexer is waited for (addressed by MLI)
Since adjacent layers select strongly overlapping token sets, a layer's attention need not wait for its own selection—an earlier layer's selection suffices.
Methodology
The MLI Schedule
Let denote the top-2048 tokens selected by full layer , and be the set of full layers. For any layer , let be the nearest full layer at or before , and step back full layers along (the lag depth). Every layer attends:
Key properties:
- With : standard DSA () or IndexCache ()
- With : MLI when , or the composition when
- The first full layers run the standard schedule (no predecessor exists)
Realization in the Decode Graph
The implementation in SGLang's DSA decode path:
- Two-stream execution: At a full layer, the indexer (projections, scan over paged index-key cache, top-k) is enqueued on a second CUDA stream while attention proceeds on the main stream with the assigned selection.
- CUDA graph capture: Cross-stream edges are ordinary graph dependencies, so the schedule is captured once and replayed without host involvement.
- Memory safety: References to the indexer's inputs/outputs are held until the join point (k full layers later) to prevent premature memory reuse.
Distillation for Quality Recovery
To recover selection quality, only the indexers (≈0.5% of parameters) are trained so that a full layer's selection matches what its consumer layers would have selected. The per-layer loss combines KL divergence and a listwise Plackett–Luce term:
where is the set of consumer layers, are softmax teacher/student scores, is the teacher's top-k ranking, and .
Fused Kernel
A fused decode kernel runs sparse attention and the indexer's prefix scan in a single launch, so the scan never competes for streaming multiprocessors with a separate launch. This helps DeepSeek-V3.2 (4–6 percentage points faster) but hurts GLM-5 (which has half as many index heads per rank).
Empirical Validation / Results
Throughput Gains (Single Request)
Table 1: Per-request decode throughput at single concurrency (percentage change over base; base in tok/s):
| DeepSeek-V3.2 retention | 10K IC | +MLI | 60K IC | +MLI | 120K IC | +MLI | 128K IC | +MLI |
|---|---|---|---|---|---|---|---|---|
| none (base, tok/s) | 89.1 | 88.0 | 85.7 | 85.5 | ||||
| all (MLI alone) | +21.5 | +19.8 | +26.1 | +25.1 | ||||
| 1/2 | +11.5 | +25.3 | +12.3 | +25.2 | +13.6 | +30.1 | +13.7 | +29.8 |
| 1/4 | +19.1 | +27.3 | +20.5 | +28.3 | +22.8 | +32.5 | +22.7 | +32.3 |
| 1/8 | +23.4 | +28.5 | +25.2 | +29.8 | +27.9 | +33.7 | +28.0 | +33.7 |
| 1/16 | +25.3 | +26.3 | +27.1 | +28.2 | +30.2 | +31.6 | +30.2 | +31.7 |
| 1/32 | +26.1 | +26.9 | +28.2 | +28.5 | +31.4 | +32.2 | +31.4 | +32.2 |
| GLM-5 retention | 10K IC | +MLI | 60K IC | +MLI | 120K IC | +MLI | 195K IC | +MLI |
|---|---|---|---|---|---|---|---|---|
| none (base, tok/s) | 68.5 | 67.8 | 66.5 | 65.0 | ||||
| all (MLI alone) | +14.6 | +14.4 | +15.6 | +16.8 | ||||
| 1/2 | +8.2 | +17.6 | +8.7 | +17.9 | +9.8 | +19.8 | +10.6 | +21.4 |
| 1/4 | +13.7 | +18.8 | +14.2 | +19.2 | +15.9 | +21.6 | +17.7 | +23.7 |
| 1/8 | +16.3 | +19.4 | +17.1 | +20.4 | +19.3 | +22.8 | +21.3 | +25.0 |
Key findings:
- Equal retention: MLI+IndexCache is faster than IndexCache at every retention, with gains largest where indexers are densest (14.5% at 1/2 on DeepSeek-V3.2)
- Free halving: At retention , MLI+IndexCache beats IndexCache at —at 128K and 1/4, it achieves 1.32× base vs. IndexCache's 1.28× at 1/8
- Gains grow with context and persist on IndexCache's searched patterns
Quality Results
Table 2: Quality at reported retentions (percentage change over base; base = absolute score):
| DeepSeek-V3.2 | LongBench IC | +MLI | RULER IC | +MLI | LB v2 IC | +MLI |
|---|---|---|---|---|---|---|
| none (base) | 0.449 | 0.975 | 0.501 | |||
| all (MLI alone) | -0.0 | -2.1 | -0.4 | |||
| 1/2 | -0.2 | -0.3 | -2.7 | -4.0 | +0.8 | +0.4 |
| 1/4 | -0.3 | +0.5 | -0.9 | -0.4 | -1.6 | -0.4 |
| 1/8 | -0.1 | -1.1 | -0.7 | +0.3 | -0.4 | +0.4 |
| GLM-5 | LongBench IC | +MLI | RULER IC | +MLI | LB v2 IC | +MLI |
|---|---|---|---|---|---|---|
| none (base) | 0.453 | 0.941 | 0.477 | |||
| all (MLI alone) | -0.2 | +2.3 | -0.8 | |||
| 1/2 | -1.2 | -1.1 | +2.9 | +2.1 | +1.2 | +0.4 |
| 1/4 | -0.2 | -0.4 | +0.7 | +1.6 | +2.5 | -0.4 |
| 1/8 | -0.0 | -1.4 | +1.6 | -1.3 | +0.8 | -1.7 |
Quality findings:
- MLI alone stays within 0.4–0.8 points of base on LongBench and LongBench v2
- On top of IndexCache, MLI costs at most 1–2 points
- Distillation is decisive at the "cliff": at 1/16 on GLM-5, it turns IndexCache's collapse (LongBench −11.5, RULER −61, needle 0.00) into recovery (LongBench −0.1, RULER −7.4, needle 1.00)
What Limits the Gain
- MLI hides about four-fifths of the exposed scan on DeepSeek-V3.2 and two-thirds on GLM-5
- The first full layer's scan remains serial (no earlier layer to overlap)
- The remaining limit is memory bandwidth: the scan hides only in HBM bandwidth that attention and feed-forward leave idle
- At full-KV-cache batch sizes (64 requests), the gain vanishes as requests saturate bandwidth
Theoretical and Practical Implications
Theoretical Significance
-
Selection-attention decoupling: The paper demonstrates that in DSA, the indexer-attention dependency is on the support of the softmax, not on shared computation—breaking this dependency is both possible and beneficial.
-
Shared primitive argument: The indexer score and attention logit are both low-rank bilinear interactions of the layer input. The indexer is a "second consumer" of what attention already computes, suggesting DSA models should be co-designed around this shared primitive rather than placing a selector ahead of attention.
-
Compositionality of optimizations: The two cost factors (number of indexers × wait time per indexer) multiply, and MLI's overlap technique composes orthogonally with IndexCache's layer-skipping.
Practical Implications
-
Drop-in deployment: MLI requires no retraining, no new weights, and runs inside SGLang's production decode graph with unchanged kernels.
-
Production serving benefits: At long contexts (where KV memory caps concurrency at 2–6 requests), MLI provides substantial throughput gains exactly where the indexer scan is most exposed.
-
Architecture guidance: The fused kernel's success on DeepSeek-V3.2 (but not GLM-5) provides evidence for co-designing DSA models around the shared bilinear primitive from the start.
Conclusion
The Multi-layer Indexer (MLI) eliminates the serial dependency between the indexer scan and attention in DSA by having each layer attend the selection of a previous indexer layer. The change is drop-in—no retraining, no new weights—and composes with IndexCache.
Key takeaways:
- Faster at equal retention: Up to 14.5% faster decode on DeepSeek-V3.2 and 9.8% on GLM-5 at retention 1/2
- Free halving of indexer budget: At retention , MLI+IndexCache is faster than IndexCache at while attending twice as many fresh selections
- Quality preserved: Within a few percentage points of IndexCache across LongBench, RULER, and LongBench v2
- Future direction: The indexer only recomputes the bilinear interaction attention already forms—co-designing DSA around this shared primitive is the natural next step
Limitations: The gain diminishes with batch size (bandwidth-bound), the fused kernel helps only DeepSeek-V3.2, and GLM-5.2 (which ships IndexShare natively) is out of scope for equal-footing comparison.
Related papers
- Do LLM Agents Still Need to Be Told How to Self-Improve?
Newer LLM generations spontaneously construct their own self-improvement algorithms from primitive tools alone, transferring learned strategies across tasks and models without prescribed learning loops.
- The Critic Can Hurt: On the Safety of Critics in LLM-Guided AutoML Search
A naive LLM critic can drive AutoML search below the no-critic random-search floor on 7 of 8 tabular datasets, with losses up to 0.245 AUC.
- Shared Weights, Selected Computations: How Looped Transformers Route What Each Loop Does
Looped Transformers perform state-dependent computation, not fixed one-loop-one-step algorithms, and a learned linear layer can steer frozen shared weights via attention routing.