# Should You Split Decode? Disaggregation Across Future Models and Accelerators

> SiliconForge shows attention/FFN disaggregation yields no robust advantage over prefill/decode disaggregation, with prefill provisioning, memory bandwidth, and pool granularity driving LLM serving performance more than the split itself.

- **Source:** [arXiv](https://arxiv.org/abs/2611.08299)
- **Published:** 2026-11-07
- **Permalink:** https://picx.dev/p/ECRbqg
- **Whiteboard:** https://picx.dev/p/ECRbqg/image

## Summary

## Summary (Overview)

- **Core Question**: Does attention/feed-forward network disaggregation (AFD) repay its costs compared to standard prefill/decode disaggregation (P/D) across future LLM models and accelerator hardware?
- **Key Contribution**: Introduces **SiliconForge**, a calibrated cross-layer model that independently varies serving decomposition (P/D vs. P/D+AFD) and hardware assignment, enabling prospective comparison before future models and accelerators are built.
- **Main Finding**: AFD shows **no robust deployment-level advantage** in SLO goodput per kW across the tested accelerator classes and model directions; most nominal estimates favor P/D.
- **Critical Insight**: The largest AFD lead comes from fitting one more prefill replica, not from a more efficient decode pool—highlighting that prefill provisioning, decode memory bandwidth, links, and pool granularity matter more than the split itself.
- **Actionable Recommendation**: Fixed hardware assignments lose advantage as models and objectives change; re-optimizing mappings and pool sizes cuts early commitment's median normalized goodput loss from **36% to 9%**.

---

## Introduction and Theoretical Foundation

### Background
LLM inference has two distinct phases with different resource demands:
- **Prefill**: compute-intensive, sets **time to first token (TTFT)**
- **Decode**: memory-bandwidth-bound, reads growing KV cache and FFN weights, sets **time per output token (TPOT)**

This mismatch creates two design decisions:
1. **Serving decomposition**: which roles get their own accelerator pool
2. **Hardware assignment**: which accelerator class serves each pool

### Two Disaggregation Strategies
- **P/D (Prefill/Decode disaggregation)**: Standard industry practice, separating prefill from decode (supported by NVIDIA Dynamo, vLLM, SGLang)
- **P/D+AFD (Attention/FFN disaggregation)**: Further splits decode attention from FFN work—deployed by ByteDance on ~10,000 GPUs and planned in NVIDIA's Vera Rubin platform (Rubin GPUs for attention, Groq 3 LPX for MoE FFNs)

### Key Theoretical Challenge
AFD adds a new serving boundary, creating:
- Additional data movement (each token crosses the boundary **twice per MoE layer**, e.g., 116 times in DeepSeek-V3)
- Placement constraints and potentially stranded capacity
- Another provisioned pool with independent sizing

The fundamental question: **Does role-specific placement repay these costs?**

### Motivation for Prospective Modeling
Infrastructure is committed before future models and hardware can be benchmarked. Prior studies often change boundary and hardware assignment together, conflating the two effects. The paper argues for studying these choices independently.

---

## Methodology

### SiliconForge Framework

**Architecture Catalog**: 39 accelerators in six archetypes, with four prospective classes for the main comparison:

| Class | Anchor | Memory | FP8 PFLOPS | Mem. TB/s | Cap. GiB | Power W |
|-------|--------|--------|------------|-----------|----------|---------|
| GPU | Rubin | HBM4 | 1.25 | 17.5 | 288 | 1,800 |
| Systolic | TPU v7 | HBM3e | 1.0 | 7.4 | 192 | 600 |
| 3D-DRAM | Raptor | 2-chip card | 1.5 | 100 | 32 | 844 |
| SRAM/dataflow | Groq 3 LPX | 8-LPU tray | 1.25 | 1,200 | 4.0 | 3,000 |

**Model Directions**: Four prospective LLM directions anchored in published models:
- **Expert-heavy MoE** (from Kimi K2): increases FFN work and expert communication
- **KV-optimized MLA** (from DeepSeek-V3): reduces KV bytes per token via latent attention
- **KV-heavy GQA-MoE** (from Qwen3-235B): increases KV cache capacity demands
- **Sparse-attention MLA** (from DeepSeek-V3.2-Exp): reduces KV reads per step

### Key Modeling Equations

**Decode Schedule Period** (homogeneous P/D+AFD stack, M ≤ 8 microbatches):
$$T = L \max\left( M \max(t_A, t_F, t_D, t_C), \; t_A + t_D + t_F + t_C \right)$$

where $t_A$, $t_D$, $t_F$, and $t_C$ are one microbatch's attention, dispatch, FFN, and combine times per layer.

**Service Time per Role**:
$$t_r = \max(\Phi_r, \Theta_r) + (1-\gamma)\min(\Phi_r, \Theta_r) + t_{TP} + t_0$$

where $\Phi_r$ and $\Theta_r$ are compute and memory times, $\gamma$ is the calibrated overlap ratio, $t_{TP}$ is TP all-reduce time, and $t_0$ is fixed per-layer latency.

**Capacity Bound**:
$$U \leq \frac{DP_A \cdot s_{kv}(\phi H - H_c - \omega)}{(C + O)\kappa}$$

where $DP_A$ counts attention replicas, $s_{kv}$ is the KV shard factor, $\kappa$ is KV bytes per token, $C+O$ is peak resident context, and $\omega$ is other resident state.

### Evaluation Protocol
- **Two-part test**: A margin is "robust" only if nominal magnitude exceeds the 9% envelope AND sign holds at every tested specification corner
- **32 specification corners** (64 with photonic band) combining compute, bandwidth, capacity, link, and power endpoints
- Common 128 kW power budget; both designs independently optimized for maximum SLO goodput

---

## Empirical Validation / Results

### Finding I: No Robust AFD Advantage

Under the headline comparisons (four directions × two fabrics × two link-sharing modes):

- Adding the attention/FFN boundary moves SLO goodput per kW by **−10.4% to +8.1%**
- **Seven of eight** AFD+ margins lie inside the 9% envelope
- Role-specific placement changes little—the extra class choices improve margins in only 4 of 8 cases

**Key Insight**: The largest AFD lead (+8.1% in KV-heavy GQA-MoE on photonic fabric) comes from fitting **one more prefill replica** (11 vs. 10), not from a more efficient decode pool.

### Finding II: What Actually Drives Performance

**Memory bandwidth dominates decode**:
- Doubling memory bandwidth cuts TPOT by **39–49%**
- Doubling FLOPs cuts TPOT by at most **1.8%**

**Context shifts bottlenecks** (Figure 5):
- Memory reads take **57–95% of TPOT**
- KV reads overtake weight reads at ~5K tokens in the KV-heavy direction
- All fixed deployments run out of capacity before TPOT reaches the 50 ms target

**Inter-pool bandwidth knees**:
- Three of four directions reach 99% of best goodput at **11–16 GB/s per attention unit** per link direction
- AFD's exchange is hidden when the busier pool sets the steady-state period (Figure 2b)

**Scale-up domain effects**:
- In 8-unit domains, KV-heavy P/D decode pool spans several domains, giving AFD a nominal **9.2% lead** (falls to 2.8% with finer P/D pools)
- Larger domains or fast fabrics for both designs weaken this lead

### Finding III: Hardware Assignment Instability

- Changing objective from goodput per kW to per deployment unit moves **6 of 8** best deployments from systolic to GPU hardware
- Fixed deployments suffer **median 35.7% loss** of scenario-best goodput per kW
- Re-optimizing mappings and pool sizes cuts this to **9.3%**
- Additional reserves (25% units or power) lower it to 5.1%; switching serving design to 5.7%

### Calibration Results

| Check | Error |
|-------|-------|
| P/D vs. unified | 6.2% |
| P/D+AFD vs. co-located | 8.0% |
| 9% comparison envelope | — |

Validated against 252 throughput results and 47 primary comparison pairs from 18 published sources.

---

## Theoretical and Practical Implications

### For System Architects
1. **Pool allocation matters more than the split**: The decision of how many replicas to provision for prefill vs. decode often determines the outcome more than whether AFD is used
2. **Memory bandwidth is the bottleneck**: Accelerator roadmaps should prioritize memory bandwidth over raw FLOPs for decode performance
3. **Pool granularity is critical**: Power-of-two EP rules can create oversized decode pools, hiding AFD's benefits; finer sizing can erase most AFD leads

### For Hardware Designers
- The 9% comparison envelope means small margins should not drive silicon decisions
- Link bandwidth has a "knee" (~11–16 GB/s per attention unit); beyond this, extra bandwidth adds little goodput
- Scale-up domain size significantly affects whether AFD provides value

### For Fleet Planners
- **Avoid early lock-in**: Hardware assignments should follow model demands as they evolve
- **Keep mappings adjustable**: Re-optimization on the same hardware recovers most of the goodput lost to early commitment
- **Capability preservation** is recommended over committing to the highest-scoring accelerator at one operating point

---

## Conclusion

### Main Takeaways

1. **No robust AFD advantage exists** across the tested design space—most nominal estimates favor P/D
2. **AFD's nominal lead is conditional**: It appears in KV-heavy models when co-located expert pools span small scale-up domains, but larger domains or fast fabrics erase it
3. **The levers that decide deployments are**: prefill provisioning, decode memory bandwidth, links, and pool granularity—not the split alone
4. **Hardware assignments are not stable**: They change with objectives, specification corners, and model directions
5. **Re-optimization recovers most of the cost of early commitment**: From 36% to 9% median normalized goodput loss

### Recommendations
> "We recommend keeping mappings and pool sizes adjustable to let boundary and hardware choices follow model demands."

### Future Directions
- Update SiliconForge as future LLM and hardware specifications become clearer
- Evaluate speculative decoding and diffusion decoding extensions
- Test with prefix caching and other production-serving optimizations
- Explore heterogeneous deployments beyond the four studied accelerator classes

---

_Markdown view of https://picx.dev/p/ECRbqg, served by PicX — AI-generated visual whiteboard summaries of research papers._