Summary (Overview)

• Core Question: Does attention/feed-forward network disaggregation (AFD) repay its costs compared to standard prefill/decode disaggregation (P/D) across future LLM models and accelerator hardware?
• Key Contribution: Introduces SiliconForge, a calibrated cross-layer model that independently varies serving decomposition (P/D vs. P/D+AFD) and hardware assignment, enabling prospective comparison before future models and accelerators are built.
• Main Finding: AFD shows no robust deployment-level advantage in SLO goodput per kW across the tested accelerator classes and model directions; most nominal estimates favor P/D.
• Critical Insight: The largest AFD lead comes from fitting one more prefill replica, not from a more efficient decode pool—highlighting that prefill provisioning, decode memory bandwidth, links, and pool granularity matter more than the split itself.
• Actionable Recommendation: Fixed hardware assignments lose advantage as models and objectives change; re-optimizing mappings and pool sizes cuts early commitment's median normalized goodput loss from 36% to 9%.

---

Introduction and Theoretical Foundation

Background

LLM inference has two distinct phases with different resource demands:

• Prefill: compute-intensive, sets time to first token (TTFT)
• Decode: memory-bandwidth-bound, reads growing KV cache and FFN weights, sets time per output token (TPOT)

This mismatch creates two design decisions:

1. Serving decomposition: which roles get their own accelerator pool
2. Hardware assignment: which accelerator class serves each pool

Two Disaggregation Strategies

• P/D (Prefill/Decode disaggregation): Standard industry practice, separating prefill from decode (supported by NVIDIA Dynamo, vLLM, SGLang)
• P/D+AFD (Attention/FFN disaggregation): Further splits decode attention from FFN work—deployed by ByteDance on ~10,000 GPUs and planned in NVIDIA's Vera Rubin platform (Rubin GPUs for attention, Groq 3 LPX for MoE FFNs)

Key Theoretical Challenge

AFD adds a new serving boundary, creating:

• Additional data movement (each token crosses the boundary twice per MoE layer, e.g., 116 times in DeepSeek-V3)
• Placement constraints and potentially stranded capacity
• Another provisioned pool with independent sizing

The fundamental question: Does role-specific placement repay these costs?

Motivation for Prospective Modeling

Infrastructure is committed before future models and hardware can be benchmarked. Prior studies often change boundary and hardware assignment together, conflating the two effects. The paper argues for studying these choices independently.

---

Methodology

SiliconForge Framework

Architecture Catalog: 39 accelerators in six archetypes, with four prospective classes for the main comparison:

Model Directions: Four prospective LLM directions anchored in published models:

• Expert-heavy MoE (from Kimi K2): increases FFN work and expert communication
• KV-optimized MLA (from DeepSeek-V3): reduces KV bytes per token via latent attention
• KV-heavy GQA-MoE (from Qwen3-235B): increases KV cache capacity demands
• Sparse-attention MLA (from DeepSeek-V3.2-Exp): reduces KV reads per step

Key Modeling Equations

Decode Schedule Period (homogeneous P/D+AFD stack, M ≤ 8 microbatches):

$$T = L \max\left( M \max(t_A, t_F, t_D, t_C), \; t_A + t_D + t_F + t_C \right)$$

where $t_A$, $t_D$, $t_F$, and $t_C$ are one microbatch's attention, dispatch, FFN, and combine times per layer.

Service Time per Role:

$$t_r = \max(\Phi_r, \Theta_r) + (1-\gamma)\min(\Phi_r, \Theta_r) + t_{TP} + t_0$$

where $\Phi_r$ and $\Theta_r$ are compute and memory times, $\gamma$ is the calibrated overlap ratio, $t_{TP}$ is TP all-reduce time, and $t_0$ is fixed per-layer latency.

Capacity Bound:

$$U \leq \frac{DP_A \cdot s_{kv}(\phi H - H_c - \omega)}{(C + O)\kappa}$$

where $DP_A$ counts attention replicas, $s_{kv}$ is the KV shard factor, $\kappa$ is KV bytes per token, $C+O$ is peak resident context, and $\omega$ is other resident state.

Evaluation Protocol

• Two-part test: A margin is "robust" only if nominal magnitude exceeds the 9% envelope AND sign holds at every tested specification corner
• 32 specification corners (64 with photonic band) combining compute, bandwidth, capacity, link, and power endpoints
• Common 128 kW power budget; both designs independently optimized for maximum SLO goodput

---

Empirical Validation / Results

Finding I: No Robust AFD Advantage

Under the headline comparisons (four directions × two fabrics × two link-sharing modes):

• Adding the attention/FFN boundary moves SLO goodput per kW by −10.4% to +8.1%
• Seven of eight AFD+ margins lie inside the 9% envelope
• Role-specific placement changes little—the extra class choices improve margins in only 4 of 8 cases

Key Insight: The largest AFD lead (+8.1% in KV-heavy GQA-MoE on photonic fabric) comes from fitting one more prefill replica (11 vs. 10), not from a more efficient decode pool.

Finding II: What Actually Drives Performance

Memory bandwidth dominates decode:

• Doubling memory bandwidth cuts TPOT by 39–49%
• Doubling FLOPs cuts TPOT by at most 1.8%

Context shifts bottlenecks (Figure 5):

• Memory reads take 57–95% of TPOT
• KV reads overtake weight reads at ~5K tokens in the KV-heavy direction
• All fixed deployments run out of capacity before TPOT reaches the 50 ms target

Inter-pool bandwidth knees:

• Three of four directions reach 99% of best goodput at 11–16 GB/s per attention unit per link direction
• AFD's exchange is hidden when the busier pool sets the steady-state period (Figure 2b)

Scale-up domain effects:

• In 8-unit domains, KV-heavy P/D decode pool spans several domains, giving AFD a nominal 9.2% lead (falls to 2.8% with finer P/D pools)
• Larger domains or fast fabrics for both designs weaken this lead

Finding III: Hardware Assignment Instability

• Changing objective from goodput per kW to per deployment unit moves 6 of 8 best deployments from systolic to GPU hardware
• Fixed deployments suffer median 35.7% loss of scenario-best goodput per kW
• Re-optimizing mappings and pool sizes cuts this to 9.3%
• Additional reserves (25% units or power) lower it to 5.1%; switching serving design to 5.7%

Calibration Results

Validated against 252 throughput results and 47 primary comparison pairs from 18 published sources.

---

Theoretical and Practical Implications

For System Architects

1. Pool allocation matters more than the split: The decision of how many replicas to provision for prefill vs. decode often determines the outcome more than whether AFD is used
2. Memory bandwidth is the bottleneck: Accelerator roadmaps should prioritize memory bandwidth over raw FLOPs for decode performance
3. Pool granularity is critical: Power-of-two EP rules can create oversized decode pools, hiding AFD's benefits; finer sizing can erase most AFD leads

For Hardware Designers

• The 9% comparison envelope means small margins should not drive silicon decisions
• Link bandwidth has a "knee" (~11–16 GB/s per attention unit); beyond this, extra bandwidth adds little goodput
• Scale-up domain size significantly affects whether AFD provides value

For Fleet Planners

• Avoid early lock-in: Hardware assignments should follow model demands as they evolve
• Keep mappings adjustable: Re-optimization on the same hardware recovers most of the goodput lost to early commitment
• Capability preservation is recommended over committing to the highest-scoring accelerator at one operating point

---

Conclusion

Main Takeaways

1. No robust AFD advantage exists across the tested design space—most nominal estimates favor P/D
2. AFD's nominal lead is conditional: It appears in KV-heavy models when co-located expert pools span small scale-up domains, but larger domains or fast fabrics erase it
3. The levers that decide deployments are: prefill provisioning, decode memory bandwidth, links, and pool granularity—not the split alone
4. Hardware assignments are not stable: They change with objectives, specification corners, and model directions
5. Re-optimization recovers most of the cost of early commitment: From 36% to 9% median normalized goodput loss

Recommendations

"We recommend keeping mappings and pool sizes adjustable to let boundary and hardware choices follow model demands."

Future Directions

• Update SiliconForge as future LLM and hardware specifications become clearer
• Evaluate speculative decoding and diffusion decoding extensions
• Test with prefix caching and other production-serving optimizations
• Explore heterogeneous deployments beyond the four studied accelerator classes