# Afore: Attention-FFN Disaggregated Serving with Dynamic Expert Reconfiguration

> AFORE exploits microbatch-level expert demand visibility and overlapped migration to dynamically reconfigure MoE expert placement, improving throughput by 10.1–17.6% and cutting P95 latency by 7.1–9.5%.

- **Source:** [arXiv](https://arxiv.org/abs/2610.03203)
- **Published:** 2026-10-17
- **Permalink:** https://picx.dev/p/hckzDY
- **Whiteboard:** https://picx.dev/p/hckzDY/image

## Summary

# AFORE: Attention–FFN Disaggregation with Overlapped Reconfiguration of Experts

## Summary (Overview)

- **Problem**: Mixture-of-Experts (MoE) models suffer from dynamic expert load imbalance in attention-FFN disaggregation (AFD) architectures, where overloaded experts directly bottleneck the independent FFN pipeline stage and degrade end-to-end serving performance.
- **Key insight**: AFD exposes two architectural opportunities: (1) microbatch-level expert demand is visible before FFN execution, and (2) expert migration can be overlapped with computation of preceding in-flight microbatches.
- **Solution**: AFORE formulates expert reconfiguration as a microbatch-aware scheduling problem with a migration-aware greedy algorithm, implementing demand prefetching and NVLink-based GPU-GPU expert migration.
- **Results**: On a 110B-parameter MoE model (GLM-4.5-Air), AFORE improves output throughput by 10.1–17.6% and reduces P95 inter-token latency by 7.1–9.5% versus the strongest baseline; 29.8% throughput improvement over static placement on average.
- **Efficiency**: Scheduler achieves near-optimal solutions (99.5% optimality across 1,400 targets) with sub-millisecond latency even at EP degree 128, and pipeline overlap fully hides migration cost.

## Introduction and Theoretical Foundation

### Background

MoE models are increasingly deployed in large-scale LLM services, activating only a small subset of experts per token to increase model capacity while keeping per-token computation manageable. However, serving MoE inference at scale faces three challenges:

1. **Large expert parameters** require distribution across devices
2. **Input-dependent expert activation** creates dynamic load imbalance
3. **Dynamic workloads** make static placement insufficient

### The AFD Bottleneck

Attention-FFN disaggregation (AFD) separates attention and feed-forward computation into independently provisioned worker pools. While this improves resource utilization, it amplifies the impact of expert load imbalance:

> "Once FFN computation becomes an independent pipeline stage, overloaded experts directly slow the FFN stage and degrade end-to-end serving performance."

**Experimental evidence**: Skewed routing reduces normalized throughput by 11.9% under monolithic execution but by 20.1% under AFD—the disaggregation makes imbalance 1.7× more harmful.

### Two Fundamental Challenges

- **C1: Dynamic and unpredictable demand** — Expert demand changes with input workload, model layer, sequence position, and request mix. Static placement or historical statistics cannot reliably describe upcoming expert-token distributions.
- **C2: Non-negligible system overhead** — Expert reconfiguration introduces runtime overhead (placement scheduling, weight migration, routing-metadata updates, worker coordination) that can offset load-balancing benefits.

### Two Key Opportunities from AFD

- **O1: Microbatch-level demand is visible** — The disaggregated pipeline exposes the expert-token distribution of a target microbatch before its FFN execution, enabling placement decisions based on near-future demand.
- **O2: Reconfiguration overhead can be overlapped** — The pipeline window between observing demand and FFN execution allows expert migration to overlap with preceding in-flight microbatch computation.

## Methodology

### Problem Formulation

AFORE formulates expert reconfiguration as an online, microbatch-driven placement problem. Key notation:

| Symbol | Meaning |
|--------|---------|
| $\mathcal{W}, N$ | Set of FFN expert workers, and their count |
| $\mathcal{E}, E$ | Set of MoE experts, and their count |
| $m_t$ | Target microbatch at scheduling epoch $t$ |
| $d_t^e$ | Token-expert assignments of $m_t$ routed to expert $e$ |
| $P = (x_{e,w})$ | Placement matrix; $x_{e,w}=1$ iff expert $e$ has a replica on worker $w$ |
| $K_w$ | Expert-slot (memory) budget of worker $w$ |
| $T_w(A_t)$ | Grouped-GEMM tile load of worker $w$ |
| $M_{\text{tile}}$ | Grouped-GEMM M-dimension tile size |

**Placement feasibility constraints** (Equation 1):

$$r_e(P) \geq 1 \ \forall e \in \mathcal{E}, \quad \sum_{e \in \mathcal{E}} x_{e,w} \leq K_w \ \forall w \in \mathcal{W}$$

**Worker tile load** (Equation 4):

$$T_w(A_t) = \sum_{e: a_t^{e,w} > 0} \left\lceil \frac{a_t^{e,w}}{M_{\mathrm{tile}}} \right\rceil$$

**Migration cost** (Equation 6-7): Switching from placement $P_{t-1}$ to $P_t$ requires transferring weights of new replicas:

$$\Delta(P_{t-1} \rightarrow P_t) = \{(e,w): x_{e,w}^t = 1 \wedge x_{e,w}^{t-1} = 0\}$$

The **exposed cost** after overlap with the pipeline window $W_t$:

$$O_t(\Delta) = \max\bigl(0, M(\Delta) - W_t\bigr)$$

**Net gain objective** (Equation 8):

$$G_t(P_t, A_t) = F(A_t^{\text{base}}) - F(A_t) - O_t\bigl(\Delta(P_{t-1} \to P_t)\bigr)$$

### Scheduling Algorithm

AFORE uses a three-phase greedy search:

1. **Assignment initialization**: Adapts the preceding assignment to target demand, dividing expert demand into tiles of size $M_{\text{tile}}$.
2. **Rebalancing within placement**: Redistributes tiles among existing replicas without adding weight copies, using move quantity:

$$q_e(h,c) = \min\left(n_t^{e,h}, \left\lfloor \frac{T_h - T_c}{2} \right\rfloor\right)$$

3. **Rebalancing beyond placement**: Explores new replicas on redundant slots, evaluating each candidate by net gain; only installs when $G_t > 0$.

### System Design

**Demand prefetching**: Attention workers compute gate/top-k results, extract compact demand vectors, and send them directly to the CPU scheduler via a registered host-memory mailbox (using StepMesh), bypassing the full execution input path.

**Overlapped migration**: FFN workers read plans from a shared-memory ring, submit missing replica copies on a dedicated migration stream (NVLink P2P transfers), and check copy-completion events before using new replicas. Exact token partitioning uses cumulative boundaries (Equation 11):

$$o_0 = 0, \qquad o_j = o_{j-1} + a_{t+1}^{e,w_j}$$

## Empirical Validation / Results

### Experimental Setup

- **Hardware**: 16 NVIDIA A100 GPUs (2 nodes × 8 GPUs), NVLink/NVSwitch intra-node (300 GB/s), RDMA inter-node (50 GB/s per bond)
- **Model**: GLM-4.5-Air (110B parameters), PD+AFD deployment: 4 prefill workers, 4 attention workers, 8 FFN workers
- **Workloads**: ShareGPT (conversational), FineWeb (document-style), CodeForces (code generation), GSM8K (mathematical reasoning)

### End-to-End Performance (RQ1)

| Metric | Improvement vs. Strongest Baseline |
|--------|-----------------------------------|
| Output throughput | +10.1% to +17.6% |
| P95 inter-token latency | −7.1% to −9.5% |

### Scheduler Quality (RQ2)

**Scheduling overhead**: Median latency ranges from 8.23 µs (EP=2) to 245.81 µs (EP=128)—sub-millisecond across all tested degrees.

**Solution quality** (Table 2):

| EP | 2 | 4 | 8 | 16 | 32 | 64 | 128 |
|----|---|---|---|---|----|----|-----|
| Optimal / 200 | 196 | 197 | 200 | 200 | 200 | 200 | 200 |
| Mean gap (%) | 0.04 | 0.03 | 0 | 0 | 0 | 0 | 0 |
| Max gap (%) | 3.60 | 1.89 | 0 | 0 | 0 | 0 | 0 |

### Component Ablation (RQ3)

| Configuration | Throughput vs. Static | P95 ITL vs. Static |
|---------------|----------------------|-------------------|
| Static | baseline | baseline |
| w/o Prefetch | −12.4% to −17.9% (vs. AFORE) | +6.3% to +10.5% |
| w/o Overlap | −15.2% to −21.3% (vs. AFORE) | +12.2% to +16.0% |
| **Full AFORE** | **+26.0% to +34.1%** | **−16.7% to −19.6%** |

### Migration Hiding (RQ4)

- **Raw migration cost**: 0.453–0.495 ms mean
- **Lookahead window**: 2.008–2.119 ms mean
- **Exposed migration cost**: 0.000 ms for all 324 sampled targets — **100% hidden**
- Median remaining time before use: 1.459–1.668 ms (minimum 0.447 ms)

## Theoretical and Practical Implications

### Theoretical Contributions

1. **Formal problem formulation**: AFORE provides the first formalization of temporal expert reconfiguration in AFD-based MoE serving as an online placement scheduling problem, jointly optimizing placement, assignment, and migration cost.

2. **Migration-aware cost model**: The key insight that only *exposed* migration cost (after pipeline overlap) matters, not raw cost, provides a principled framework for reasoning about reconfiguration benefits.

3. **Primary-plus-redundant layout**: The architectural pattern of fixed primaries plus reusable redundant slots offers a scalable approach to elastic expert placement.

### Practical Implications

1. **Production MoE serving**: AFORE demonstrates that dynamic expert reconfiguration is both necessary and feasible at microbatch granularity, with sub-millisecond scheduling overhead.

2. **AFD architecture adoption**: By mitigating the FFN-stage bottleneck, AFORE makes attention-FFN disaggregation more attractive for production deployment.

3. **Hardware utilization**: The 29.8% average throughput improvement over static placement translates directly to reduced GPU cost per token for MoE services.

## Conclusion

AFORE addresses the FFN-side load imbalance amplified by attention-FFN disaggregation in MoE serving by exploiting two properties of the disaggregated pipeline: upcoming microbatch demand is visible before FFN execution, and expert migration can overlap with preceding in-flight computation. The system formulates expert reconfiguration as a microbatch-aware scheduling problem with a migration-aware greedy algorithm, achieving:

- **10.1–17.6% throughput improvement** and **7.1–9.5% P95 latency reduction** over the strongest baseline
- **Near-optimal scheduling** (99.5% optimality) with sub-millisecond overhead
- **Full hiding of migration cost** through pipeline overlap

The results demonstrate that AFD exposes not only a new bottleneck for MoE serving, but also the temporal structure needed to mitigate it efficiently. Future directions include extending the approach to other disaggregated architectures, exploring multi-layer joint optimization, and adapting the scheduling framework to heterogeneous GPU clusters.

---

_Markdown view of https://picx.dev/p/hckzDY, served by PicX — AI-generated visual whiteboard summaries of research papers._