AFORE: Attention–FFN Disaggregation with Overlapped Reconfiguration of Experts

Summary (Overview)

• Problem: Mixture-of-Experts (MoE) models suffer from dynamic expert load imbalance in attention-FFN disaggregation (AFD) architectures, where overloaded experts directly bottleneck the independent FFN pipeline stage and degrade end-to-end serving performance.
• Key insight: AFD exposes two architectural opportunities: (1) microbatch-level expert demand is visible before FFN execution, and (2) expert migration can be overlapped with computation of preceding in-flight microbatches.
• Solution: AFORE formulates expert reconfiguration as a microbatch-aware scheduling problem with a migration-aware greedy algorithm, implementing demand prefetching and NVLink-based GPU-GPU expert migration.
• Results: On a 110B-parameter MoE model (GLM-4.5-Air), AFORE improves output throughput by 10.1–17.6% and reduces P95 inter-token latency by 7.1–9.5% versus the strongest baseline; 29.8% throughput improvement over static placement on average.
• Efficiency: Scheduler achieves near-optimal solutions (99.5% optimality across 1,400 targets) with sub-millisecond latency even at EP degree 128, and pipeline overlap fully hides migration cost.

Introduction and Theoretical Foundation

Background

MoE models are increasingly deployed in large-scale LLM services, activating only a small subset of experts per token to increase model capacity while keeping per-token computation manageable. However, serving MoE inference at scale faces three challenges:

1. Large expert parameters require distribution across devices
2. Input-dependent expert activation creates dynamic load imbalance
3. Dynamic workloads make static placement insufficient

The AFD Bottleneck

Attention-FFN disaggregation (AFD) separates attention and feed-forward computation into independently provisioned worker pools. While this improves resource utilization, it amplifies the impact of expert load imbalance:

"Once FFN computation becomes an independent pipeline stage, overloaded experts directly slow the FFN stage and degrade end-to-end serving performance."

Experimental evidence: Skewed routing reduces normalized throughput by 11.9% under monolithic execution but by 20.1% under AFD—the disaggregation makes imbalance 1.7× more harmful.

Two Fundamental Challenges

• C1: Dynamic and unpredictable demand — Expert demand changes with input workload, model layer, sequence position, and request mix. Static placement or historical statistics cannot reliably describe upcoming expert-token distributions.
• C2: Non-negligible system overhead — Expert reconfiguration introduces runtime overhead (placement scheduling, weight migration, routing-metadata updates, worker coordination) that can offset load-balancing benefits.

Two Key Opportunities from AFD

• O1: Microbatch-level demand is visible — The disaggregated pipeline exposes the expert-token distribution of a target microbatch before its FFN execution, enabling placement decisions based on near-future demand.
• O2: Reconfiguration overhead can be overlapped — The pipeline window between observing demand and FFN execution allows expert migration to overlap with preceding in-flight microbatch computation.

Methodology

Problem Formulation

AFORE formulates expert reconfiguration as an online, microbatch-driven placement problem. Key notation:

Placement feasibility constraints (Equation 1):

$$r_e(P) \geq 1 \ \forall e \in \mathcal{E}, \quad \sum_{e \in \mathcal{E}} x_{e,w} \leq K_w \ \forall w \in \mathcal{W}$$

Worker tile load (Equation 4):

$$T_w(A_t) = \sum_{e: a_t^{e,w} > 0} \left\lceil \frac{a_t^{e,w}}{M_{\mathrm{tile}}} \right\rceil$$

Migration cost (Equation 6-7): Switching from placement $P_{t-1}$ to $P_t$ requires transferring weights of new replicas:

$$\Delta(P_{t-1} \rightarrow P_t) = \{(e,w): x_{e,w}^t = 1 \wedge x_{e,w}^{t-1} = 0\}$$

The exposed cost after overlap with the pipeline window $W_t$:

$$O_t(\Delta) = \max\bigl(0, M(\Delta) - W_t\bigr)$$

Net gain objective (Equation 8):

$$G_t(P_t, A_t) = F(A_t^{\text{base}}) - F(A_t) - O_t\bigl(\Delta(P_{t-1} \to P_t)\bigr)$$

Scheduling Algorithm

AFORE uses a three-phase greedy search:

1. Assignment initialization: Adapts the preceding assignment to target demand, dividing expert demand into tiles of size $M_{\text{tile}}$.
2. Rebalancing within placement: Redistributes tiles among existing replicas without adding weight copies, using move quantity:

$$q_e(h,c) = \min\left(n_t^{e,h}, \left\lfloor \frac{T_h - T_c}{2} \right\rfloor\right)$$

3. Rebalancing beyond placement: Explores new replicas on redundant slots, evaluating each candidate by net gain; only installs when $G_t > 0$.

System Design

Demand prefetching: Attention workers compute gate/top-k results, extract compact demand vectors, and send them directly to the CPU scheduler via a registered host-memory mailbox (using StepMesh), bypassing the full execution input path.

Overlapped migration: FFN workers read plans from a shared-memory ring, submit missing replica copies on a dedicated migration stream (NVLink P2P transfers), and check copy-completion events before using new replicas. Exact token partitioning uses cumulative boundaries (Equation 11):

$$o_0 = 0, \qquad o_j = o_{j-1} + a_{t+1}^{e,w_j}$$

Empirical Validation / Results

Experimental Setup

• Hardware: 16 NVIDIA A100 GPUs (2 nodes × 8 GPUs), NVLink/NVSwitch intra-node (300 GB/s), RDMA inter-node (50 GB/s per bond)
• Model: GLM-4.5-Air (110B parameters), PD+AFD deployment: 4 prefill workers, 4 attention workers, 8 FFN workers
• Workloads: ShareGPT (conversational), FineWeb (document-style), CodeForces (code generation), GSM8K (mathematical reasoning)

End-to-End Performance (RQ1)

Scheduler Quality (RQ2)

Scheduling overhead: Median latency ranges from 8.23 µs (EP=2) to 245.81 µs (EP=128)—sub-millisecond across all tested degrees.

Solution quality (Table 2):

Component Ablation (RQ3)

Migration Hiding (RQ4)

• Raw migration cost: 0.453–0.495 ms mean
• Lookahead window: 2.008–2.119 ms mean
• Exposed migration cost: 0.000 ms for all 324 sampled targets — 100% hidden
• Median remaining time before use: 1.459–1.668 ms (minimum 0.447 ms)

Theoretical and Practical Implications

Theoretical Contributions

1. Formal problem formulation: AFORE provides the first formalization of temporal expert reconfiguration in AFD-based MoE serving as an online placement scheduling problem, jointly optimizing placement, assignment, and migration cost.

2. Migration-aware cost model: The key insight that only exposed migration cost (after pipeline overlap) matters, not raw cost, provides a principled framework for reasoning about reconfiguration benefits.

3. Primary-plus-redundant layout: The architectural pattern of fixed primaries plus reusable redundant slots offers a scalable approach to elastic expert placement.

Practical Implications

1. Production MoE serving: AFORE demonstrates that dynamic expert reconfiguration is both necessary and feasible at microbatch granularity, with sub-millisecond scheduling overhead.

2. AFD architecture adoption: By mitigating the FFN-stage bottleneck, AFORE makes attention-FFN disaggregation more attractive for production deployment.

3. Hardware utilization: The 29.8% average throughput improvement over static placement translates directly to reduced GPU cost per token for MoE services.

Conclusion

AFORE addresses the FFN-side load imbalance amplified by attention-FFN disaggregation in MoE serving by exploiting two properties of the disaggregated pipeline: upcoming microbatch demand is visible before FFN execution, and expert migration can overlap with preceding in-flight computation. The system formulates expert reconfiguration as a microbatch-aware scheduling problem with a migration-aware greedy algorithm, achieving:

• 10.1–17.6% throughput improvement and 7.1–9.5% P95 latency reduction over the strongest baseline
• Near-optimal scheduling (99.5% optimality) with sub-millisecond overhead
• Full hiding of migration cost through pipeline overlap

The results demonstrate that AFD exposes not only a new bottleneck for MoE serving, but also the temporal structure needed to mitigate it efficiently. Future directions include extending the approach to other disaggregated architectures, exploring multi-layer joint optimization, and adapting the scheduling framework to heterogeneous GPU clusters.