Summary (Overview)

  • EchoMoE is a lightweight architecture modification that reduces expert parallelism (EP) communication in existing Mixture-of-Experts (MoE) large language models by exploiting cross-layer predictability of activations.
  • The method caches a full-width activation from an earlier layer as a reference and communicates only compressed differences relative to that reference, reducing inter-node RDMA traffic by 58% with only a 3.8 percentage point drop in task performance across 18 benchmarks.
  • EchoMoE improves inference throughput by 22.6% and reduces prefill latency by 23%, requiring approximately 8,500× fewer training tokens than full pretraining.
  • The approach works across multiple MoE architectures (Qwen3, Gemma 4, Kanana-2, Nemotron, Moonlight), roughly halving RDMA traffic with 2.4–5.9 pp task performance drops.
  • EchoMoE composes cleanly with FP8 quantization and achieves up to 2.38× throughput improvement over bandwidth-constrained Ethernet, with an efficient implementation released for vLLM integration.

Introduction and Theoretical Foundation

Background

Scaling large language models has driven the adoption of sparse Mixture-of-Experts (MoE) architectures, which increase parameter capacity without proportional increases in computation. At frontier scale, MoE models rely on Expert Parallelism (EP) to distribute experts across devices. Each MoE layer requires two synchronous collective communication steps:

  1. Dispatch: sending hidden states to ranks hosting selected experts
  2. Combine: returning expert outputs to the token's owner rank

As compute throughput has outpaced interconnect bandwidth, these synchronous collectives have become a major bottleneck. For DeepSeek-V3 served with 128 GPUs, a single 128K-token prefill moves up to 812 GiB of activations between nodes, with collectives accounting for 60% of transformer-layer latency (32% even after overlapping with computation).

Theoretical Foundation

The key insight is that transformer blocks update the residual stream incrementally, causing nearby MoE layers to receive highly correlated inputs. The paper quantifies this via two metrics:

  • Cosine similarity between a token's inputs to adjacent MoE layers
  • Predictability: fraction of variance in later-layer input explained by a per-dimension affine predictor fitted on earlier-layer input

For models with L ≥ 47 layers (Qwen3, DeepSeek-V3, GLM-5.3, Kanana-2), adjacent MoE inputs have mean cosine similarity above 90%, with predictors recovering over 80% of variance at k=1 and over 50% at k=4.

Communication Cost Model

The baseline inter-node activation traffic across L MoE layers is:

Cbase(T)=bdLmˉT⏟dispatch+bdLmˉT⏟combine=2bdLmˉT(1)C_{\text{base}}(T) = \underbrace{b d L \bar{m} T}_{\text{dispatch}} + \underbrace{b d L \bar{m} T}_{\text{combine}} = 2 b d L \bar{m} T \tag{1}

Where:

  • TT = number of tokens processed across the EP group
  • bb = bytes per activation dimension
  • dd = hidden dimension width
  • LL = number of MoE layers
  • mˉ\bar{m} = average number of remote nodes reached per token per collective

Methodology

EchoMoE Architecture

EchoMoE partitions the L MoE layers into groups of ss consecutive layers (stride). The first layer of each group is a reference layer that broadcasts its full-width state xℓ∈Rdx_\ell \in \mathbb{R}^d to all ranks, where it is cached as z∈Rdz \in \mathbb{R}^d. The s−1s-1 subsequent layers reuse zz and communicate only compressed changes.

Compressing Dispatch

For each non-reference layer, the owner rank compresses the difference between the current activation and the rescaled reference:

cd=Cd(xℓ−a⊙z)(2)c_d = C_d(x_\ell - a \odot z) \tag{2}

Where:

  • Cd∈Rrd×dC_d \in \mathbb{R}^{r_d \times d} = linear compressor
  • a∈Rda \in \mathbb{R}^d = learned gain vector
  • rdr_d = dispatch code width

Each expert rank reconstructs the activation:

x^ℓ=Ddcd+Gz(3)\hat{x}_\ell = D_d c_d + G z \tag{3}

Where Dd∈Rd×rdD_d \in \mathbb{R}^{d \times r_d} is the decompressor and G∈Rd×dG \in \mathbb{R}^{d \times d} is a learned gain matrix.

Folding: The decompressor DdD_d, reconstruction gain GG, and expert projections fold into a single expert-specific matrix:

W~in,i=[Wgate,iWup,i][GDd]∈R2f×(d+rd)(4)\widetilde{W}_{\text{in},i} = \begin{bmatrix} W_{\text{gate},i} \\ W_{\text{up},i} \end{bmatrix} \begin{bmatrix} G & D_d \end{bmatrix} \in \mathbb{R}^{2f \times (d + r_d)} \tag{4}

This avoids explicit reconstruction and adds no serving-time operations.

Compressing Combine

Expert outputs are compressed before the combine collective. Node j forms:

cc(j)=∑i∈SjwiCcEi(x^ℓ)=Cc∑i∈SjwiEi(x^ℓ)(5)c_c^{(j)} = \sum_{i \in S_j} w_i C_c E_i(\hat{x}_\ell) = C_c \sum_{i \in S_j} w_i E_i(\hat{x}_\ell) \tag{5}

The owner rank decompresses:

y^=Dc∑jcc(j)∈Rd(6)\hat{y} = D_c \sum_j c_c^{(j)} \in \mathbb{R}^d \tag{6}

Folding for combine: The compressor and down projection compose:

W~out,i=CcWdown,i∈Rrc×f(7)\widetilde{W}_{\text{out},i} = C_c W_{\text{down},i} \in \mathbb{R}^{r_c \times f} \tag{7}

Communication Cost

CEcho(T)=bdLsmmax⁡T⏟reference states+brds−1sLmˉT⏟dispatch compressed+brcLmˉT⏟combine compressed(8)C_{\text{Echo}}(T) = \underbrace{b d \frac{L}{s} m_{\max} T}_{\text{reference states}} + \underbrace{b r_d \frac{s-1}{s} L \bar{m} T}_{\text{dispatch compressed}} + \underbrace{b r_c L \bar{m} T}_{\text{combine compressed}} \tag{8}

Retrofitting Procedure

  1. Initialization: Run the uncompressed model, record activations, and solve closed-form minimization problems for compressors/decompressors:
min⁡Cd,DdE∥R1/2(I−DdCd)(xℓ−a⊙z)∥2andmin⁡Cc,DcE∥(I−DcCc)y∥2(10)\min_{C_d, D_d} \mathbb{E}\left\|R^{1/2}(I - D_d C_d)(x_\ell - a \odot z)\right\|^2 \quad \text{and} \quad \min_{C_c, D_c} \mathbb{E}\left\|(I - D_c C_c)y\right\|^2 \tag{10}

Where RR weights the reconstruction error by expert routing frequencies.

  1. End-to-End Training: Adapt the compressed model using KL divergence from the uncompressed model's output distribution plus small cross-entropy loss, using only 4.32B tokens (~8,500× fewer than pretraining).

Empirical Validation / Results

Task Performance and RDMA Reduction

EchoMoE configuration (s=4,rd=512,rc=768)(s=4, r_d=512, r_c=768) achieves 2.39× RDMA reduction with only 3.8 pp drop in task performance. Key results:

ConfigurationRDMA ReductionOverall Performance (Δ)
Uncompressed (d=2048)1.00×66.9
(4, 1280, 1280)1.46×65.5 (-1.4)
(4, 512, 1280)1.84×64.9 (-2.0)
(4, 512, 1024)2.08×63.8 (-3.1)
(4, 512, 768)2.39×63.1 (-3.8)
(6, 256, 768)3.02×58.3 (-8.6)
(8, 256, 512)4.03×55.7 (-11.2)

Cross-Model Generalization

ModelRDMA ReductionPerformance Drop
Gemma 4 26B-A4B2.01×-4.4 pp
Kanana-2-30B-A3B2.01×-2.4 pp
Nemotron-3.5-Lightning-30B-A3B1.91×-4.6 pp
Moonlight-16B-A3B1.99×-5.9 pp

Throughput and Latency

  • Prefill: 22.6% throughput improvement at 2.39× RDMA reduction; 42.9% at 4.03×
  • Time to first token (TTFT): reduced by 23.0% and 36.9% respectively
  • Decode: gains grow with requests per GPU, reaching 1.27× at 512 requests per GPU
  • Communication time: drops from 115 ms to 64 ms for prefill at (4, 512, 768)

Key Ablation Findings

  1. Reference states substantially improve dispatch compression: At width 512, plain dispatch loses 21.1 pp while reference dispatch at stride 4 loses only 2.6 pp
  2. Expert outputs are more compressible than dispatch activations due to lower-dimensional structure
  3. Smaller strides with narrower codes outperform larger strides with wider codes at comparable RDMA savings
  4. Efficient configuration search: Dispatch and combine performance drops are approximately additive, enabling compute-efficient search (94% of selections within 0.5 pp of best tested)

Theoretical and Practical Implications

Theoretical Contributions

  • Establishes cross-layer predictability as a practical mechanism for reducing EP communication in existing MoE models
  • Quantifies the redundancy in dispatched activations across MoE layers, showing that successive full-width activations are highly correlated
  • Demonstrates that expert outputs have lower-dimensional structure amenable to compression

Practical Implications

  • Retrofitting existing models: EchoMoE requires no architectural changes before pretraining and can be applied to off-the-shelf pretrained MoE models
  • Complementary to existing optimizations: Works with DeepEP, FP8 quantization, and other transport-level optimizations
  • Scalable to larger architectures: Throughput improvements of 1.24× on Qwen3-235B-A22B, 1.15× on DeepSeek-V3, and 1.09× on Kimi-K2.6
  • Wide-area inference: Enables efficient inference over bandwidth-constrained networks (2.38× over TCP)
  • Open-source release: Efficient implementation for vLLM integration, code, and models publicly available

Conclusion

EchoMoE introduces a lightweight architectural retrofit that reduces expert-parallel communication in existing MoE models by exploiting cross-layer predictability of activations. By periodically caching full-width reference states and communicating only compressed differences, EchoMoE achieves a 58% reduction in inter-node RDMA traffic with minimal task performance degradation (3.8 pp) and improves inference throughput by 22.6%, all while requiring approximately 8,500× fewer training tokens than pretraining.

Future directions identified by the authors include:

  • Incorporating EchoMoE as a novel architecture for pretraining, where reducing representational redundancy may enable stronger performance
  • Extending communication savings to the training setting
  • Further exploration of the configuration space for different model architectures and deployment scenarios

Related papers