HKUST and Huawei propose BalanceRoute to cut MoE LLM serving load-balancing bottleneck
Researchers from HKUST and Huawei's 2012 Lab propose BalanceRoute, an online routing method that models synchronous waiting in MoE LLM serving and converts short-term load prediction into routing decisions. In a 96-GPU test, the predictor-based variant BR-H improves decode throughput by about 15% over the strongest baseline. The work is accepted by NeurIPS 2026 and deployed in the jointly built MaaS system MathBot-Pie.
Coverage timeline
机器之心机器之心
大模型走向大规模在线服务，一个常见却又有些反直觉的现象是： 卡加得越来越多，吞吐却涨得越来越慢 。 问题出在哪？在 DeepSeek 等 MoE 模型的一类部署中，多组计算卡并行处理大量请求，每组既处理各自的请求，又要通过跨组通信，共同完成每一步解码。这意味着解码是齐步走： 只要最忙的一组没算完，其他组算得再快也只能等 。 更麻烦的是，请求即使分得一样多，各组实际承担的负载也可能不同，于是较轻的一组提前算完、空等最忙的一组，就成了常态。这种等待会随 token 生成反复发生，每一步都等一点，累积下来，部署规模扩大的收益便被悄悄吃掉了。 所以，瓶颈并不卡在算力那里，而是卡在「谁来等谁」上。 针对这一瓶颈， 香港科技大学周子杰团队与华为 2012 实验室理论部联合提出 BalanceRoute 。团队的思路很直接： 先把问题说清楚，再谈优化，即通过数学建模刻画同步等待，再将短期负载预测转化为在线路由决策。 效果也经得起检验。在论文的 96 卡主实验中，采用实际预测器的 BR-H 相对所比较的最强基线， 解码吞吐提升约 15% 。 这项工作已被 NeurIPS 2026 接收，并 已部署于双方共建的 MaaS 推理服务系统 MathBot-Pie 。 论文标题：Tackling the Data-Parallel Load Balancing Bottleneck in LLM Serving: Practical Online Routing at Scale 论文链接：https://arxiv.org/pdf/2605.06113 代码链接：https://github.com/b010001y/BalanceRoute 请求数相同，负载仍会不同 大模型生成回答，通常分为预填充（prefill）和解码（decode）两个阶段。预填充先处理整段输入，把后续生成需要复用的中间结果保存为 KV 缓存；解码再逐个生成输出 token。每生成一步，都需要读取此前输入和已生成内容对应的 KV 信息。 在本文研究的预填充 — 解码分离（PD 分离）系统中，这两个阶段由不同的计算资源承担。请求完成预填充后，路由器为它选择一个解码工作单元，已有的 KV 缓存也随之传送过去。一个单元可由多张计算卡组成。 轮询等方法可以依次把新请求分给各组，却没有区分每条请求需要处理多少 KV 信息。同
Related stories
Tsinghua and Qujing Introduce KTransformers-FineTune for CPU-GPU MoE Fine-Tuning
Tsinghua University's MADSys lab and Qujing Technology, with LlamaFactory, introduced KTransformers-FineTune (KT-FT), a CPU-GPU co-computation method for fine-tuning mixture-of-experts (MoE) models. It claims 2.5-21.2x throughput gains over traditional offloading by having the CPU compute expert weights directly instead of transferring them to the GPU. The approach includes a semantic-aware proactive layout engine and dynamic thread scheduling for routing skew, and is fully open-sourced.
1 reports · 1 sources
Tencent Hunyuan analyzes batch size impact on scaling RL training at 10K-100K GPU clusters
Tencent Hunyuan's team published a technical analysis on scaling reinforcement learning (RL) training for clusters of 10,000 to 100,000 GPUs, focusing on batch size tuning and scaling. They found that moderately increasing batch size with adjusted learning rates improves throughput, but at 16B batch size the additional sample cost outweighs the gains, slowing time-to-target.
2 reports · 2 sources
Tsinghua, Infinigence-AI, SJTU Open-Source On-Device Inference Engine APXInf for Embodied Models
On September 15, Tsinghua University, Infinigence-AI, and Shanghai Jiao Tong University open-sourced APXInf, an on-device inference engine for embodied models. The engine reportedly achieves a 10.7x end-to-end speedup on robot hardware, reducing inference latency from 278ms to 26ms on Thor chips with FP8 configuration, enabling real-time control at 38.46Hz.
Huawei, CUHK, HKUST release LegoFlow for automated code data engineering
Researchers from Huawei, CUHK, and HKUST introduced LegoFlow, an automated framework that streamlines code data engineering from repository collection to training and evaluation. They released LegoFlow-SWE with 5,000 tasks and 2,780 verified trajectories, and demonstrated recursive self-improvement: two iterations raised Qwen3.5-35B-A3B-Base's SWE-bench Verified score from 7.6% to 64.4%.
News updates every hour. Subscribe in your RSS reader to get new stories automatically.
Subscribe