Nine Chapter Cloud's Alaya-DFlash2 speeds up GLM-5.3 decoding by 3.41x
Nine Chapter Cloud introduced Alaya-DFlash2, a speculative decoding solution for GLM-5.3 that combines a parallel draft model with a path selection mechanism to maintain high acceptance length in long contexts. Benchmarks on 8 H200 GPUs show a 3.41x single-user decoding speedup (288.2 vs 84.5 tokens/s) and up to 2.24x total output throughput at 4 concurrent requests, with a 44.9% throughput gain at 64 concurrency.
Coverage timeline
机器之心新闻资讯
长文本的等待，发生在输出阶段。 32k 的文档读进去了，模型开始回答。第一个字很快，后面每个字都要等——主模型一次只往前一步，每生成一个 token，都要把整段上下文重新算一遍。上下文越长，这一步越贵。 推测解码换了个做法：让一个更小的草稿模型先往后猜几步，主模型一次性验证。猜中的部分直接接收，一轮就能推进好几个 token。猜得准不准，用一个指标衡量——接受长度，即每次验证平均能通过多少个候选。 Alaya-DFlash2 是九章云极为 GLM-5.3 打造的一套推测解码加速方案：一个能并行起草的草稿模型，加一套路径选择机制，让它在长上下文里依然猜得准。评测显示，它在长上下文下保持了较高的草稿接受长度，并在从单并发到 64 并发的全部已测档位中，同时取得高于原生自回归解码的单用户解码速度与总输出吞吐。 一句话：交互场景可以追求更流畅的生成，多请求场景能用同样的卡承接更多输出。 从单人交互到并发服务，速度与吞吐双提升 图 1：横轴为单用户解码速度，纵轴为总输出吞吐，点旁数字表示并发数。在相同并发档位下，Alaya-DFlash2 的测量点均位于原生自回归AR的右上方。 TP=8, 8 × H200, 单并发时，Alaya-DFlash2 的解码速度约为 288.2 tokens/s ，原生自回归约为 84.5 tokens/s ，前者达到后者的 3.41 倍 。对应的总输出吞吐从78.95 tokens/s 提升至 236.47 tokens/s，约为原来的 3 倍 。对于需要持续输出较长内容的应用，这意味着生成阶段可以更快地向用户交付文本。 并发增加后，优势仍然存在。4 并发时，Alaya-DFlash2 的总输出吞吐达到 506.56 tokens/s ，为原生自回归的 2.24 倍 ；64 并发时，总输出吞吐达到 608.71 tokens/s ，较原生自回归的 420.13 tokens/s 提升 44.9% 。在本轮负载范围内，Alaya-DFlash2 为低并发交互和较高并发服务都提供了可量化的性能增益。 上下文拉长至 32k，接受长度保持领先 长上下文中的表现，是这组结果尤其值得关注的部分。草稿模型需要预测主模型接下来会生成什么；每轮能够被接受的内容越多，就越有机会减少主模型反复解码的轮次。“接受长度”正是衡量这项能力的指标。 在覆盖 1k
Related stories
vLLM Boosts DeepSeek-V4.1-Flash: 1.9x Low-Concurrency Speed, 5x Agentic Throughput
According to a vLLM blog post dated October 7, 2026, within three weeks of DeepSeek-V4.1-Flash's release, vLLM optimizations made the model 1.9x faster at low concurrency and increased its throughput 5x on the SemiAnalysis AgentX benchmark. The improvements are attributed to SWA bounded replay, CUDA graphs, DeepSeek's new kernels, and vLLM kernel fusions.
1 reports · 1 sources
Stanford, Together AI propose Next-Concept Modeling for parallel text generation, 10-31x speedup
Researchers from Stanford University and Together AI introduced Next-Concept Modeling (NCM), a coarse-to-fine parallel text generation paradigm adapted from next-scale prediction used in image generation. In experiments, NCM achieved 10-31x speedup over diffusion language models with comparable or better quality, offering an alternative to both autoregressive and diffusion approaches.
1 reports · 1 sources
Tsinghua, Infinigence-AI, SJTU Open-Source On-Device Inference Engine APXInf for Embodied Models
On September 15, Tsinghua University, Infinigence-AI, and Shanghai Jiao Tong University open-sourced APXInf, an on-device inference engine for embodied models. The engine reportedly achieves a 10.7x end-to-end speedup on robot hardware, reducing inference latency from 278ms to 26ms on Thor chips with FP8 configuration, enabling real-time control at 38.46Hz.
DeepSeek releases V4.1-Flash-1017 open-weight model with faster, cheaper agent tasks
DeepSeek released V4.1-Flash-1017, an open-weight model checkpoint with 763B parameters, claiming stronger benchmark performance, faster task completion, and shorter outputs than its predecessor. The model is optimized for agent frameworks like DeepSeek Harness, Tencent CodeBuddy, and WorkBuddy, and is now live on the website, app, and API. DeepSeek also opened its Search & Fetch APIs to everyone, priced at $0.75 per 1000 successful requests.
8 reports · 5 sources955 points · 717 comments
News updates every hour. Subscribe in your RSS reader to get new stories automatically.
Subscribe