Back to News

NVIDIA SEED cuts prefill compute, boosts per-GPU throughput up to 42%

#nvidia#llm-inference#prefill#throughput

NVIDIA researchers propose SEED, a technique that reduces prefill computation by running only the first 31 of 48 layers for non-final tokens and compressing recurrent state via low-rank decomposition. On GB300 GPUs with SGLang and coding-agent traces, SEED improves per-GPU peak throughput by 16–42% for Qwen3.8-Flash-Next, with 23–42% gains at matched P90 end-to-end output rates. The paper, titled 'Shallow Encoding Emits Deep Memories', is available as an arXiv preprint.

Coverage timeline

  1. 机器之心机器之心

    Qwen3.8-Flash-Next 有 48 层。处理一段输入时,SEED 让除最后一个以外的 token 只运行前 31 层,再由轻量的记忆写入组件,为后续 17 层生成 KV Cache 和循环状态。到了生成阶段,每个 token 仍然运行完整的 48 层。 这个改动利用了一个区别: 为后续生成写下记忆,所需的深层计算可以远少于预测下一个 token 。SEED 在五种已训练的混合大模型上测量了这种差别,并把它用于减少预填充计算。 解码侧还有另一处空间。模型每步读写的循环状态,往往包含一个接近固定方向的主分量,剩余内容则具有低秩结构。SEED 将状态保存在这种分解形式下,让模型直接读取和更新它。在 Qwen3.8-Flash-Next 上,单请求循环状态从 113.2MB 降到刚压缩后的 29.2MB,整个压缩周期平均为 43.4MB。 在 GB300 GPU、SGLang 和编码 Agent 会话轨迹上,SEED 使 Qwen3.8-Flash-Next 的 每 GPU 峰值吞吐提升 16%~42%;在相同 P90 用户端到端输出速率的比较点,吞吐提升 23%~42% 。五种模型、六个检查点的质量评测则检验了这些改动的精度代价。原模型权重全程冻结,新增组件通过蒸馏训练。 论文《Shallow Encoding Emits Deep Memories》已发布 arXiv 预印本。Mingyuan Ma 与 Yangmin Li 共同主导、同等贡献;其余作者为 Harris Nover、Zheyu Fu、Po-Han Huang、Alex Settle 和 Ashwin Nanjappa,作者均来自 NVIDIA。 论文链接:https://arxiv.org/abs/2611.06084v1 项目链接:https://latent-voyager-seed.static.hf.space/index.html 固定大小的状态仍然可能很贵 混合架构用少量 softmax 注意力层保存 KV Cache,再用大量线性注意力或状态空间层维护循环状态。KV Cache 随上下文增长,循环状态大小固定,但后者并不小:Qwen3.8-Flash-Next 每个请求的状态,占用空间相当于约 4500 个 token 的注意力缓存。 这些状态每生成一个 token 就要更新一

Related stories

  • Open-source projects unlock hidden Tensor Core throughput on GeForce GPUs, doubling RTX 4090 performance

    Open-source developers released projects on GitHub that unlock higher Tensor Core throughput on NVIDIA GeForce GPUs by modifying boot-time configuration, without driver swaps or BIOS flashes. Tests reportedly doubled the FP16/FP32 and BF16/FP32 throughput of an RTX 4090 from about 165 TFLOPS to over 330 TFLOPS, with gains also observed on RTX 3090 Ti, RTX 4070 Ti SUPER, and RTX 2080 Ti. The projects target RTX 20, 30, and 40 series, with some experiments done on Windows 11.

    1 reports · 1 sources
  • FlashStates Hybrid Cache Management Boosts Prefill Throughput Up to 3.14x vs vLLM

    Researchers from Together AI, UIUC, and UT Austin propose FlashStates, a hybrid state cache management system for mixed-architecture LLMs that combine full attention with linear layers. On NVIDIA B200 GPUs, replaying real Claude Code traces, FlashStates achieves up to 3.14x prefill throughput over vLLM and reduces p90 time-to-first-token (TTFT) by up to 88.4%, with benefits persisting when integrated with a 512GiB Mooncake CPU cache.

    1 reports · 1 sources
  • Meta and academic researchers propose World-T³, a video world model with test-time training for long-term memory

    Researchers from Meta Reality Labs, the University of Illinois Chicago, and Brown University propose World-T³, a camera-controllable video world model that uses test-time training to store historical observations in fast weights, keyed by camera pose, enabling consistent scene recall when revisiting locations. On a long-term revisit benchmark of 120 trajectories, it improves revisit PSNR from 13.52 dB to 18.31 dB and reduces absolute rotation error from 3.344° to 1.075°, while a 4-step distilled model runs at 20.6 FPS on a single NVIDIA B200 GPU at 1280×704 resolution.

    1 reports · 1 sources
  • OpenAI says models improved AMD GPU IP by up to 14% in PPA

    OpenAI reported that its models improved power, performance, and area (PPA) by up to 14% on AMD's flagship GPU IP, with some changes included in a design AMD sent for manufacturing. The company also used its models to optimize circuits in Jalapeño silicon and improve RISC-V core designs, showcasing progress in AI-assisted chip design with GPT-6 Astra.

    2 reports · 2 sources221 points · 169 comments

News updates every hour. Subscribe in your RSS reader to get new stories automatically.

Subscribe