vLLM-Omni Adds Distributed Layerwise Offload for 200B+ DiT Models

last week#vllm#offload#dit#distributed

The vLLM blog announced distributed layerwise offload in vLLM-Omni, enabling efficient scaling of 200B+ parameter diffusion transformer (DiT) models across NPUs or GPUs. The feature includes flags to control weight sharding via AllGather, with usage examples for Cosmos3-Nano and Cosmos3-Super (64B parameters, 124 GB).

Coverage timeline

vLLM BlogvLLM-Omni Diffusion Teamlast week
Distributed Layerwise Offload: Scaling Toward 200B+ DiT Models Efficiently in vLLM-Omni
# 4× NPU or GPU — Cosmos3-Nano with DP=4 vllm serve /path/to/Cosmos3-Nano --omni \ --enable-distributed-layerwise-offload \ --data-parallel-size 4 # 2× devices — Cosmos3-Super (124 GB) with DP=2 vllm serve /path/to/Cosmos3-Super --omni \ --enable-distributed-layerwise-offload \ --data-parallel-size 2 # Disable AllGather (each rank loads full weights, no sharding) vllm serve /path/to/Cosmos3-Nano --omni \ --enable-distributed-layerwise-offload \ --data-parallel-size 4 \ --dlo-no-use-allgather ``` The `--dlo-use-allgather` / `--dlo-no-use-allgather` flag controls whether weights are sharded (default: sharded). When disabled, each rank loads the standard loader's rank-local tensors — in pure-DP configurations this is a full model copy, while existing TP shards are already rank-local and are reused as-is. This mode is useful when AllGather synchronization overhead outweighs the memory savings. ## The Problem: Large Diffusion Models vs. HBM and Host Memory Cosmos3-Super (64B parameters, 124