Back to News

vLLM Adds Day-Zero Support for Mistral Large 4 on ROCm with TP8/TP4

#vllm#mistral-large-4#rocm#amd

vLLM announced day-zero support for serving Mistral Large 4 (mistralai/Mistral-Large-4-1T-A52B) on AMD ROCm, with configurations for tensor-parallel size 8 (one replica per node) and tensor-parallel size 4 (two replicas per node). The setup uses AITER optimizations, FP8 MoE requantization (mxfp4), speculative decoding with dspark, and tool-call/reasoning parsers.

Coverage timeline

  1. vLLM BlogMistral and the vLLM Team
    Day 0 support for Mistral Large 4 on vLLM

    # TP8, one replica per node VLLM_ROCM_USE_AITER=1 VLLM_FP8_MOE_REQUANT=mxfp4 VLLM_FP8_MOE_REQUANT_ACT=a8w4 \ VLLM_ROCM_USE_AITER_TRITON_SPARSE_MLA=1 AITER_CUSTOM_AR_MAX_SIZE=134217728 \ vllm serve mistralai/Mistral-Large-4-1T-A52B --tensor-parallel-size 8 \ --speculative-config '{"method": "dspark", "num_speculative_tokens": 3}' \ --tool-call-parser mistral --enable-auto-tool-choice --reasoning-parser mistral # TP4 with the specialized decode, for few concurrent users; two replicas per node (second on GPUs 4-7) VLLM_ROCM_USE_AITER=1 VLLM_FP8_MOE_REQUANT=mxfp4 VLLM_FP8_MOE_REQUANT_ACT=a4w4 \ VLLM_ROCM_USE_AITER_TRITON_SPARSE_MLA=1 AITER_CUSTOM_AR_MAX_SIZE=134217728 \ vllm serve mistralai/Mistral-Large-4-1T-A52B --tensor-parallel-size 4 \ --additional-config '{"decode_backend": "latency_specialized"}' --max-num-seqs 64 \ --speculative-config '{"method": "dspark", "num_speculative_tokens": 3}' \ --tool-call-parser mistral --enable-auto-tool-choice --reasoning-parser mistral ``` ### ROCm i

Related stories

  • vLLM Adds Day-0 Support for Flint Sparse Model Checkpoints

    vLLM announced day-0 support for Flint checkpoints, a sparse model with 40 billion total parameters and 2.5 billion active parameters. On a single GPU, vLLM decodes a single request at approximately 500 tokens per second using these checkpoints.

    1 reports · 1 sources

  • Mistral Large 4 preview: 1T-parameter open-weight model, weights due Oct 27

    Mistral AI released a public preview of Mistral Large 4 (ML4), a 1-trillion-parameter multimodal model with 49 billion active parameters, trained on 3,800 Nvidia Grace Blackwell GPUs in its European data centers. The open weights are promised by October 27. Early benchmarks show an Artificial Analysis Intelligence Index score of 38, comparable to DeepSeek V4.1 Flash, and the model is claimed to top any open model from the US or Europe.

    10 reports · 5 sources2033 points · 1210 comments

  • vLLM Adds Day-0 Support for NVIDIA Vera Rubin NVL72, Claims 7.8x Throughput over GB200

    vLLM announced support for NVIDIA Vera Rubin NVL72, claiming 7.8x throughput over GB200 NVL72. The platform offers daily container builds and day-0 model support for DeepSeek, Moonshot AI, Z.ai, and MiniMax, leveraging Blackwell kernel compatibility and Rubin-tuned kernels via FlashInfer 0.7.0.

    1 reports · 1 sources

  • NVIDIA launches Nemotron 3.5 Super VL, 120B open model; vLLM adds Day 0 support

    NVIDIA has launched Nemotron 3.5 Super VL, a 120B-parameter open model for enterprise agents, claiming it outperforms frontier models on specialized benchmarks. vLLM announced Day 0 support for the model, enabling multimodal agent workflows with image, document, and video understanding, OpenAI-compatible serving, and multi-token prediction.

    2 reports · 2 sources

News updates every hour. Subscribe in your RSS reader to get new stories automatically.

Subscribe