v0.16.0

✨ New Features

• Full support for Async scheduling combined with Pipeline Parallelism, yielding significant throughput improvements.
• Introduction of a new WebSocket-based Realtime API for streaming audio interactions.
• Native NCCL-based weight syncing API for RLHF workflows.
• Layerwise weight reloading support for QeRL.
• Engine pause/resume functionality with request preservation.
• Unified Parallel Drafting implementation for speculative decoding.
• Speculative decoding now supports structured outputs.
• Support for penalty application in Model Runner V2 for speculative decoding.
• XPU platform overhaul including support for MoE, MXFP4 MoE, WNA16, scaled_mm, and FP8 MoE using vllm-xpu-kernels.
• Support for new model architectures including GLM-OCR with MTP, Qwen3-ASR, DeepSeek-OCR-2, Intern-S1-Pro, MiniCPM-o 4.5, openPangu7B-VL, NemotronHPuzzle heterogeneous, MusicFlamingo, FunAudioChat, ColBERT late interaction, voyage-4-nano, and GLM-5.
• Speculative decoding support added for EAGLE3 (Hunyuan/HunyuanVL), AFMoE, and Mistral3.
• LoRA expansion for Gemma3 vision components, Nemotron-H MTP models, and Qwen3 output embedding, along with optimized fused MoE-LoRA kernels.
• Qwen3-Omni transcription support.
• Mistral Large 3 support with FlashInfer MoE.
• LFM2 SigLIP2 intermediate encoder layer support.
• Support for embedding input for disabled modalities.
• FlashInfer TRTLLM BF16 MoE integration for NVIDIA.
• SM100 INT4 W4A16 kernel support.
• SM121 (DGX Spark) CUTLASS support.
• MNNVL protocol support for GB series.
• FlashInfer MLA concat optimization.
• GDN attention layout optimization.
• DeepGEMM FP8 MLA performance improvements.
• wvSplitK_fp8 performance improvements.
• B200 MoE configs for Nemotron Nano and Super B200 TP2 support.
• Mamba selective scan tuning for B200.
• QWEN3-NEXT FP8 tunings and AITER attention backend for AMD ROCm.
• Fused_add_rmsnorm_pad for GPT-OSS on AMD.
• ARM CPU support for KleidiAI INT4 dynamic quant with BF16 activations, NEON BFMMLA BF16 paged attention, and vectorization backend optimization.
• BF16 kernel type support for IBM Z (s390x).
• torch.compile improvements including stopping compilation of identical artifacts and MoE cold start optimization.
• Chat completion streaming optimization.
• ORJSONResponse for faster API responses.
• MoE permute optimization for CUTLASS FP8.
• Shared/routed overlap optimization for latent MoE on Nemotron-H.
• FlashInfer autotune control flag.
• Disaggregated serving improvements including Mooncake connector rework and cross-layer KV cache layout.
• EPLB improvements including capturing logical experts with router replay.
• New quantization methods: FP8 block quant for CompressedTensorsW8A16Fp8, ModelOpt MXFP8 for dense models, NVFP4/FP8 on Turing GPUs, and TP > 4 for FP4 Gemm.
• WebSocket-based streaming API implementation.
• Responses API enhancements to return sampling parameters, token IDs, and prompt token IDs.
• Pooling API standardization.
• Tool calling fixes and GLM-4 incremental string streaming.
• Structured outputs performance optimization with reasoning.
• CLI option `--disable-access-log-for-endpoints`.
• UX improvements: Nested configs in YAML, GGUF repo_id:quant_type syntax, default DeepSeek ReasoningParser thinking, removal of noisy CT warning, early tokenization validation, and reasoning_content backward compatibility.
• run_batch transcription/translation support.
• /server_info collect_env endpoint.
• OTEL tracing during model loading.
• Ability to clear MM and encoder cache.
• HF Hub LoRA resolver.