v0.27.0

✨ New Features

• Kimi K3 support including core model files, kernels, Python and Rust frontends, AttnRes kernels, DeepGEMM support, compressed-tensors quantized checkpoints, DSpark AR fusion, and an option to shard the shared expert instead of replicating it.
• Support for new models: Qwen3.5 text-only dense and MoE models with EVS video token pruning, K-EXAONE-2.0-750B-A37B, VaultGemma via the Transformers modeling backend, and jina-embeddings-v5-text-nano.
• PyTorch 2.13.0 upgrade along with torchvision 0.28.0 and Triton 3.7.1.
• FlashAttention 4 integration deepens on SM100 with FP8 KV cache support and headdim-256 support, backed by new JIT warmup infrastructure and runner-owned Triton kernel warmup to remove first-request compilation stalls.
• DeepSeek-V4 performance improvements including sequence parallelism, kernel improvements by skipping empty c128 launches, E2E TTFT improvements from skipping unneeded topk/router and workspace reuse, kernel improvements from removing redundant full kernels, adaptive topk width, GPU memory savings in the PP buffer, a compact MXFP4 indexer KV cache, and removal of sparse-MLA q-head padding on FlashInfer >= 0.6.14.
• Model Runner V2 expands to non-generative workloads: encoder-only attention, sequence pooling for embedding/classification, encoder token classification, token embedding, BGE-M3 pooling, multimodal on CPU, a multi-layer MTP speculator, and PCP now selects MRV2.
• Resilient large-scale serving with a fault tolerance framework for DP+EP external load-balancer deployments and async preparation for elastic EP scaling.
• Disaggregation for hybrid models: NIXL P/D for hybrid MLA+SSM models, heterogeneous P/D block sizes for hybrid models, and MoRIIO heterogeneous TP<->DP prefill/decode read routing.
• Rust frontend grows a gRPC control plane with engine-aware health reporting, abort control, server and model discovery, KV event source discovery, and `vllm-bench` integrated into the `vllm` CLI.
• Early next-gen hardware enablement: `sm_107` target for NVIDIA Rubin with NVLink all-reduce paths on SM107, and ROCm gfx1250 architecture enabled.
• Inkling support for llm-compressor NVFP4 weights and compressed-tensors dynamic FP8.
• Multimodal support enhancements: VidCom2 video token pruning, EVS for Qwen3.5, ViT CUDA graph for Gemma-4, Cosmos3 FP8 ModelOpt/Diffusers remapping, MiniMax-M3 MSA speculative decode verification and default video processor, DeepSeek-OCR-2 TTFT optimization, and longer max audio duration for MOSS-TD.
• Diffusion models support for top_k and top_p sampling for DiffusionGemma.
• Transformers modeling backend improvements: audio model support, improved `fx` tracer, fused residual-add + RMSNorm compilation pass, and fixes for MLA padding + grouped topk routing, MQA with TP, and Qwen3-VL M-RoPE.
• New JIT warmup infrastructure and runner-owned Triton kernels warmed before the first request, and proper renderer warmup.
• FlashAttention 4 SM100 FP8 KV cache and headdim-256 support; query replication for MLA decode under DCP for DeepSeek-V2/R1 and Kimi-K2.5; masked MHA for sparse MLA prefills; skip sparse indexer scoring for short dense prefills; FlexAttention epilogue hook and encoder block-mask compile explosion avoided; attention backends stay eligible for text-only serving of prefix-LM models; merge-attention context count as a runtime argument; unified multi-path encoder CUDA graph support; encoder cache extension hooks.
• Model Runner V2 enhancements: attention metadata always built at capture time, encoder cache profiling, skipped no-op FP32 logits materialization, chunked rejection sampler to avoid OOM, and fewer GPU<->CPU syncs in hybrid Mamba.
• KV offloading enhancements: generic P2P secondary tier with peer lookup/serving, per-request tier filtering with TierFilter/TierMatcher, self-describing KV events with TieringOffloadingSpec, pluggable eviction policies via CachePolicyFactory, deduplicated replicated MLA KV in the shared CPU region, single-copy MLA layout for CPUOffloadingSpec, TP-independent compact secondary identity, batched C store/load for filesystem offload, reliable partial-tail offload for sub-block prompts, and per-layer canonical KV page mappings for parallelism-agnostic offload.
• Mamba/hybrid improvements: ReplaySSM caching for faster Mamba2 standard decode, fused align-mode DS-conv state migration with num_accepted_tokens > 1, FlashInfer Mamba SSU algorithm selection.
• Spec decode enhancements: DSpark Markov head replicated across TP ranks, `sample_from_anchor` loaded from speculators config, and earliest-completing stop string selected.
• Structured outputs grammar advanced across the reasoning boundary with spec decode.
• Preprocessing performance improvements: derender CPU work offloaded to the renderer thread pool, raw-prompt preprocessing off the event loop in AsyncLLM, multimodal preprocessing isolated on its own executor, MM embeds loading deferred off the event loop, parallel preprocessing within a request for online pooling, videos hashed by source bytes, and original image mode preserved in ImageIO.
• RL enhancements: weight version tagging for RL rollouts, stateful trainer-send IPC, vLLM config set during weight reload, and router replay output from the FlashInfer monolithic MoE kernel.
• Memory & robustness improvements: CuMem slept-L1 fragmentation accounting, cgroup memory limits respected on all platforms, fail fast when /dev/shm is too small, zero-copy tensor pickling in shm_broadcast, LRU hash-split skipped in free_blocks when prefix caching is off, location-derived path vars excluded from torch.compile cache factors, `CustomOp.forward_native` compiled for ReLU^2, HF config used for HF tokenizers, and batch-invariant RMSNorm via pinned block size.