v0.28.0

✨ New Features

• Kimi-K3 performance optimizations including Decode Context Parallel (DCP) support, fused FlashKDA decode and prefill kernels, SiTU activation support for MegaMoE, GEMM-RS for sequence parallelism, combined all-gathers with significant kernel-level speedup, adaptive speculative token budget for improved DSpark TTFT, and optional shared-expert sharding for memory savings.
• Kimi-K3 now runs on ROCm with the V2 model runner.
• DeepSeek V4 support with sparse MLA for plain decode, MTP, and DSpark speculative decoding.
• AMD Quark NVFP4 support for DeepSeek V4.
• Reasoning-effort prompts and mappings for DeepSeek V4.
• Sparse top-k metadata kernel optimizations for DeepSeek V4.
• Narrowed eager CUDA graph regions for DeepSeek V4.
• ROCm enablement for DeepSeek V4 on gfx11 and gfx950.
• Speculative decoding enhancements including DFlash2 with local convolution and a candidate selector, DSpark confidence-scheduled verification, and asynchronous scheduling auto-enabled for draft models.
• Model Runner V2 maturation with E/P/D disaggregation, weight offloading, multi-layer MTP KV cache support, encoder CUDA graphs, decoder token-wise pooling, Transformers pooling models, attention-free models, and `thinking_token_budget` support.
• Tiered KV cache offloading with disk offloading support, out-of-tree secondary tier managers via `module_path`, partial secondary-tier load results, tiering metrics, and a canonical CPU layout for parallelism-agnostic offload.
• Rust frontend and gRPC capabilities including a standalone renderer, multimodal image inference over gRPC, explicit data-parallel rank routing, RL lifecycle control, and published protobuf schemas to Buf.
• New defaults: `max_num_batched_tokens` raised to 16384, prefix caching enabled by default for Mamba models, and Blackwell CUDA graph capture default raised to 1024.
• New models supported: Muse Glimmer, Ling 3.0 Flash (BF16, MTP, parser, FP8 variant, hybrid MXFP4), Dots3 NOTE (native multimodal), and Interns2mobius.
• Qwen model improvements including Qwen3.8 on AMD ROCm, fused CUDA post-conv MTP decode kernel for Qwen3.5 GDN, GDN gates aligned with speculative tokens, and fixes for Qwen3.5 text-only checkpoints.
• Transformers modeling backend enhancements: MLA support, hardware-agnostic model definition, generalized input embedding handling, logit softcapping, and a hardened multimodal path.
• LoRA support for vision towers (Gemma4) and tower/connector (Keye, Ultravox).
• Vision encoder improvements: ViT full CUDA graph for Kimi-K2.5 and Ernie-4.5-VL, `torch.compile` for Qwen3-VL encoder, and avoidance of long-blocking H2D copies in ViT.
• MoE enhancements: extended EPLB support for Mistral Large 3 and additional MoE backends, CuTe DSL skinny GEMM extended to GLM-5.2.
• Speculative decoding coverage expanded for EAGLE3 on KimiLinear and Qwen3.6 dSpark.
• Correctness fixes for MiniMax-M3 NVFP4 inference, compressed-tensors FP8 MoE SwiGLU params, Gemma3n/Gemma4 variable-length audio batch padding, Gemma 4 compatibility with upcoming Transformers, and Qwen3-Omni crash on video without audio.
• Multimodal performance improvements: fused on-device multimodal preprocess normalization, faster placeholder and token-match scanning, and avoidance of repeated prompt-update scans.
• Engine Core speculative decoding: DSpark confidence-scheduled verification, top-k DSpark Markov projection, DFlash2 with local convolution and candidate selector, async scheduling auto-enabled for draft models, fused MTP trailing all-reduce with local-argmax draft tokens, and adaptive budget for speculative scheduled input tokens.
• Engine Core KV cache & scheduling: per-request scheduling for MLA chunked context, partial-tail prefix reuse with fine-grained prefix matching, backend-published KV packing in KV-cache layout refactor, LIFO free-block reuse order restored when prefix caching is off, and silent request skipping in priority scheduling fixed.
• Engine Core performance: continued elimination of GPU<->CPU syncs guarded by CI sync check, new JIT warmup infrastructure with predicate filtering, top-k/top-p Triton sampler launched with 8 warps, detokenization skipped in offline beam search, Mask Replay, optimized long-context MLA cache gathers, and HF revisions resolved to a commit hash once per model load.
• Engine Core Hybrid/Mamba: prefix caching on by default, final part of Mamba attention module refactor, 3D-grid tiling of state-copy Triton kernels, and Mamba alignment applied before encoder caps.
• Engine Core RL workflows: stateful trainer send over NCCL and sparse NCCL, `CuMemAllocator.discard()` for tag-selective GPU memory release, level-2 sleep/wake/reload fixed with LoRA enabled, and rewritten weight-transfer docs with standardized examples.
• Engine Core startup robustness: file:// rendezvous for single-node executors eliminates startup port races, frontend processes watched during engine startup, `get_open_port()` livelock on DP-reserved ports fixed, and NVML no longer re-initialized on every device-capability check.
• Hardware & Performance: FlashInfer XQA decode support on SM12x, CuTeDSL fused query kernel on SM100, programmatic dependent launch for DSA decode kernels, native DSA decode path for MTP=3 on SM90.