v4.8.0

✨ New Features

• Introduced vllm.cpp, a new C++20 engine for LocalAI, as an alpha development build. It offers a V1 serving architecture with paged KV cache, continuous batching, prefix caching, scheduler, and sampler, without Python, PyTorch, or ggml at inference. It supports safetensors and GGUF, enforces structured output, and runs on CPU, CUDA, Metal, and Vulkan. The Apple Silicon build includes the MLX GEMM provider.
• Added 3D generation as a new modality, including the Generate3D RPC, FLAG_3D capability, POST /v1/3d/generations endpoint, and the trellis2cpp image-to-3D backend. A UI page with a native GLB viewer and print remeshing is also included.
• Introduced audio.cpp, a new backend process serving six audio endpoints: speech, transcription, VAD, diarization, source separation, and sound generation. Model families are determined from GGUF's metadata.
• Gallery entries can now declare 'variants:' to allow LocalAI to install the largest build compatible with the host hardware, based on VRAM or RAM constraints. This simplifies model selection.
• Significantly improved web UI performance with gzip compression, immutable caching for hashed assets, and paginated trace endpoints, resulting in a 3.48x smaller React bundle and a drastically reduced trace poll size.
• Added an Activity page to track installed, failed, or cancelled operations, providing a persistent record instead of temporary display.
• Implemented Hugging Face artifact materialization with immutable snapshot resolution, authenticated downloads with progress reporting, and staged artifacts for remote workers.
• Introduced VRAM budgets to cap GPU usage per node, configurable as a percentage or absolute amount.
• Added two new TTS engines: magpie-tts-cpp (NVIDIA Magpie Multilingual) and moss-tts-cpp (48 kHz stereo with reference-audio voice cloning).
• Added a new 'bonsai' backend to serve sub-2-bit models, specifically the 1-bit and ternary Bonsai quantizations of Qwen3 and Qwen3.6-27B.
• Hardened distributed mode with improvements to the reaper, cleanup of phantom replicas, and prevention of 'in_flight' counter leaks.
• Added a Valkey vector store option.
• Implemented systemd socket activation.
• Enabled persistent trace history.
• Released a new localai.io website with a project site and documentation under /docs/, including an engines page, blog, and ecosystem band.
• Added support for chat and tool calling via the llama.cpp autoparser path in vllm.cpp, including minja templates, 'tool_choice: auto', 30 tool dialects, 7 reasoning parsers, and streamed ChatDelta/ToolCallDelta.