llama.cpp

This release focuses heavily on optimizing Hexagon (HMX) performance by introducing asynchronous workers and queues to overlap computation stages, alongside fixing a race condition in the worker drain mechanism.

This release removes the deprecated ggml-ext.h file and corrects its placement, alongside providing numerous pre-compiled binaries for various operating systems and hardware configurations.

This release includes a fix for the FA support logic within the metal backend and provides updated binaries for various operating systems and hardware configurations.

This release introduces the new mtmd_image_tokens_get_decoder_pos() API and includes fixes for build issues and naming consistency.

This release focuses on Vulkan shader improvements by conditionally enabling RoundingModeRTE and refactors SPIRV-Headers fetching logic, resolving a build issue on Ubuntu.

This release re-enables macOS CI workflows and includes fixes for Vulkan compilation warnings, alongside providing numerous pre-built binaries for various platforms and hardware configurations.

This release introduces the XIELU unary operation support for the metal backend and provides updated pre-compiled binaries for macOS, Linux, Windows, and openEuler across various hardware and acceleration configurations.

This release updates the internal BoringSSL dependency to version 0.20260413.0 and provides a comprehensive set of pre-built binaries for macOS, Linux, Windows, and openEuler across various architectures and acceleration backends.

This release addresses a specific bug in the ggml library concerning ARM NEON nvfp4 dot product calculations on non-dotprod targets.

This release addresses a CMake warning on Windows/MSVC by adjusting policy settings in the build system. It also includes various pre-compiled binaries for multiple operating systems and hardware configurations.

This release focuses on updates to the ggml-webgpu backend, specifically improving matmul accumulation precision and fixing several related bugs across different platforms.

This release optimizes performance by conditionally creating the reasoning budget sampler, ensuring backend sampling remains enabled when no token budget is set. It also preserves sampler creation when grammar is lazy to maintain tool usage compatibility.

This release introduces NVFP4 support within the Vulkan backend for several core tensor operations. It also provides a comprehensive set of pre-compiled binaries for diverse operating systems and hardware accelerators.

The server component has been updated to support the OpenAI /v1/audio/transcriptions API, alongside various platform-specific binary releases and a fix for the default response_format value.

This release primarily addresses parsing edge cases for the common/gemma4 component and provides updated binary distributions for numerous operating systems and hardware configurations.

This release introduces dedicated support and an official template for the DeepSeek v3.2 model parser. It also provides numerous pre-compiled binaries for various operating systems and hardware configurations.

This release introduces Vulkan Flash Attention DP4A support for quantized KV caches using integer dot products and includes several fixes related to indexing and quantization checks in the Vulkan backend.

This release introduces download cancellation and temporary file cleanup features. It also provides updated pre-compiled binaries for various operating systems and hardware configurations.

This release introduces the exposure of build_info when operating in router mode. It also provides numerous pre-compiled binaries for macOS, iOS, Linux, Windows, and openEuler targeting various CPU/GPU backends.

This release limits DeviceSegmentedSort to immediate mode due to CUDA graph capture limitations, ensuring stability when using CUDA graphs, and includes performance comparisons between the two sorting methods.

This release updates the model processing for Gemma 4 audio by implementing causal attention. It also provides a comprehensive set of pre-compiled binaries for numerous platforms and hardware configurations.

This release removes an unnecessary conditional check related to debug mode and provides updated binary distributions for macOS, Linux, Windows, and openEuler targeting various CPU/GPU architectures.

This release disables Q1_0 in the SYCL backend and performs cleanup of unused variables within the backend code. It also provides updated binary distributions for macOS, Linux, Windows, and openEuler.

This release primarily addresses a stability issue in the mtmd module related to small image handling. Various pre-compiled binaries for different operating systems and hardware configurations are provided.

This release introduces comprehensive support for Qwen3 audio models (omni and ASR) and includes several internal fixes, notably removing the deepstack dependency for audio.

This release introduces support for the Gemma 4 audio conformer encoder, detailing its specific architecture and preprocessing steps. Several internal fixes were implemented related to tensor loading and mask matching.

This release provides updated pre-compiled binaries for macOS, Linux (including specialized builds for ROCm 7.2 and OpenVINO), and Windows across multiple hardware architectures and acceleration backends.

This release introduces comprehensive support for the MERaLiON-2 multimodal audio model, including its specific architecture components and supported tasks. It also includes minor cleanups in the MERaLiON adaptor comments.

This release introduces basic support for the q5_k quantization format on OpenCL, including necessary matrix operation implementations and associated unit test fixes. It also provides a wide array of pre-compiled binaries for diverse operating systems and hardware accelerators.

This release primarily addresses a bug related to data splitting for the Qwen 3 Next model. It also provides updated binary distributions for macOS, Linux, Windows, and openEuler platforms.

This release addresses missing cases for GGML_TYPE_Q1_0 within the ggml library and provides a comprehensive set of updated pre-compiled binaries for diverse operating systems and hardware configurations.

This release primarily focuses on distributing pre-built binaries for numerous platforms including macOS, Linux (with Vulkan, ROCm, OpenVINO support), and Windows (with CUDA, Vulkan, SYCL, HIP support). A minor change was made in CUDA to store node->src ne/nb for graph equality.

This release addresses a bug related to structured output generation when JSON schema $refs are utilized. It also provides numerous pre-compiled binaries for different platforms.

This release focuses heavily on expanding hardware and OS support, particularly adding Linux on Snapdragon support via the hexagon backend, and updating build configurations and documentation.

This release significantly improves hexagon performance through op request batching, buffer management rewrite, and explicit L2 cache control. It also removes the deprecated GGML_HEXAGON_EXPERIMENTAL environment variable.

This release updates common components to align with the official gemma4 template and provides numerous pre-compiled binaries for macOS, Linux, Windows, and openEuler across various CPU/GPU architectures.

This release introduces an update to Gemma 4 model loading, making shared-KV tail attention tensors optional. It also provides a comprehensive set of pre-compiled binaries for diverse platforms and hardware configurations.

This release introduces a new callback interface for monitoring download progress and provides updated binary distributions for macOS, Linux, Windows, and openEuler across various hardware architectures and acceleration backends.

This release introduces support for non-square subgroup matrix configurations in ggml-webgpu for Intel GPUs and provides updated pre-compiled binaries across macOS, Linux, Windows, and openEuler platforms.

This release addresses numerous quantization precision issues within ggml WebGPU, especially concerning f16 stability and NaN handling. It also improves backend lifecycle management for WebGPU and cleans up deprecated code.

This release fixes an issue in llama-server where the --alias flag conflicted with model presets, and provides updated pre-compiled binaries for broad platform compatibility.

This release includes a fix for loading cached Hugging Face models when the API is unavailable and provides updated binary distributions for macOS, Linux, Windows, and openEuler.

This release introduces the experimental status for the --split-mode tensor flag and provides a comprehensive set of pre-built binaries for numerous platforms and hardware accelerators.

This release enables the reasoning budget sampler for gemma4 by updating parameter initialization and fixing a related parsing issue in the thought block handling.

This release introduces support for Q1_0 quantization format within the Vulkan backend and updates internal dependency usage by incorporating 'get_dm'.

This release introduces fluidity enhancements to the progress bar and provides updated pre-compiled binaries for macOS, Linux, Windows, and openEuler targeting various CPU/GPU backends.

This release introduces performance improvements on CUDA devices via kernel fusion for multiplications and provides updated binary distributions for numerous platforms and hardware configurations.

This release introduces support for the AMD CDNA4 architecture (gfx950) for MI350X/MI355X accelerators, adjusting matrix multiplication paths accordingly. Various pre-compiled binaries for different operating systems and hardware configurations are also provided.

This release introduces experimental backend-agnostic tensor parallelism in ggml, supporting models like GPT-OSS and Qwen 3 MoE across multiple GPUs. Numerous bug fixes address stability, quantization handling, and backend-specific issues across Vulkan, Metal, ROCm, and various model implementations.

This release addresses an ambiguous grammar rule in gemma4 and resolves a missing comma issue. It also provides extensive pre-compiled binaries for macOS, Linux, Windows, and openEuler platforms.

This release includes a stability fix in ggml by checking the return values of CUB calls used in argsort and top-k operations. It also provides numerous pre-compiled binaries for various operating systems and hardware configurations.

This release simplifies autoparser tagged parser rules and fixes several issues related to parameter initialization and optional argument ordering within the continuation logic. It also provides numerous pre-compiled binaries for diverse hardware and operating systems.

This release primarily addresses a bug by fixing the multimodal padding token for gemma3n/gemma4 models and includes minor nits.

This release introduces new tokenizer tests for Gemma 4, fixes an associated edge case, and includes minor internal code cleanup.

This release introduces support for dots in model names within the mtmd feature and adds GGUF conversion capabilities, alongside various platform-specific binary updates.

This release enhances the Jinja engine with Python-style string repetition, improved ASCII handling in tojson, and identity operations for int/float types. A bug related to invalid UTF-8 byte escaping during JSON serialization was also fixed.

This release introduces missing mm-id specializations for q1_0 within the metal backend and provides updated pre-compiled binaries for macOS, Linux, Windows, and openEuler platforms.

This release includes a fix for grammar commandline arguments in the server component and provides updated binary distributions for macOS, Linux, Windows, and openEuler targeting various hardware architectures and accelerators.

This release enhances SYCL performance by adding Flash Attention support for head size 512 and cleans up backend initialization logic. It also removes defunct mxfp4 reordering logic.

This release unifies Vulkan type macros to use Vx instead of the older _VECx convention. It also provides extensive pre-built binaries for diverse hardware and operating system configurations.

This release addresses an issue where GGUF split files were incorrectly ordered during model selection. It ensures robustness by skipping non-primary split files regardless of their listing order.

This release includes an internal fix for CUDA equality checks by storing source data pointers and provides updated pre-compiled binaries across multiple platforms and accelerators.

This release addresses a significant memory leak in the optimization context freeing routine (ggml_opt_free) by ensuring the per-batch context copy (ctx_copy) is properly released.

This release introduces a server-side feature to respect the ignore eos flag and provides numerous pre-compiled binaries for diverse operating systems and hardware configurations.

This release updates binary builds for numerous platforms (macOS, Linux, Windows, openEuler) and implements a vocabulary change by removing the </s> eog token for Gemma 4 models.

This release primarily addresses minor documentation issues by fixing a couple of typos. It also provides updated binary distributions across various operating systems and hardware configurations.

This release enhances KV cache quantization checks to correctly account for enabled flash attention. It also provides numerous pre-compiled binaries for various operating systems and hardware configurations.

This release introduces adapter support querying for the WebGPU backend and provides updated binary distributions for numerous operating systems and hardware configurations.

This release introduces the initial Q1_0 Metal backend, including kernel tuning and associated testing infrastructure. It also provides a wide array of pre-compiled binaries for macOS, Linux, Windows, and openEuler platforms.

This release introduces architectural optimizations for the Gemma model by restructuring projection operations within the first layer and before the main layer loop. Pre-compiled binaries are provided for various operating systems and hardware configurations.

This release updates the debug example to disable the cb_eval callback when using the --save-logits flag, reducing noise in the output. It also provides numerous pre-compiled binaries for macOS, Linux, Windows, and openEuler.

This release includes a fix for MiniMax handling within the parser and provides updated pre-compiled binaries for macOS, Linux, Windows, and openEuler across various CPU/GPU configurations.

This release focuses on updating pre-compiled binaries across multiple operating systems and hardware targets, including new support for ROCm 7.2 and CUDA 13.1, alongside minor test cleanup.

This release introduces support for the step3-vl-10b model and includes several internal optimizations and refactoring, such as using fused QKV and updating parameter handling for MmprojModel.

This release introduces KleidiAI-enabled ARM artifacts for macOS and standardizes the macOS release build process. It also provides an extensive set of pre-compiled binaries for various Linux, Windows, and openEuler configurations.

This release significantly speeds up CUDA graph property checks using hashing and introduces optimizations like 'seen node' and 'memcp'. It also provides extensive pre-built binaries for diverse operating systems and hardware configurations.

This release introduces performance improvements for q4_0 and q4_1 mmq kernels on AMD GPUs via ds_read_b128 optimization and includes various bug fixes and code cleanup in the CUDA implementation.

This release introduces support for attention rotation within the kv-cache for heterogeneous iSWA configurations and removes an unnecessary assertion.

This release introduces a safety check in CUDA operations to prevent buffer overlap during fusion and provides numerous updated pre-built binaries across different operating systems and hardware configurations.

This release focuses heavily on optimizing and stabilizing the ggml-webgpu backend by parameterizing submission sizes, adding iOS limits, and removing internal deadlocks. Internal refactoring includes moving types and simplifying profiling futures.

This release addresses a bug in llama-server where model parameters were not being propagated correctly. It also provides numerous pre-compiled binaries for different operating systems and hardware setups.

This release removes per-architecture tensor name lists from the llama component and provides updated binary distributions for numerous operating systems and hardware configurations.

This release addresses a specific bug in the server component related to checkpoint restoration when pos_min is 0. It also provides a comprehensive set of pre-compiled binaries for different operating systems and hardware architectures.

This release deprecates the GGML_OP_ADD1 operation and includes minor internal cleanup, alongside providing updated pre-built binaries for numerous operating systems and hardware configurations.

This release introduces Vulkan build support for ggml on Linux and improves error reporting for fork failures. It also provides a comprehensive set of pre-compiled binaries across multiple platforms and hardware configurations.

This release introduces support for FA dequantization of Q4_1, Q5_0, Q5_1, and IQ4_NL formats within the Vulkan backend. Various pre-compiled binaries for different operating systems and hardware configurations are provided.

This release primarily fixes an incorrect compute capability constant for CDNA2 (gfx90a/MI210) within the ggml-cuda backend. It also provides updated pre-compiled binaries across multiple platforms.

This release introduces a significant Q8_0 reorder optimization for the SYCL backend, improving performance on Intel Arc hardware, and fixes a bug preventing this optimization from activating for Q8_0 tensors.

This release introduces support for the MUL_MAT_ID operation within the ggml-webgpu backend and provides updated pre-compiled binaries for macOS, Linux, Windows, and openEuler platforms.

This release introduces Q1_0 1-bit quantization support for the CPU, involving renaming and removing specific quantization variants and fixing related enum issues.

This release fixes an issue where newline characters were incorrectly stripped in multiline input for llama-cli and includes an internal change replacing '&' with 'string_view'.

This release introduces an optimized flash_attn_stream_k_fixup kernel for CUDA to improve performance under specific conditions. It also provides updated pre-compiled binaries across multiple operating systems and hardware targets.

This release updates the llama-bench tool by adding new command-line arguments (`-fitc` and `-fitt`) and provides a comprehensive set of pre-compiled binaries for numerous operating systems and hardware configurations.

This release introduces byte token handling support for the Gemma4 BPE detokenizer and provides updated pre-compiled binaries for numerous operating systems and hardware configurations.

This release fixes an issue in the server's chunked stream provider to correctly handle unsuccessful writes to the sink, ensuring stream abortion on connection failure. It also provides updated pre-compiled binaries for numerous platforms.

This release includes a minor optimization for argosrt output initialization within the hexagon component and provides updated binary distributions for macOS, Linux, Windows, and openEuler targeting various CPU/GPU architectures.

This release focuses on fixing issues related to GGUF boolean metadata loading, ensuring platform-independent behavior for BOOL metadata within the llama component.

This release introduces comprehensive support for HunyuanOCR models, including vision capabilities and a new chat template. It also includes various fixes related to token IDs and tensor mappings during conversion.

This release updates the llama-server startup logging to remove redundancy and include build/commit information. It also provides numerous pre-compiled binaries for various operating systems and hardware configurations.

This release introduces specialized parsing and improved tool call handling for Gemma 4, alongside various internal cleanups and bug fixes.