llama.cpp

This release includes an update to ggml to add MUL_MAT to operations that may require additional memory for WebGPU. Various pre-compiled binaries for different platforms and configurations are also provided.

This release introduces Vulkan backend improvements for top-k operations with Qwen 3.8 Flash Next, including a new radix sort shader and qsa fusion. It also includes updated test cases for this functionality.

This release addresses a CPY fence bug in the hexagon backend. It also provides pre-compiled binaries for various platforms and configurations.

This release adds remaining Q4_1/Q5_0/Q5_1 fa-vec tunings for M2 on metal. It also provides pre-compiled binaries for various platforms and configurations.

Improved RPC handling by preventing serialization of buffers from other servers. This change enhances the robustness of cross-server communication within the RPC system.

This release includes a fix for the ggml_backend_buft_get_alloc_size() guard. It also provides pre-compiled binaries for various platforms and hardware configurations.

Optimized kv-cells sequence scan for improved generation performance on longer contexts. This release also provides pre-compiled binaries for various platforms and hardware configurations.

This release introduces SWIGLU_CLAMP to ggml and adds Vulkan shader support, enhancing computational capabilities.

This release improves TENSOR_READ_LAZY handling on CPU and provides updated binary releases for various platforms including macOS, Linux, Android, and Windows.

Introduced an optimized CUDA path for mm_ids_helper, improving performance for specific expert configurations. This update expands support for more n_expert_used values on the fast path.

This release includes tuning for RDNA 3 MMQ configuration, along with various pre-compiled binaries for different platforms and hardware accelerators.

Optimized HIP's Q2_0 dot-product path for gfx1201 and improved MMQ unpack with native permutations. Restored MMQ tile index handling.

This release addresses a critical bug in DFlash2 NVFP4 models by ensuring that Q, K, V, and output projection scales are correctly passed to attention operations, resolving issues with speculative token generation. Various pre-compiled binaries for different platforms and hardware are also provided.

The --tensor-read-lazy CLI argument has been renamed to --lazy-mode to align with the internal parameter, and a new -lzm shorthand has been added.

Introduced `ggml_backend_op_alloc_size_may_expand` to better manage memory allocation for specific operations across various backends. The RPC backend has been updated to utilize this new helper function.

This release addresses an RPC issue related to Apple RDMA teardown errors. It also provides pre-compiled binaries for various platforms and hardware configurations.

This release introduces fa-vec tunings for M3 Ultra on metal, enhancing performance. It also provides a comprehensive set of pre-compiled binaries for various platforms and configurations.

This release introduces fa-vec tuning for M3 Pro on Metal, enhancing performance. It also provides updated pre-compiled binaries for various platforms and configurations.

Updated the cpp-httplib dependency to versions 0.54.0 and 0.54.1. This release also provides numerous pre-compiled binaries for various platforms and hardware configurations.

This release includes a fix for pre-RDMA support on older macOS versions. Various pre-compiled binaries for different platforms and configurations are also provided.

Introduced Hexagon device support for discovery and on-demand session creation, including lazy allocation and runtime NPU core discovery. Non-existing devices are now rejected early during initialization.

The SYCL backend has been updated to split long rows in TOP_K operations, enhancing performance. This release also provides pre-compiled binaries for various platforms and hardware configurations.

This release addresses a critical null-pipeline crash in Metal for F16 src1 mul_mat/mul_mat_id operations and includes related stability improvements.

This release addresses a memory-related crash during context shifts for unquantized K caches. It ensures the Hadamard matrix is copied to the k_rot tensor only when a buffer is assigned, preventing potential issues.

This release introduces the ability to pass alloc dependencies into the graph_optimize function within ggml. It also includes various pre-compiled binaries for different platforms and hardware accelerators.