b10643
Breaking Changes📦 llama-cppView on GitHub →
⚠ 1 breaking✨ 11 features🐛 14 fixes🔧 33 symbols
Summary
This release introduces significant enhancements to the Hexagon backend, including multi-NPU device support, a fully asynchronous backend, and improved performance for various operations like ALLREDUCE and matrix multiplications. It also includes numerous bug fixes and optimizations across different Hexagon modules.
⚠️ Breaking Changes
- The hex-hb backend no longer supports optional hostbuf. This change simplifies the backend by removing this option.
Migration Steps
- If you were using optional hostbuf support on the hex-hb backend, you will need to adapt your code as it has been removed.
- Update run.py and build.py scripts for the Hexagon backend.
- Consider setting the AR_SELECT environment variable to choose the ALLREDUCE kernel or fallback if needed.
✨ New Features
- Support for multi-NPU devices (IQ9, IQ10) on the Hexagon backend.
- Fully asynchronous backend for Hexagon.
- Added support for Q8_0 quantization with inplace dequantizers in hex-fa.
- Added support for cloning buffers between sessions to speed up tensor copies in hex-bufs.
- Added support for auto-cloning buffers from different sessions in hex-bufs.
- Added support for ALLREDUCE operation on Hexagon.
- Improved ALLREDUCE with DMA pipeline.
- Added AR_SELECT environment variable to select ALLREDUCE kernel or fallback.
- Fused ALLREDUCE+ADD operation.
- Added row-broadcast to fuse with bias ADD.
- Made fused 2x and 3x matmuls more generic in hex-mm.
🐛 Bug Fixes
- Fixed async copy in hex-hb.
- Relaxed supported unary check in hex-unary.
- Used same get_alignment for host bufs in hex-bufs.
- Fixed Q8_0 support for get/set rows in hex-rows and hex-get-rows.
- Fixed perf regression with larger tensors in hex-geglu.
- Added missing headers in hex-get-rows and hex-set-rows.
- Resurrected GGML_HEXAGON_HOSTBUF but disabled it by default in hex-bufs.
- Did not reject ops with non-hexagon buffers in hexagon.
- Applied >=32 restriction only for q8_0 in hex-get-rows.
- Fixed Q8_0 support in hex-set-rows.
- Handled get_tensor_2d offsets in hex-repack.
- Fixed incorrect MUL_MAT reordering in hex-fusion.
- Fixed offset calculation with row-chunking in hex-get-rows.
- Fixed get_tensor_2d fixes for non-zero offsets in hex-repack.