b9893
📦 llama-cppView on GitHub →
✨ 6 features🐛 2 fixes🔧 10 symbols
Summary
This release focuses heavily on OpenCL performance optimizations for flash attention decoding, including vectorized kernels and specific tuning for various quantization levels and model architectures like gemma-4. Several critical bugs related to DK=512 decode crashes and OpenCL compilation errors were also resolved.
Migration Steps
- For DK=512 decode-only programs, note that the operation now runs on the CPU because it was found to be bandwidth-bound and faster there than on the GPU; prefill remains on the GPU.
- For OpenCL builds targeting multiquery GQA where DK>=256, the compilation program is now minimal (FA_MQ_ONLY) to prevent Adreno compiler out-of-memory errors.
✨ New Features
- OpenCL: Implemented vectorized flash-attention decode kernels for f16/q8_0/q4_0 KV.
- OpenCL: Improved non-FA KQ mv kernels.
- OpenCL: Added tweaks for multiquery FA.
- OpenCL: Added tweaks for FA q1 kernels.
- OpenCL: Enabled FA with DK=DV=512 for gemma-4.
- OpenCL: Split f16-KV FA decode finer using FD_KV_PER_SPLIT_F16 (default 512 instead of 2048 for f16 KV).
🐛 Bug Fixes
- OpenCL: Fixed FA decode crash for DK=512 (gemma-4) by guarding block_n to avoid out-of-range read at dispatch.
- OpenCL: Removed stray token (".") that broke the f32_f16 program build.