b9911
📦 llama-cppView on GitHub →
✨ 4 features🐛 3 fixes🔧 4 symbols
Summary
This release focuses heavily on CUDA performance optimizations by fusing MMVQ post-scale operations for NVFP4 and adding dense MMVQ fusion. Several cleanups and reordering steps were performed to stabilize and restrict fusion benefits to appropriate kernels.
Migration Steps
- Remove the environment variable previously used to disable fusion, as fusion is now managed differently/enabled by default under certain conditions.
✨ New Features
- Fused MMVQ post-scale operation for NVFP4 kernels in CUDA.
- Added dense MMVQ fusion.
- Enabled fusion for MoE models in shared MMVQ operations.
- Merged old mul_mat patterns into the lane-based approach.
🐛 Bug Fixes
- Reordered bias/scale operations to adhere to requirements specified in #24331.
- Restricted scale_view_nodes and enrolled MM + ADD into lane-matcher.
- Restricted scale-fusion specifically to NVFP4 kernels due to potential performance regressions on other quantizations caused by a heavy prologue in GEMV.