b10236
📦 llama-cppView on GitHub →
✨ 5 features🐛 3 fixes🔧 1 symbols
Summary
Introduced the DSv4 Lightning Indexer with F16 support, improving performance for large models. This release also includes optimizations for K cache handling and bug fixes.
✨ New Features
- Implement GGML_OP_LIGHTNING_INDEXER for 128-dimensional, 64-head inputs with F32 queries and weights plus F16 keys and masks.
- Add tiled and tail kernels and test KV lengths around 8- and 64-element boundaries for the F16 Lightning Indexer.
- Stage and dequantize K in F16 threadgroup memory before simdgroup matrix loads.
- Zero-fill partial tiles and guard stores so all KV segments use the same numerical path.
- Support F32, F16, BF16, Q4_0, Q4_1, Q5_0, Q5_1, and Q8_0 K caches for the Lightning Indexer.
🐛 Bug Fixes
- Dedup Lightning Indexer constants.
- Fix flaky test related to Lightning Indexer.
- Fix whitespace issues in 'cont' module.