b10242
📦 llama-cppView on GitHub →
✨ 4 features🐛 3 fixes🔧 7 symbols
Summary
Introduced a new backend sampler for penalties, enhancing penalty handling and adding support for top-k penalties. This release also includes bug fixes for numerical stability and penalty validation.
Migration Steps
- Replace penalty_last_n x n_candidates comparison matrix with a vocabulary-sized count tensor.
✨ New Features
- Add backend sampler for penalties sampler.
- Enhance penalty handling in common_sampler_init by setting default value for penalty_last_n based on model context if not specified, ensuring penalty_last_n and n_prev are non-negative, updating llama_sampler_penalties structure to inherit from llama_sampler_backend and add backend input handling for penalties, and implementing backend initialization and application logic for penalties, including frequency and presence adjustments.
- Add support for top-k penalties in backend sampling.
- Add support for penalties in sampler chain with configurable positions.
🐛 Bug Fixes
- Ensure stable numerical results by preserving masked logits as -Inf and no longer generating NaN.
- Validate repeat penalty to ensure it is finite and greater than 0; add tests for invalid values.
- Add validation for penalty parameters and enhance tests for non-finite values.