v0.23.2
📦 tokenizersView on GitHub →
✨ 2 features🐛 5 fixes🔧 2 symbols
Summary
This release includes the implementation of Parity-aware BPE and various CI/CD improvements. It also marks the final v0 release before the transition to v1.
✨ New Features
- Implementing Parity-aware BPE
- Add a ParityBpeTrainer example and list it in the trainers API docs
🐛 Bug Fixes
- Ci test fixes
- Avoid full vocab clone in get_vocab_size()
- Bindings: acquire model read-lock once per call instead of per pre-token
- Make doc builds faster
- fix(python): pin the ruff rule set so a ruff release can't redden main