v0.22.0
📦 tokenizersView on GitHub →
✨ 3 features🐛 3 fixes🔧 2 symbols
Summary
This release introduces native async bindings and adds `from_bytes`/`read_bytes` methods to WordPiece Tokenizer for WebAssembly compatibility. It also includes several bug fixes and dependency updates.
✨ New Features
- Implement `from_bytes` and `read_bytes` Methods in WordPiece Tokenizer for WebAssembly Compatibility.
- New stream implementation.
- Add native async bindings via py03-async-runtimes.
🐛 Bug Fixes
- Use AHashMap to fix compile error.
- Fix missing parenthesis in `EncodingVisualizer.calculate_label_colors`.
- Replace paste with pastey to address RUSTSEC-2024-0436.
🔧 Affected Symbols
WordPiece TokenizerEncodingVisualizer.calculate_label_colors