Tokenizers

This release includes the implementation of Parity-aware BPE and various CI/CD improvements. It also marks the final v0 release before the transition to v1.

Version 0.23.1 is the first stable release in the 0.23 line, featuring massive performance gains in vocabulary loading, full Python type hints, and stable multi-platform Node.js bindings. Python 3.9 support has been dropped.

Ensures that necessary npm directories are created before artifacts are written, resolving a publish failure for the `darwin-universal` target. This change makes the artifact creation process more robust by handling missing directories automatically.

The `@napi-rs/cli` has removed the `--name` flag from the `napi universal` command, now inferring the name from `package.json`. This change requires users to adjust their build commands.

This release focuses on performance improvements, achieving 4x to 8x faster vocab loading with many added tokens due to GIL-free operations, alongside general typing and bug fixes.

Release v0.22.1 primarily involves bumping the upper version constraint for huggingface_hub and includes several documentation updates and trainer signature improvements.

This release introduces native async bindings and adds `from_bytes`/`read_bytes` methods to WordPiece Tokenizer for WebAssembly compatibility. It also includes several bug fixes and dependency updates.

This release (v0.21.4) is a re-release of v0.21.3 because the initial v0.21.3 release failed.

This patch release addresses Clippy fixes and resolves an introduced backward breaking change in the Rust APIs.

This release focuses on performance optimizations, enabling broader Python no GIL support, and fixing several issues related to onig compilation and training logic.

This release upgrades the underlying Rust bindings to PyO3 0.23, drops support for older Python versions (3.7/3.8), and includes several bug fixes related to streaming and string normalization.

This release upgrades the underlying Rust bindings to PyO3 0.23 and drops support for older Python versions (3.7, 3.8). It also introduces feature updates like support for updating template processors and several bug fixes.