●Change8

v0.21.1

Breaking Changes
📦 tokenizersView on GitHub →
⚠ 1 breaking✨ 2 features🐛 4 fixes🔧 2 symbols

Summary

This release upgrades the underlying Rust bindings to PyO3 0.23, drops support for older Python versions (3.7/3.8), and includes several bug fixes related to streaming and string normalization.

⚠️ Breaking Changes

  • Support for Python 3.7 and Python 3.8 has been dropped. Users must upgrade to Python 3.9 or newer.

Migration Steps

  1. Ensure your environment uses Python 3.9 or newer, as support for Python 3.7 and 3.8 has been removed.

✨ New Features

  • Added support for the `rustls-tls` feature.
  • Template processors update support was temporarily removed to maintain backward compatibility.

🐛 Bug Fixes

  • Fixed a panic in `DecodeStream::step` caused by incorrect index usage.
  • Fixed an issue in stream handling by removing `read_index`.
  • Fixed an issue when appending to `NormalizedString` when the normalized string is empty.
  • Fixed typos in documentation (Split docstrings and general typos).

🔧 Affected Symbols

DecodeStream::stepNormalizedString