●Change8

v0.21.1rc0

Breaking Changes
📦 tokenizersView on GitHub →
⚠ 1 breaking✨ 2 features🐛 5 fixes🔧 2 symbols

Summary

This release upgrades the underlying Rust bindings to PyO3 0.23 and drops support for older Python versions (3.7, 3.8). It also introduces feature updates like support for updating template processors and several bug fixes.

⚠️ Breaking Changes

  • Python 3.7 and Python 3.8 support has been dropped. Users must upgrade to Python 3.9 or newer.

Migration Steps

  1. Ensure your environment uses Python 3.9 or newer, as Python 3.7 and 3.8 support has been removed.

✨ New Features

  • Added support for updating template processors.
  • Added rustls-tls feature for TLS handling.

🐛 Bug Fixes

  • Fixed a panic in DecodeStream::step caused by incorrect index usage.
  • Fixed an issue in stream handling by removing read_index.
  • Fixed an issue with NormalizedString append when the normalized string is empty.
  • Fixed typos in Split docstrings.
  • Fixed general typos in documentation.

🔧 Affected Symbols

DecodeStream::stepNormalizedString