●Change8

Migrating to Tokenizers v0.21.1

Version v0.21.1 introduces 1 breaking change. This guide details how to update your code.

Released: 3/13/2025

1
Breaking Changes
1
Migration Steps
2
Affected Symbols

⚠️ Check Your Code

If you use any of these symbols, you need to read this guide:

DecodeStream::stepNormalizedString

Breaking Changes

●Issue #1

Support for Python 3.7 and Python 3.8 has been dropped. Users must upgrade to Python 3.9 or newer.

Migration Steps

  1. 1
    Ensure your environment uses Python 3.9 or newer, as support for Python 3.7 and 3.8 has been removed.

Release Summary

This release upgrades the underlying Rust bindings to PyO3 0.23, drops support for older Python versions (3.7/3.8), and includes several bug fixes related to streaming and string normalization.

Need More Details?

View the full release notes and all changes for Tokenizers v0.21.1.

View Full Changelog