●Change8

Migrating to Tokenizers v0.21.1rc0

Version v0.21.1rc0 introduces 1 breaking change. This guide details how to update your code.

Released: 3/12/2025

1
Breaking Changes
1
Migration Steps
2
Affected Symbols

⚠️ Check Your Code

If you use any of these symbols, you need to read this guide:

DecodeStream::stepNormalizedString

Breaking Changes

●Issue #1

Python 3.7 and Python 3.8 support has been dropped. Users must upgrade to Python 3.9 or newer.

Migration Steps

  1. 1
    Ensure your environment uses Python 3.9 or newer, as Python 3.7 and 3.8 support has been removed.

Release Summary

This release upgrades the underlying Rust bindings to PyO3 0.23 and drops support for older Python versions (3.7, 3.8). It also introduces feature updates like support for updating template processors and several bug fixes.

Need More Details?

View the full release notes and all changes for Tokenizers v0.21.1rc0.

View Full Changelog