●Change8

Migrating to Tokenizers v0.23.1

Version v0.23.1 introduces 5 breaking changes. This guide details how to update your code.

Released: 4/27/2026

5
Breaking Changes
4
Migration Steps
8
Affected Symbols

⚠️ Check Your Code

If you use any of these symbols, you need to read this guide:

tokenizers (Python package)add_tokenstokenizer.json (added_tokens block)RobertaProcessing.__init__Tokenizer (Python class)AddedToken (Python class)Encoding (Python class)models.Unigram

Breaking Changes

●Issue #1

Python 3.9 support has been dropped; users must use Python >=3.10.

●Issue #2

The 'content' field in 'added_tokens' block of 'tokenizer.json' is now normalized upon insertion via 'add_tokens'. Re-saved files may differ in this block.

●Issue #3

Type stubs are now precise. Methods previously returning 'Any' now return concrete types, which may cause 'mypy --strict' to surface new errors.

●Issue #4

The stub layout has changed from 'tokenizers/submodule/__init__.pyi' to 'tokenizers/submodule.pyi', potentially breaking imports relying on the old structure (e.g., 'RobertaProcessing.__init__').

●Issue #5

On Python 3.14t (free-threaded), setters/getters now return 'PyResult<T>' because the underlying state uses 'Arc<RwLock<Tokenizer>>'. A poisoned lock surfaces as a 'PyException' instead of causing a panic.

Migration Steps

  1. 1
    Upgrade Python version to 3.10 or newer.
  2. 2
    If using 'add_tokens', be aware that re-saving 'tokenizer.json' will reflect normalized content in the 'added_tokens' block.
  3. 3
    If using strict type checking (e.g., 'mypy --strict'), review code for newly surfaced type errors due to precise type hints.
  4. 4
    If relying on internal stub structure, update imports referencing submodule stubs (e.g., 'tokenizers/processor.pyi' instead of 'tokenizers/processor/__init__.pyi').

Release Summary

Version 0.23.1 is the first stable release in the 0.23 line, featuring massive performance gains in vocabulary loading, full Python type hints, and stable multi-platform Node.js bindings. Python 3.9 support has been dropped.

Need More Details?

View the full release notes and all changes for Tokenizers v0.23.1.

View Full Changelog