●Change8

Migrating to Tokenizers v0.21.2

Version v0.21.2 introduces 1 breaking change. This guide details how to update your code.

Released: 6/24/2025

1
Breaking Changes
1
Migration Steps
3
Affected Symbols

⚠️ Check Your Code

If you use any of these symbols, you need to read this guide:

BPE Python binding (merges type notation)continuing_prefix_trainer_mismatch (test data path)from_pretrained function (now uses ApiBuilder::from_env())

Breaking Changes

●Issue #1

Fix training with special tokens: Behavior when training tokenizers might have changed regarding how special tokens are handled.

Migration Steps

  1. 1
    Review training logic if you rely on specific behavior regarding special tokens during tokenizer training, due to breaking change in PR #1617.

Release Summary

This release focuses on performance optimizations, enabling broader Python no GIL support, and fixing several issues related to onig compilation and training logic.

Need More Details?

View the full release notes and all changes for Tokenizers v0.21.2.

View Full Changelog