v4.57.2
Breaking Changes📦 transformersView on GitHub →
⚠ 1 breaking✨ 3 features🐛 2 fixes🔧 3 symbols
Summary
This patch release focuses on fixing Mistral tokenizer mappings and Tekken pattern matching, while also correcting a decorator error in the device memory utility.
⚠️ Breaking Changes
- The @staticmethod decorator was removed from the module-level function get_device_and_memory_breakdown, which may cause errors if code was attempting to access it as a static method of a class or via an incorrect namespace.
Migration Steps
- Ensure calls to get_device_and_memory_breakdown are treated as standard module-level function calls rather than static methods.
✨ New Features
- Added AutoTokenizer mapping support for mistral3 and ministral models.
- Added automatic conversion for tekken.json tokenizer files.
- Added input validation checks for model hidden states.
🐛 Bug Fixes
- Fixed pattern matching issues for Tekken tokenizers.
- Fixed an invalid @staticmethod decorator on the module-level get_device_and_memory_breakdown function.
🔧 Affected Symbols
AutoTokenizerget_device_and_memory_breakdowntekken.json