tokenizer
Found in 4 packages: htmx, ollama, llama-cpp, unsloth
htmx(1 releases)
ollama(8 releases)
v0.22.1-rc0This release introduces model batching support and fixes several issues related to tokenization and desktop application startup behavior. It also includes support for NVIDIA TensorRT Model Optimizer import.
v0.22.1This release introduces model batching support and TensorRT Model Optimizer import for the mlx backend. It also includes several bug fixes related to tokenization and desktop application startup behavior.
v0.22.1-rc1This release introduces model batching support and adds NVIDIA TensorRT Model Optimizer import capability. Several minor bugs related to tokenization and desktop app session handling were also resolved.
v0.22.0-rc1This release introduces support for NVIDIA TensorRT Model Optimizer import within mlx and fixes an issue related to multi-regex BPE offset handling in the tokenizer. It also includes performance improvements by batching the sampler across multiple sequences in mlxrunner.
v0.20.0-rc1This release introduces the new Gemma 4 model family variants (E2B, E4B, 26B, 31B) and enhances tokenizer capabilities with SentencePiece-style BPE support.
v0.20.0This release introduces the new Gemma 4 model family variants (E2B, E4B, 26B, 31B) and enhances tokenizer capabilities with SentencePiece-style BPE support.
v0.17.0-rc1This release introduces UI exposure of the server context length and implements OpenClaw onboarding, alongside internal consolidation of the tokenizer.
v0.9.2This patch release focuses on bug fixes for tool calling, generation errors, and tokenization issues across specific model architectures.
llama-cpp(1 releases)
unsloth(1 releases)
Track Symbol Changes
Use the Change8 MCP server or GitHub Action to get notified when tokenizer changes.
Learn More