●Change8

2025-01

📦 unslothView on GitHub →
✨ 6 features🐛 14 fixes🔧 6 symbols

Summary

This release introduces full support for the Phi-4 model, including fixes for tokenization and chat templates, alongside significant bug fixes for gradient accumulation, vision models, and performance regressions. It also brings Windows support and performance improvements via updated Xformers.

Migration Steps

  1. If you are seeing significant or unusual loss results, update Unsloth by following the latest instructions: https://docs.unsloth.ai/get-started/installing-+-updating/updating
  2. To update Unsloth without dependency updates, run: pip install --upgrade --force-reinstall --no-deps --no-cache-dir unsloth unsloth_zoo
  3. Alternatively, install the latest version via: pip install unsloth

✨ New Features

  • Added support for Phi-4 model.
  • Phi-4 model has been converted to Llama architecture and fixed versions/GGUFs uploaded to Hugging Face.
  • Enabled automatic gradient accumulation fixes for all models.
  • Added Windows support via Triton Windows and Xformers for Windows.
  • Upgraded Xformers support to version 0.0.29, resulting in 10% faster H100 training.
  • Updated all notebooks to use Github links.

🐛 Bug Fixes

  • Fixed an issue causing significant or unusual loss results due to a new transformers version (requires Unsloth update).
  • Corrected Phi-4 EOS token to be <|im_end|> instead of <|endoftext|>.
  • Fixed missing `add_generation_prompt` in Phi-4 chat template.
  • Corrected Phi-4 pad token to be <|dummy_87|> instead of <|endoftext|>.
  • Fixed gradient accumulation bug on Llama 3.2 Vision.
  • Fixed attention backwards performance regression for T4s and V100s when using Xformers < 0.0.29.
  • Enabled gradients correctly for PEFT modules on vision models by applying `requires_grad_(True)` beyond just `model.get_input_embeddings()`.
  • Fixed `train_on_responses_only` functionality for Phi-4.
  • Fixed vision model tokenizer padding side.
  • Fixed loader.py to work correctly on Windows.
  • Updated save.py warning message.
  • Fixed chat template logic to handle cases where a template has both endif and endfor.
  • Passed position embeddings explicitly from the decoder layer.
  • Fixed a Name Error by correctly importing Version from packaging.version.

🔧 Affected Symbols

Phi-4Llama 3.2 VisionXformersloader.pysave.pyvision model tokenizer