●Change8

v0.18.0

Breaking Changes
📦 peftView on GitHub →
⚠ 2 breaking✨ 13 features🔧 7 symbols

Summary

This release introduces seven new PEFT methods including RoAd, ALoRA, and DeLoRA, alongside significant enhancements like stable integration interfaces and support for negative weights in weighted LoRA merging. It also drops support for Python 3.9 and requires an upgrade for compatibility with Transformers v5.

⚠️ Breaking Changes

  • Python 3.9 is no longer supported. Users must upgrade to Python 3.10 or newer.
  • PEFT version < 0.18.0 is incompatible with the imminent Transformers v5 release. Users planning to use Transformers v5 must upgrade PEFT to 0.18.0+.

Migration Steps

  1. If you are using Python 3.9, upgrade your environment to Python 3.10 or newer.
  2. If you plan to use Transformers v5, ensure you upgrade PEFT to version 0.18.0 or later.
  3. If you are using the OFT method, existing checkpoints may yield slightly different results due to numerical stabilization and speed improvements; retraining the adapter is recommended.

✨ New Features

  • Added RoAd (2D Rotary Adaptation) method, supporting mixed adapter batches efficiently.
  • Added ALoRA (Activated LoRA) technique for causal LMs, enabling selective adapter enabling based on token sequence.
  • Added Arrow, a dynamic routing algorithm between multiple loaded LoRAs.
  • Added GenKnowSub, a technique built upon Arrow that modifies the LoRA library by subtracting general knowledge adapters.
  • Added WaveFT (Wavelet Fine-Tuning) method, which trains sparse updates in the wavelet domain.
  • Added DeLoRA (Decoupled Low-rank Adaptation) method, which decouples angle and magnitude similarly to DoRA but aims to prevent divergence.
  • Added OSF (Orthogonal Fine-Tuning) method, which freezes high-rank subspaces and projects gradient updates to a low-rank subspace, useful for continual learning.
  • Introduced a text generation benchmark framework for comparing PEFT methods on metrics like runtime and memory usage.
  • Provided a stable interface of functional utilities (e.g., `set_adapter`) for reliable integration with libraries like Transformers and Diffusers.
  • Added `ensure_weight_tying=True` argument to `LoraConfig` to enforce weight tying (e.g., embedding and LM head) when saving/merging modules.
  • Extended LoHa and LoKr to support `nn.Conv1d` layers and `nn.Conv2d` layers with 1x1 kernels.
  • Added a new random discrete initialization option for prompt tuning.
  • Enabled combining LoRA adapters with negative weights using `model.add_weighted_adapter`.

🔧 Affected Symbols

LoraConfigmodel.add_weighted_adapterset_adapterpeft.tuners.tuners_utils.set_adapterpeft.functionalLoraModel.add_weighted_adapterOFT