●Change8

Migrating to PEFT v0.16.0

Version v0.16.0 introduces 4 breaking changes. This guide details how to update your code.

Released: 7/3/2025

4
Breaking Changes
4
Migration Steps
6
Affected Symbols

⚠️ Check Your Code

If you use any of these symbols, you need to read this guide:

LoRADoRAOFTPrefix Tuninginject_adaptervlm.language_model

Breaking Changes

●Issue #1

Orthogonal Finetuning (OFT) refactor makes it incompatible with old OFT checkpoints. Users must either pin PEFT version to <0.16.0 or retrain checkpoints using the new PEFT version.

●Issue #2

Due to a Transformers VLM refactor, PEFT prompt learning methods applied to `vlm.language_model` will no longer work; apply them directly to `vlm` instead.

●Issue #3

Loading checkpoints trained after the Transformers VLM refactor is not possible with PEFT versions before this release; upgrade both PEFT and transformers.

●Issue #4

Prefix tuning may show numerical differences due to attention mask refactors in Transformers. If performance is degraded, especially for models using 4d attention masks like Gemma, re-train checkpoints or pin PEFT to <0.16.0 and transformers to <4.52.0.

Migration Steps

  1. 1
    If using OFT checkpoints trained with older PEFT versions, either pin PEFT to `<0.16.0` or retrain the checkpoints using this new PEFT version.
  2. 2
    If using PEFT prompt learning methods on Vision Language Models (VLMs) from Transformers, change the target from `vlm.language_model` to `vlm` directly.
  3. 3
    If using prompt learning methods (especially prefix tuning) and observing numerical differences or performance issues after upgrading Transformers, re-train the affected checkpoints or pin PEFT to `<0.16.0` and transformers to `<4.52.0`.
  4. 4
    When using LoRA with `Conv2d` layers where `groups != 1`, ensure the rank `r` is divisible by `groups`.

Release Summary

This release introduces three major new PEFT methods: LoRA-FA, RandLoRA, and C3A, alongside significant enhancements like QLoRA support and broader layer compatibility for LoRA and DoRA. It also includes critical compatibility updates related to recent changes in the Hugging Face Transformers library.

Need More Details?

View the full release notes and all changes for PEFT v0.16.0.

View Full Changelog