●Change8

v1.10.0

Breaking Changes
📦 accelerateView on GitHub →
⚠ 1 breaking✨ 4 features🐛 6 fixes🔧 5 symbols

Summary

This release introduces comprehensive N-D Parallelism support via `ParallelismConfig` integrated with `Accelerator`, alongside significant FSDP improvements, particularly for MoE models.

⚠️ Breaking Changes

  • Removed `ParallelismConfig` from `PartialState`. If you were accessing this configuration via `PartialState`, you must now retrieve it through the `Accelerator` object or other appropriate means.

Migration Steps

  1. If you were relying on `ParallelismConfig` being present in `PartialState`, update your code to retrieve parallelism configuration directly from the `Accelerator` instance.
  2. If you were using FSDP with ignored modules, consider using regex strings for more flexible exclusion patterns.

✨ New Features

  • Introduced an easy-to-use integration for N-D Parallelism (combining TP, CP, DP) via the `ParallelismConfig` object passed to the `Accelerator` constructor.
  • Added support for context parallel v2.0.
  • FSDP ignored modules can now be specified as a regex string.
  • Added a `Parallelism` getter property to the `Accelerator` class.

🐛 Bug Fixes

  • Fixed an issue where the default `submesh_tp_size` was not set, leading to an unset local variable error.
  • Fixed an issue where `accelerator.prepare` failed when only TP parallelism was specified.
  • Fixed an issue where TP size would not be read from environment variables.
  • Fixed FSDP ignored modules attribute, allowing training of PEFT models with MoE layers containing `q_proj` and `v_proj` parameters (important for models like `gpt-oss`).
  • CpuOffload's `pre_forward` step now checks if tensors are already on the target device before attempting to move them.
  • Ensured environment variable values are read case-insensitively in Accelerate.

🔧 Affected Symbols

AcceleratorParallelismConfigPartialStateAutoModelForCausalLMFSDP