v4.51.3-CSM-preview
Breaking Changes📦 transformersView on GitHub →
⚠ 1 breaking✨ 5 features🔧 6 symbols
Summary
This release introduces the Conversational Speech Model (CSM), an open-source contextual text-to-speech model capable of generating natural speech from multi-turn dialogue context.
⚠️ Breaking Changes
- The new CSM model and related components are introduced, which requires updating the `transformers` library version.Fix: If you intend to use CSM, you must install the specific preview tag: `pip install git+https://github.com/huggingface/transformers@v4.51.3-CSM-preview`. Otherwise, no action is needed unless you rely on strict versioning, as this tag does not follow semantic versioning.
Migration Steps
- If you wish to use the new CSM model immediately, update your installation by running: pip install git+https://github.com/huggingface/transformers@v4.51.3-CSM-preview.
- If you are not using CSM, be aware that the next minor release, v4.52.0, will include this model, which might affect dependency resolution if you rely on strict minor version pinning.
- When using CSM, ensure you import `CsmForConditionalGeneration` from `transformers`.
- When preparing inputs for CSM, use the `AutoProcessor` and structure your input as a conversation list, potentially including text and audio content types, as demonstrated in the usage examples.
- When generating output, use `model.generate(**inputs, output_audio=True)` and process the resulting audio using `processor.save_audio()`.
✨ New Features
- Introduction of the Conversational Speech Model (CSM), a contextual text-to-speech model.
- Support for multi-turn dialogue context using text and spoken audio sequences.
- Support for batched inference.
- Support for full-graph compilation with CUDA graphs and static cache for optimized performance.
- Integration with the Mimi codec model for discrete codebook token encoding/decoding.