v4.53.2-modernbert-decoder-preview
Breaking Changes📦 transformersView on GitHub →
⚠ 2 breaking✨ 5 features🔧 5 symbols
Summary
This release introduces a preview of the ModernBERT Decoder, a causal language model variant of the ModernBERT architecture designed for autoregressive generation and sequence classification.
⚠️ Breaking Changes
- The release notes describe a preview tag (`v4.53.2-modernbert-decoder-preview`) which is not a standard semantic version bump. Migrating to this specific version requires installing directly from the Git tag, which bypasses standard PyPI releases.Fix: If you need the preview features immediately, install using the provided Git command: `pip install git+https://github.com/huggingface/transformers@v4.53.2-modernbert-decoder-preview`. Otherwise, wait for the official minor release `v4.54.0`.
- The usage examples show that the ModernBERT Decoder model can be loaded using `AutoModelForCausalLM` for generation tasks, but also incorrectly suggests loading it via `AutoModelForSequenceClassification`.Fix: For text generation tasks using the decoder model, use `AutoModelForCausalLM`. If you intend to use the decoder for classification (as shown in the example), ensure the model configuration supports it, but typically decoders are used for generation. For standard generation, stick to `AutoModelForCausalLM`.
Migration Steps
- If you require immediate access to the ModernBERT Decoder architecture before the official v4.54.0 release, update your installation by installing directly from the preview tag:
- Run the following command: `pip install git+https://github.com/huggingface/transformers@v4.53.2-modernbert-decoder-preview`
- When using the model for text generation, ensure you are loading it with `AutoModelForCausalLM` (as demonstrated in the usage examples).
- When generating text, ensure you set `pad_token_id=tokenizer.eos_token_id` if you are using batch generation or need to explicitly define padding behavior, as shown in the example.
- If you are relying on standard semantic versioning, plan to upgrade to version `v4.54.0` when it is officially released, which will include this model without requiring Git installation.
✨ New Features
- Introduction of ModernBERT Decoder architecture for causal language modeling (CLM).
- Support for autoregressive text generation tasks using ModernBERT architecture.
- Incorporates rotary positional embeddings supporting sequences up to 8192 tokens.
- Includes architectural improvements like unpadding, GeGLU layers, and alternating attention patterns.
- Support for ModernBERT Decoder in pipeline (text-generation, text-classification) and AutoModel classes.