v4.49.0-SigLIP-2
Breaking Changes📦 transformersView on GitHub →
⚠ 4 breaking✨ 5 features🔧 5 symbols
Summary
This release introduces SigLIP-2, a new family of multilingual vision-language encoders featuring improved semantic understanding and support for native aspect ratio image processing via the NaFlex variant.
⚠️ Breaking Changes
- When using SigLIP2 models, the output logits should be processed using the sigmoid activation function instead of softmax, as the training loss differs from standard CLIP models.Fix: Replace any usage of softmax on the output logits of SigLIP2 models with torch.sigmoid().
- If using the standalone [GemmaTokenizerFast] for text preprocessing with SigLIP2, specific padding arguments are required to match training configuration.Fix: When using GemmaTokenizerFast, ensure you pass padding="max_length" and max_length=64 during tokenization.
- The model was trained with lowercased text, meaning input text labels must match this format for optimal performance.Fix: Ensure all text labels used for classification or retrieval tasks are converted to lowercase before processing.
- To achieve results consistent with the provided examples (especially for zero-shot classification), a specific prompt template must be used.Fix: Use the prompt template "this is a photo of {label}" when formatting text inputs for classification tasks.
Migration Steps
- If you are installing this new version via Git, update your installation command to use the specific tag: `pip install git+https://github.com/huggingface/transformers@v4.49.0-SigLIP-2`.
- If you are using SigLIP2 models, modify your inference code to apply torch.sigmoid() to the raw output logits instead of torch.softmax().
- If you are manually tokenizing text inputs for SigLIP2 using GemmaTokenizerFast, ensure you set `padding="max_length"` and `max_length=64`.
- Verify that all text inputs (labels or prompts) passed to the SigLIP2 model are lowercased.
- If using the model for zero-shot classification, adopt the recommended prompt template: "this is a photo of {label}".
✨ New Features
- Added SigLIP-2 model to transformers, supporting both FixRes and NaFlex variants.
- Support for NaFlex (Native Flexible) variant which handles variable image aspect ratios and resolutions.
- Support for FixRes variant which is backward compatible with SigLIP v1.
- Integration with the Pipeline API for zero-shot image classification.
- Support for higher resolution processing via the max_num_patches parameter in the Processor.