v5.6.1
Breaking Changes📦 sentence-transformersView on GitHub →
⚠ 1 breaking🐛 2 fixes🔧 3 symbols
Summary
This patch release fixes silently degraded embeddings for RoBERTa-family models when flash attention is used with `transformers` v5. The issue affected specific multilingual embedding models and has been resolved by correcting `position_ids` offsetting.
⚠️ Breaking Changes
- Users who encoded text with flash attention requested on `transformers` v5 with a RoBERTa-family checkpoint will experience silently degraded embeddings. To fix this, re-encode the corpus after upgrading to v5.6.1, as pre-fix embeddings score notably worse and should not be mixed with post-fix embeddings.
Migration Steps
- If you encoded text with flash attention requested on `transformers` v5 with a RoBERTa-family checkpoint, re-encode your corpus after upgrading to v5.6.1.
🐛 Bug Fixes
- Fixed silently degraded embeddings for RoBERTa-family models when flash attention is requested with `transformers` v5. This bug affected v5.5.0, v5.5.1, and v5.6.0.
- Corrected `position_ids` offsetting for RoBERTa-family models when flattening inputs for flash attention, resolving an issue where tokens were read with a shifted position embedding.