●Change8

v5.7.0

Breaking Changes
📦 sentence-transformersView on GitHub →
⚠ 2 breaking✨ 10 features🐛 7 fixes⚡ 1 deprecations🔧 13 symbols

Summary

This release focuses on correctness and performance, rebuilding gradient-cached losses on a shared engine for faster training and fixing several gradient bugs. It also significantly improves inference speed when using `model.compile()`.

⚠️ Breaking Changes

  • int8/uint8 embedding quantization now clips out-of-range values and floors bucket values. This means int8 outputs are no longer bit-identical with earlier versions. Re-quantize existing int8 corpora rather than mixing old and new quantized embeddings.
  • `AdaptiveLayerLoss` and `Matryoshka2dLoss` now weight prior-layer losses uniformly by default. This change affects how losses from different layers are combined during training.

Migration Steps

  1. If you are using int8 quantization, re-quantize your existing int8 corpora as outputs are no longer bit-identical.
  2. If you are using `AdaptiveLayerLoss` or `Matryoshka2dLoss` and relied on the previous default layer weighting, you may need to adjust the `layer_weighting` parameter.
  3. If you are loading models with custom code from outside `sentence_transformers`, ensure you are prepared to use `trust_remote_code=True` in v6.0.

✨ New Features

  • Rebuilt all gradient-cached losses on one shared engine for correctness and performance.
  • Added token-based mini-batching (`mini_batch_num_tokens`) for up to 3.9x faster cached-loss training.
  • `model.compile()` now speeds up inference (`encode()` and `predict()`) by running the forward pass through `__call__`.
  • Introduced `mode="reduce-overhead"` for `torch.compile` to apply CUDA graphs for significant speedups on larger models.
  • Fixed an autocast dtype crash in the backward pass of cached losses.
  • Fixed trainer retaining autograd graphs between logging steps when tracking loss components.
  • Updated `MegaBatchMarginLoss` to train on the full batch, work with `MatryoshkaLoss`, evaluate under `torch.no_grad`, and raise for a third input column instead of silently ignoring it.
  • Packed bits per embedding in binary/ubinary quantization (`precision="binary"`/`"ubinary"`) to fix crashes with embedding dimensions not divisible by 8.
  • Added `detached_kl_teacher` argument to `AdaptiveLayerLoss` to prevent the KL divergence loss from influencing the teacher model's gradients.
  • Added `layer_weighting` argument to `AdaptiveLayerLoss` to allow custom weighting of prior-layer losses.

🐛 Bug Fixes

  • Fixed silently wrong gradients in several cached-loss scenarios.
  • Fixed Cross Encoder `CachedMultipleNegativesRankingLoss` on GPU where backward pass used different dropout masks than forward pass, silently biasing gradients.
  • Fixed `CachedGISTEmbedLoss` and Cross Encoder loss backpropagating wrong batch's gradients when two forward passes were run before a backward pass.
  • Fixed `Pooling(include_prompt=False)` mutating attention mask in place, causing backward re-embedding to use a different mask.
  • Fixed `MatryoshkaLoss(GISTEmbedLoss(...))` where the guide model overwrote cached embeddings, training only the largest Matryoshka dimension.
  • Fixed `quantize_embeddings` where values outside the calibration range wrapped around during float-to-integer cast.
  • Fixed `quantize_embeddings` with `precision="binary"`/`"ubinary"` crashing when embedding dimension was not a multiple of 8.

Affected Symbols

CachedMultipleNegativesRankingLossCachedGISTEmbedLossCachedSpladeLossCrossEncoder.CachedMultipleNegativesRankingLossMegaBatchMarginLossPoolingMatryoshkaLossAdaptiveLayerLossMatryoshka2dLossquantize_embeddingssentence_transformers.sentence_transformer.lossestorch.compilenn.Module.__call__

⚡ Deprecations

  • Loading models whose modules import classes from outside `sentence_transformers` will require `trust_remote_code=True` from v6.0.