●Change8

v1.31.17

📦 weaviateView on GitHub →
✨ 10 features🐛 33 fixes🔧 27 symbols

Summary

This release focuses heavily on stability, particularly around RAFT consensus, replication, and LSM storage performance, introducing numerous bug fixes and significant metric enhancements. Key changes include improved handling of failed broadcasts, better RAFT memberlist configuration, and the addition of extensive new metrics across LSM components.

Migration Steps

  1. If using generative-cohere or generative+rerank modules, note that model validation has been removed.
  2. If you rely on specific metrics, note that `weaviate_lsm_bucket_cursor_duration_seconds` related metrics have been removed.
  3. If you are migrating vector indexes, follow steps related to legacy and named vectors migration: mark migration as done when both legacy and named vectors are defined, and handle situations where you start with a legacy vector index and then add a named vector.

✨ New Features

  • feat(metrics): add new compaction metrics
  • feat(modules): add support for images in generative-cohere module
  • feat: :loud_sound: add 1% sampled queries to slow log
  • feat(metrics): lsm wal recovery metrics
  • feat(metrics): memtable flushing metrics
  • feat bucket lifecycle metrics
  • feat(metrics): segment metrics
  • feat(metrics): lsm cursor metrics
  • feat(metrics): bucket read/write ops metrics
  • feat(debug): add debug endpoints for shard and lock status monitoring

🐛 Bug Fixes

  • fix(replication-broadcast): communicate failed broadcast to commit to avoid lost writes
  • fix(raft-transport): avoid invalid ports to avoid rollout errors and tweak memberlist config
  • fix(raft-rpc): client polic retry and unit test
  • fix(raft): avoid raft node discovery for local node
  • fix(modules): remove model validation in generative-cohere module
  • fix(raft): pass local node name and address to store to use it in discovery
  • Fix appending to broken .wal files
  • fix: context-aware propagation triggering
  • fix: ensure proper unlocking of asyncReplicationRWMux in initHashtree
  • fix: increase IdLockPoolSize and improve uuidToIdLockPoolId calculation
  • Fix case of a missing user prefix
  • Avoid collisions on compressed named vector buckets
  • hnsw: prevent race between UpdateUserConfig and ShouldUpgrade
  • Wait for cache prefill before async indexing can trigger an upgrade
  • fix: panic caused by dropping store before queues
  • Requantize api should skip uncompressed vectors
  • Add info PQ compression log if fails
  • ensure localResults has enough space
  • fix TestTotalDimensionTrackingMetrics race
  • Fix fetchObject with certainty for multi-named-vectors
  • Ensure error returned by `getOrInitShard` is passed to the error response
  • Add more information to compression logs
  • Add buffer queue to the replicated indices handler
  • skip large ids when little-endian and big-endian ids are mixed in the same bucket
  • use correct boundary for mix endianness detection
  • fix: remove all possible weaviate_lsm_bucket_cursor_duration_seconds related metrics
  • fix: don't cause cluster crash when runtime config is malformed
  • Remove model validation from generative+rerank modules
  • fix(migration): create symlink to vectors_compressed folder after migration
  • fix(geo): merge object with geo properties
  • hnsw: ensure candidate lock is released if the code panics
  • Block backup mutex per shard
  • fix(migration): use relative path for vectors_compressed folder when creating a symlink

🔧 Affected Symbols

replication-broadcastraft-transportraft-rpcraftgenerative-cohere modulecycle managerasyncReplicationRWMuxMEMBERLIST_FAST_FAILURE_DETECTIONRoaringSetIdLockPoolSizeuuidToIdLockPoolIdHNSWUpdateUserConfigShouldUpgradegetOrInitShardphonenumbersfetchObjectlocalResultsTestTotalDimensionTrackingMetricsshardlock status monitoringlsm bucket cursor metricruntime configgenerative+rerank modulesvectors_compressed foldergeo propertiesbackup mutex