●Change8

v0.31.0

Breaking Changes
📦 lancedbView on GitHub →
⚠ 2 breaking✨ 28 features🐛 28 fixes⚡ 1 deprecations🔧 19 symbols

Summary

This release introduces significant new features including table branching, Polars integration, and OAuth support, alongside breaking changes related to IndexStatistics and query filter combination logic.

⚠️ Breaking Changes

  • The 'loss' field has been removed from IndexStatistics.
  • Repeated 'where' filters in queries are now combined using AND logic instead of the previous behavior of replacing them.

Migration Steps

  1. If you relied on repeated 'where' filters being replaced, update your query logic as they are now combined with AND.
  2. Replace usage of 'replace_field_metadata' with 'update_field_metadata'.

✨ New Features

  • Unified sync 'create_index' API in Python to match the async API.
  • Routed 'merge_insert' through the MemWAL LSM write path.
  • Support for remote tables in PyTorch dataloaders.
  • Added 'update_field_metadata' to edit per-field metadata.
  • Support for blob modes in query 'to_pandas'.
  • Support for datafusion expressions for merge insert predicates in Rust.
  • Implemented set/unset_lsm_write_spec REST variant for remote operations.
  • Added table branch support.
  • Support for checking out a specific version on a branch.
  • Support for FM-Index scalar index for substring search.
  • Added 'isin' support to the 'Expr' builder.
  • Expanded IndexConfig with rich per-index metadata.
  • Accepting 'Expr' in Table.delete and 'when_not_matched_by_source_delete'.
  • Support for 'rename_table' on LanceNamespaceDatabase.
  • Exposed new IndexConfig fields in Python and Node.js bindings.
  • Added skills to connect and update column metadata.
  • Added table branch support to remote tables and Python/TS bindings.
  • Added approximate mode to vector queries.
  • Sending read-freshness signal on the lance-namespace path.
  • Added blob v2 schema declaration and write path in Rust.
  • Added blob read and materialization APIs in Rust.
  • Added OAuth header provider in Rust.
  • Exposed OAuth connection config in Python.
  • Exposed OAuth connection config in Node.js.
  • Added skill to work with branches better.
  • Added Polars DataFrame integration.
  • Implemented monotonic reads via x-lancedb-min-read-version watermark for remote operations.
  • Re-exported arrow and datafusion crates from lancedb in Rust.

🐛 Bug Fixes

  • Support for blob modes in query to_pandas (also listed as a feature, likely a fix related to its implementation).
  • Raised ValueError on empty vector_results list in rerankers/mrr.
  • Guarded against empty vector_results in RRFReranker.rerank_multivector.
  • Routed blob query pandas through scanner in Python.
  • Made LanceDBClientError pickleable in Python.
  • Ran AsyncTable.search embeddings on a dedicated executor in Python.
  • Pushed down namespace full reads in Python.
  • Converted numpy scalars in value_to_sql.
  • Treated NAPI_RS_FORCE_WASI as truthy only when set to 'true' in Node.js.
  • Supported LargeList label list indexes.
  • Raised clear TypeError for bare List/Tuple in pydantic schema conversion in Python.
  • Reported local write progress bytes from Lance in Rust.
  • Escaped single quotes in struct field names in value_to_sql.
  • Clarified single dictionary input error in Python.
  • Returned typed errors instead of panicking in Bedrock embedding path in Rust.
  • Allowed shell pre-commit hooks in bumpversion configs in CI.
  • Fixed overflow using sys.maxsize for k in query with namespace connection.
  • Parsed RFC 3339 created_at and improved IndexConfig repr.
  • Added missing stacklevel=2 to warnings.warn() and fixed broken message concatenation.
  • Omitted empty api key header in remote operations.
  • Supported namespace clients with dynamic headers in remote operations.
  • Ensured read freshness provider is built into namespace client.
  • Raised clear error when permutation API is used on remote tables in Python.
  • Added missing stacklevel=2 to warnings.warn() calls.
  • Routed sync namespace connections through rust in Python.
  • Routed async namespace connections through rust in Python.
  • Combined repeated where filters with AND instead of replacing (also listed as breaking change).
  • Averaged MRR reciprocal ranks over all rankings in Python.

Affected Symbols

IndexStatisticscreate_indexmerge_insertPyTorch dataloadersupdate_field_metadatareplace_field_metadataquery.to_pandasTable.deletewhen_not_matched_by_source_deleterename_tableLanceNamespaceDatabaseIndexConfigTable.searchvalue_to_sqlRRFReranker.rerank_multivectorLanceDBClientErrorlist_indicesdescribe_indicespermutation API

⚡ Deprecations

  • The 'replace_field_metadata' function is deprecated in favor of 'update_field_metadata'.