●Change8

v0.38.0

Breaking Changes
📦 lancedbView on GitHub →
⚠ 6 breaking✨ 35 features🐛 21 fixes⚡ 1 deprecations🔧 16 symbols

Summary

This release introduces significant improvements to computed columns, materialized views, and function handling across SDKs. It also includes breaking changes such as requiring pydantic v2 and Node.js v22, alongside various bug fixes and performance enhancements.

⚠️ Breaking Changes

  • Table existence is now manifest-authoritative. This may affect operations that relied on implicit table creation or deletion.
  • Python API now requires pydantic v2. Update your pydantic dependencies.
  • Node.js: Key parsed embedding configurations by vector column. This might break existing configurations that relied on a different parsing method.
  • Renamed 'branch merge' to 'cherry_pick'. Update any code that uses the 'branch merge' terminology.
  • Node.js: Requires Node.js version >= 22 and drops npm lockfiles. Ensure your Node.js environment is updated to v22 or later.
  • Page table listings from the store's own cursor. This change affects how table listings are retrieved, potentially impacting pagination logic.

Migration Steps

  1. Update pydantic to v2 for Python users.
  2. Update Node.js to version 22 or later.
  3. Replace usage of `branch merge` with `cherry_pick`.
  4. Replace usage of `tableNames` with `listTables` in Node.js.

✨ New Features

  • Python: Added `on_transform_error` fault tolerance to `StreamingDataset`.
  • Added asynchronous `drop_table` API.
  • LSM: Require recorded index catch-up as an explicit activation.
  • Declare computed columns by SQL expression.
  • Refresh computed columns.
  • `refresh_column_async` now returns a job handle.
  • Computed columns on remote tables.
  • Python: Expose LSM checkpoint and stats on sync `RemoteTable`.
  • Bring the MemWAL LSM surface to parity across the SDKs.
  • Add first-class function wire contracts.
  • Add grouped function column bindings.
  • Add scalar function authoring and catalog client.
  • Python: Bind function versions to columns.
  • Support remote tables in the data loader.
  • Materialized view declarations on local tables.
  • Refresh materialized views.
  • Python: Materialized view bindings.
  • Python: Add backpressure to `StreamingDataset` post-transform queue.
  • Python: Support sequence packing in streaming dataset.
  • Return typed refresh job results.
  • Bind a materialized view refresh to the view incarnation.
  • Node.js: Added `listTables` API.
  • Pin the base table version for data loader reads.
  • Accept blob URI writes.
  • Declare conda environments on Functions.
  • Expose list-element FTS document granularity.
  • Python: Accept expressions in update filters.
  • Allow a computed-column batch to read its own earlier declarations.
  • Support blob computed column refresh.
  • Node.js: Require Node >= 22 and drop npm lockfiles.
  • Python: Support nested Function Arrow types.
  • Functions: Support GPU resource requirements.
  • Support Blob v2 UDF signatures.
  • Python: Support `large_utf8` function signatures.
  • Update lance dependency to v11.0.0.

🐛 Bug Fixes

  • Python: Set native module metadata.
  • Table existence is now manifest-authoritative.
  • LSM: Gate SSTable exclusion on every index a query relies on.
  • Python: Treat naive lit(datetime) as UTC wall clock.
  • Hybrid search silently ignores `.offset()`.
  • Python: Preserve JSON encoding in merge insert.
  • Python: Set LsmWriteSpec module metadata.
  • Python: Return None from unit jobs.
  • Python: Use canonical remote function endpoints.
  • Accept non-null Function inputs for nullable parameters.
  • Node.js: Key parsed embedding configs by vector column.
  • Remote: Fence refresh submissions after `add_columns`.
  • Python: Skip the unrunnable `FunctionVersion` doctest example.
  • Python: Expose FTS build memory limits.
  • Rust: Reject bitmap indexes on JSON fields.
  • Listing: Do not drop a table at a page boundary.
  • Python: Commit streaming worker checkpoints on consumption.
  • Remote: Stop `table_names` inventing a page token for a namespace.
  • Pin remote snapshots during permutation construction.
  • Node: Validate inferred types across records.
  • Package ordinary @udf bodies and emit only the V1 type grammar.

Affected Symbols

StreamingDatasetpydanticNode.jsbranch mergecherry_pickNode >= 22npm lockfilestable existence manifestvector columnFunctionVersionLsmWriteSpecRemoteTabletable_nameslistTablesUDFBlob v2 UDF

⚡ Deprecations

  • Node.js: `tableNames` API is deprecated in favor of `listTables`.