●Change8

py-1.42.0

Breaking Changes
📦 polarsView on GitHub →
⚠ 1 breaking✨ 12 features🐛 25 fixes⚡ 1 deprecations🔧 8 symbols

Summary

This release introduces performance improvements across cloud IO and expression evaluation, adds experimental features like out-of-core spilling and strict mode, and fixes numerous bugs related to SQL, projection pushdown, and joins. Notably, it deprecates implicit string-to-temporal casts.

⚠️ Breaking Changes

  • Deprecate casts from string to temporal dtypes. This may break code relying on implicit string-to-datetime/date/time conversions.

Migration Steps

  1. If you relied on implicit string-to-temporal casting, update your code to explicitly cast strings to temporal types.

✨ New Features

  • Added naive out-of-core spilling to Polars.
  • Added experimental strict mode.
  • Support SQL's implicit JOIN syntax.
  • Added Expr.is_sorted.
  • Added suggestions in wrong api usage or mismatched column names.
  • Added external object_store support for non-native schemes.
  • Skip Delta files and row groups using per-field struct statistics.
  • Improved IR sortedness detection for with_row_index and explode.
  • Added DataFrame.is_sorted().
  • Exposed fixed-size rolling window expressions in Python visitor.
  • Added tiered multi-file parquet metadata resolver.
  • Cache and shuffle DNS for cloud object_store.

🐛 Bug Fixes

  • Added a deprecation warning for .explode() without empty_as_null argument.
  • Fix select(len()) incorrectly returned 0 when using scan_iceberg with pyiceberg as reader override.
  • Ensure slice expr always observes input order.
  • Fix projection pushdown with row index out of order.
  • Fix cross-join predicates not applied when using sink_* functions.
  • Grammar corrections (recommend to → recommend, allows to → allows using).
  • Ensure optimisations aren't omitted by cached IR nodes.
  • Computed GROUP keys in SQL projections.
  • Handle empty DataFrames in dt.truncate and dt.round.
  • Address join alias state leakage in SQL CTEs.
  • Support scanning file:// URIs with percent-encoded paths.
  • Lower SQL [NOT] IN (subquery) to semi/anti join.
  • Check for continuity across morsels in AsOf join node.
  • Add test for projection pushdown with hconcat regression.
  • Traverse caches only once in explain.
  • Preserve null placement when collapsing sort.reverse() into single sort.
  • Incorrect schema order for hstack in projection pushdown optimizer.
  • Fix duplicated map_elements warning on streaming engine.
  • Use block_in_place_on for calls which can come from executor thread.
  • Mismatch in max_threads -> pipeline configuration.
  • Keep maintain_order on sliced unique.
  • Fix SchemaError using lazy HConcat->Sink (duplicate entry, noted once).
  • Fix incorrect projection height when selecting only literals.
  • Select with expr slice and len gave incorrect len.
  • Prevent import panic when environment variable set to unexpected value.

Affected Symbols

Expr.is_sortedDataFrame.is_sorted()dt.truncatedt.round.explode()scan_icebergsink_* functionsHConcat

⚡ Deprecations

  • Casts from string to temporal dtypes are deprecated.