Change8

rs-0.55.1

📦 polarsView on GitHub →
26 features🐛 60 fixes1 deprecations🔧 115 symbols

Summary

This release introduces significant performance improvements across various operations, including pushdown optimizations, enhanced join strategies, and more efficient memory handling. It also brings a host of new features like `struct.drop()`, improved CSV scanning, and experimental out-of-core spilling, alongside numerous bug fixes for enhanced stability and correctness.

✨ New Features

  • Added `struct.drop()`.
  • Added `infer_schema_files` parameter to `scan_csv`.
  • Added support for reading IEEE 754 total order Parquet column order.
  • Exposed more Expr nodes for cudf_polars.
  • Added callback sinks on cloud.
  • Added pre-partition group-by on hive keys.
  • Added `ewm_sum` and `ewm_sum_by`.
  • Serve stale records from object_store DNS cache.
  • Exposed more expr nodes for cudf-polars.
  • Made `partition_hive` an opt flag.
  • Added Clone to SpillFrame/Token.
  • Added Series.degrees/radians and clarified trig functions use radians.
  • Added `list` expression which consistently packs elements together into new List type.
  • Constructed pyarrow predicates via PyO3 instead of eval str.
  • Implemented missing datetime options for `CastColumnsPolicy`.
  • Added `Expr.cat.to` and `Expr.cat.physical`.
  • Added `POLARS_OOC_DISK_BUDGET_MB` environment variable.
  • Added naive out-of-core spilling to Polars.
  • Added experimental strict mode.
  • Supported SQL's implicit `JOIN` syntax.
  • Added `Expr.is_sorted`.
  • Added suggestions in wrong api usage or mismatched column names.
  • Added external object_store support for non-native schemes.
  • Skipped Delta files and parquet row groups using per-field struct statistics.
  • Added re-registration of OOC SpillTokens.
  • Added `DataFrame.is_sorted()`.

🐛 Bug Fixes

  • Fixed IO plugins receiving predicates they cannot evaluate.
  • Fixed hanging or crashing on `sqrt`/`cbrt`/`pct_change`/`ewm_*` with a Struct column.
  • Raised on overflow in Decimal sum instead of silently wrapping.
  • Preserved nulls when importing Arrow maps.
  • Propagated null `by` column values in `rolling_*_by`.
  • Fixed self-referencing `field` in `struct.with_fields` with `over`.
  • Fixed Arrow buffer offset for `Utf8` and `Binary`.
  • Clamped group-by slice offset.
  • Ensured categorical `fill_null(strategy="min"/"max")` uses lexical min/max.
  • Released the GIL while resolving the schema in `__arrow_c_stream__`.
  • Fixed data corruption on Series from nested Arrow `LargeList`.
  • Properly propagated enum metadata for iceberg column mapping.
  • Correctly wrote parquet field IDs for enums and categoricals.
  • Fixed arrow C interop binview offset.
  • Fixed duplicate hive values in hive pre-partition rewrite.
  • Corrected double-counted offset for Arrow export of sliced `Series(Array)`.
  • Fixed Struct/List/Array equality handling in sort and join.
  • Used try_new in StructArray construction in polars-json.
  • Raised ComputeError instead of panicking in repeat_by when output exceeds IdxSize::MAX.
  • Ran type coercion pass on pivot's internally generated group_by.
  • Improved slice pushdown into joins.
  • Preserved ordering in sliced unions.
  • Dropped input sortedness when casting to a string.
  • Cleared sortedness flags in `StringChunked` substring kernels.
  • Fixed bad mask handling when reading optional parquet column.
  • Ensured any operation on `Unknown(Int)` and `Unknown(Float)` results in `Unknown(Float)`.
  • Flipped `nulls_last` after `Expr.reverse()`.
  • Fixed high blocking thread use in sink_parquet with async local path.
  • Serialized LazyFrames backed by bytes.
  • Released GIL in `SQLContext.execute()`.
  • Propagated `nulls_last` in `function_expr_sortedness`.
  • Fixed incorrect slicing when a join requires sorting.
  • Fixed panic on self join of scan_delta / scan_iceberg.
  • Decremented ParkGroup worker count when worker exits.
  • Added missing Send bounds in Connector.
  • Fixed UB on first/last_non_null with empty chunk.
  • Did not CSE non-column height expr on streaming engine.
  • Propagated adjusted sortedness through Gather.
  • Deduplicated the expression equality checks.
  • Addressed 3VL issue with SQL `NOT IN` interaction with `NULL` values and joins.
  • Used total comparison in Expr literal values.
  • Ensured SQL `SUM` and `CORR` aggregates return NULL for all-null inputs, added `TOTAL`.
  • Shared `null_count_dtype` helper between Delta and Iceberg, fixing `SchemaError`.
  • Removed non-output columns from the equi-join and semi/anti-join operators.
  • Fixed dropped slice on multiple unions.
  • Fixed in-memory engine incorrect slice on maintain order join.
  • Checked join schema by position.
  • Blocked predicate pushdown past overwritten window keys.
  • Avoided panic when union slice skips all rows.
  • Solved panic in `dt.replace` when there were multiple chunks.
  • Propagated `is_scalar` from the input to the output of `.sort()` and `.sort_by()`.
  • Inserted missing coercions from `Unknown(_)` in list/array arithmetic.
  • Resolved CSV column names overwrite in DSL->IR conversion.
  • Avoided IEJoin rewrite for Categorical comparisons.
  • Did not rewrite `sort().reverse()` to `sort(descending=True)` when `maintain_order=True`.
  • Propagated bigidx to `polars-plan` from `polars`.
  • Ensured BinaryView offset+len does not exceed i32::MAX where possible.
  • Fixed invalid offset in strptime.
  • Fixed incorrect schema type for decimal <-> primitive division.
  • Fixed panic in in-memory CSEE handling.

Affected Symbols

struct.drop()scan_csvParquet column ordercudf_polarsSinkTypeIR::Partitionedobject_store DNS cacheExpr nodespartition_hiveIRExprsassert_frame_equalSpillFrameTokenSeries.degreesSeries.radianstrig functionslist expressionpyarrow predicatesCastColumnsPolicyExpr.cat.toExpr.cat.physicalPOLARS_OOC_DISK_BUDGET_MBDatabase testsIRFunctionExpr::Fusedpython node visitorcasts from string to temporal dtypesout-of-core spillingstrict modelen()null_count comparisonsSQL JOIN syntaxThompson samplingExpr.is_sortedDataFrame.is_sorted()Delta filesparquet row groupsOOC SpillTokenswith_row_indexexplodesqrt()cbrt()pct_change()ewm_*()Struct columnDecimal sumArrow mapsrolling_*_bystruct.with_fieldsUtf8Binarygroup-by slicecategorical fill_null__arrow_c_stream__Arrow LargeListiceberg column mappingenumscategoricalsarrow C interop binviewhive pre-partition rewriteSeries(Array)StructArraypolars-jsonrepeat_bypivotjoinsunionsStringChunkedoptional parquet columnUnknown(Int)Unknown(Float)Expr.reverse()sink_parquetasync local pathLazyFramesSQLContext.execute()function_expr_sortednessscan_deltascan_icebergParkGroup workerConnectorfirst_non_nulllast_non_nullstreaming engineGatherexpression equality checksSQL NOT INExpr literal valuesSQL SUMSQL CORRnull_count_dtypeDeltaIcebergequi-joinsemi/anti-join operatorsmultiple unionsmaintain order joinjoin schemawindow keysunion slicedt.replacesort()sort_by()list arithmeticarray arithmeticCSV column namesDSL->IR conversionCategorical comparisonssort().reverse()sort(descending=True)maintain_order=Truepolars-planpolarsBinaryView offset+lenstrptimedecimal <-> primitive divisionin-memory CSEE handling

⚡ Deprecations

  • Casts from string to temporal dtypes are deprecated.