rs-0.55.1
📦 polarsView on GitHub →
✨ 26 features🐛 60 fixes⚡ 1 deprecations🔧 115 symbols
Summary
This release introduces significant performance improvements across various operations, including pushdown optimizations, enhanced join strategies, and more efficient memory handling. It also brings a host of new features like `struct.drop()`, improved CSV scanning, and experimental out-of-core spilling, alongside numerous bug fixes for enhanced stability and correctness.
✨ New Features
- Added `struct.drop()`.
- Added `infer_schema_files` parameter to `scan_csv`.
- Added support for reading IEEE 754 total order Parquet column order.
- Exposed more Expr nodes for cudf_polars.
- Added callback sinks on cloud.
- Added pre-partition group-by on hive keys.
- Added `ewm_sum` and `ewm_sum_by`.
- Serve stale records from object_store DNS cache.
- Exposed more expr nodes for cudf-polars.
- Made `partition_hive` an opt flag.
- Added Clone to SpillFrame/Token.
- Added Series.degrees/radians and clarified trig functions use radians.
- Added `list` expression which consistently packs elements together into new List type.
- Constructed pyarrow predicates via PyO3 instead of eval str.
- Implemented missing datetime options for `CastColumnsPolicy`.
- Added `Expr.cat.to` and `Expr.cat.physical`.
- Added `POLARS_OOC_DISK_BUDGET_MB` environment variable.
- Added naive out-of-core spilling to Polars.
- Added experimental strict mode.
- Supported SQL's implicit `JOIN` syntax.
- Added `Expr.is_sorted`.
- Added suggestions in wrong api usage or mismatched column names.
- Added external object_store support for non-native schemes.
- Skipped Delta files and parquet row groups using per-field struct statistics.
- Added re-registration of OOC SpillTokens.
- Added `DataFrame.is_sorted()`.
🐛 Bug Fixes
- Fixed IO plugins receiving predicates they cannot evaluate.
- Fixed hanging or crashing on `sqrt`/`cbrt`/`pct_change`/`ewm_*` with a Struct column.
- Raised on overflow in Decimal sum instead of silently wrapping.
- Preserved nulls when importing Arrow maps.
- Propagated null `by` column values in `rolling_*_by`.
- Fixed self-referencing `field` in `struct.with_fields` with `over`.
- Fixed Arrow buffer offset for `Utf8` and `Binary`.
- Clamped group-by slice offset.
- Ensured categorical `fill_null(strategy="min"/"max")` uses lexical min/max.
- Released the GIL while resolving the schema in `__arrow_c_stream__`.
- Fixed data corruption on Series from nested Arrow `LargeList`.
- Properly propagated enum metadata for iceberg column mapping.
- Correctly wrote parquet field IDs for enums and categoricals.
- Fixed arrow C interop binview offset.
- Fixed duplicate hive values in hive pre-partition rewrite.
- Corrected double-counted offset for Arrow export of sliced `Series(Array)`.
- Fixed Struct/List/Array equality handling in sort and join.
- Used try_new in StructArray construction in polars-json.
- Raised ComputeError instead of panicking in repeat_by when output exceeds IdxSize::MAX.
- Ran type coercion pass on pivot's internally generated group_by.
- Improved slice pushdown into joins.
- Preserved ordering in sliced unions.
- Dropped input sortedness when casting to a string.
- Cleared sortedness flags in `StringChunked` substring kernels.
- Fixed bad mask handling when reading optional parquet column.
- Ensured any operation on `Unknown(Int)` and `Unknown(Float)` results in `Unknown(Float)`.
- Flipped `nulls_last` after `Expr.reverse()`.
- Fixed high blocking thread use in sink_parquet with async local path.
- Serialized LazyFrames backed by bytes.
- Released GIL in `SQLContext.execute()`.
- Propagated `nulls_last` in `function_expr_sortedness`.
- Fixed incorrect slicing when a join requires sorting.
- Fixed panic on self join of scan_delta / scan_iceberg.
- Decremented ParkGroup worker count when worker exits.
- Added missing Send bounds in Connector.
- Fixed UB on first/last_non_null with empty chunk.
- Did not CSE non-column height expr on streaming engine.
- Propagated adjusted sortedness through Gather.
- Deduplicated the expression equality checks.
- Addressed 3VL issue with SQL `NOT IN` interaction with `NULL` values and joins.
- Used total comparison in Expr literal values.
- Ensured SQL `SUM` and `CORR` aggregates return NULL for all-null inputs, added `TOTAL`.
- Shared `null_count_dtype` helper between Delta and Iceberg, fixing `SchemaError`.
- Removed non-output columns from the equi-join and semi/anti-join operators.
- Fixed dropped slice on multiple unions.
- Fixed in-memory engine incorrect slice on maintain order join.
- Checked join schema by position.
- Blocked predicate pushdown past overwritten window keys.
- Avoided panic when union slice skips all rows.
- Solved panic in `dt.replace` when there were multiple chunks.
- Propagated `is_scalar` from the input to the output of `.sort()` and `.sort_by()`.
- Inserted missing coercions from `Unknown(_)` in list/array arithmetic.
- Resolved CSV column names overwrite in DSL->IR conversion.
- Avoided IEJoin rewrite for Categorical comparisons.
- Did not rewrite `sort().reverse()` to `sort(descending=True)` when `maintain_order=True`.
- Propagated bigidx to `polars-plan` from `polars`.
- Ensured BinaryView offset+len does not exceed i32::MAX where possible.
- Fixed invalid offset in strptime.
- Fixed incorrect schema type for decimal <-> primitive division.
- Fixed panic in in-memory CSEE handling.
Affected Symbols
struct.drop()scan_csvParquet column ordercudf_polarsSinkTypeIR::Partitionedobject_store DNS cacheExpr nodespartition_hiveIRExprsassert_frame_equalSpillFrameTokenSeries.degreesSeries.radianstrig functionslist expressionpyarrow predicatesCastColumnsPolicyExpr.cat.toExpr.cat.physicalPOLARS_OOC_DISK_BUDGET_MBDatabase testsIRFunctionExpr::Fusedpython node visitorcasts from string to temporal dtypesout-of-core spillingstrict modelen()null_count comparisonsSQL JOIN syntaxThompson samplingExpr.is_sortedDataFrame.is_sorted()Delta filesparquet row groupsOOC SpillTokenswith_row_indexexplodesqrt()cbrt()pct_change()ewm_*()Struct columnDecimal sumArrow mapsrolling_*_bystruct.with_fieldsUtf8Binarygroup-by slicecategorical fill_null__arrow_c_stream__Arrow LargeListiceberg column mappingenumscategoricalsarrow C interop binviewhive pre-partition rewriteSeries(Array)StructArraypolars-jsonrepeat_bypivotjoinsunionsStringChunkedoptional parquet columnUnknown(Int)Unknown(Float)Expr.reverse()sink_parquetasync local pathLazyFramesSQLContext.execute()function_expr_sortednessscan_deltascan_icebergParkGroup workerConnectorfirst_non_nulllast_non_nullstreaming engineGatherexpression equality checksSQL NOT INExpr literal valuesSQL SUMSQL CORRnull_count_dtypeDeltaIcebergequi-joinsemi/anti-join operatorsmultiple unionsmaintain order joinjoin schemawindow keysunion slicedt.replacesort()sort_by()list arithmeticarray arithmeticCSV column namesDSL->IR conversionCategorical comparisonssort().reverse()sort(descending=True)maintain_order=Truepolars-planpolarsBinaryView offset+lenstrptimedecimal <-> primitive divisionin-memory CSEE handling
⚡ Deprecations
- Casts from string to temporal dtypes are deprecated.