●Change8

py-1.36.0-beta.2

📦 polars
✨ 49 features🐛 54 fixes🔧 40 symbols

Summary

This release introduces significant enhancements across SQL support, expression capabilities (like rolling windows and aggregation contexts), and adds support for Extension types. Numerous performance improvements and bug fixes address stability and correctness across various operations, especially in streaming and Parquet handling.

✨ New Features

  • Added Extension types.
  • Added SQL support for ROW_NUMBER, RANK, and DENSE_RANK functions.
  • Added SQL support for named WINDOW references.
  • Added BIT_NOT support to the SQL interface.
  • Added LazyFrame.pivot.
  • Added allow_empty flag to item.
  • Added empty_as_null and keep_nulls flags to Expr.explode.
  • Added empty_as_null and keep_nulls to {Lazy,Data}Frame.explode.
  • Added having to group_by context.
  • Added ignore_nulls to first / last.
  • Added maintain_order to Expr.mode.
  • Added quantile for missing temporals.
  • Added leftmost option to str.replace_many / str.find_many / str.extract_many.
  • Added strict parameter to pl.concat(how='horizontal').
  • Added support for Float16 dtype.
  • Added unstable Schema.to_arrow.
  • Allowed Expr.rolling in aggregation contexts.
  • Allowed Expr.unique on List/Array with non-numeric types.
  • Allowed glimpse to return a DataFrame.
  • Allowed hash for all List dtypes.
  • Allowed implode and aggregation in aggregation context.
  • Allowed slice on scalar in aggregation context.
  • Allowed arbitrary Expressions in "subset" parameter of unique frame method.
  • Allowed arbitrary expressions as the Expr.rolling index_column.
  • Allowed bare .row on a single-row DataFrame, equivalent to .item on a single-element DataFrame.
  • Allowed elementwise Expr.over in aggregation context.
  • Allowed pl.Object in pivot value.
  • Automatically Parquet dictionary encode floats.
  • Display function of streaming physical plan map node.
  • Exposed and documented pl.Categories.
  • Exposed fields for generating physical plan visualization data.
  • Extended SQL UNNEST support to handle multiple array expressions.
  • Improved SQL UNNEST behaviour.
  • Improved error message on unsupported SQL subquery comparisons.
  • Made DSL-hash skippable.
  • Minor improvement for as_struct repr.
  • Moved GraphMetrics into StreamingQuery.
  • Raised suitable error on non-integer "n" value for clear.
  • Rewrote IR::Scan to IR::DataFrameScan in expand_datasets when applicable.
  • Set polars/<version> user-agent.
  • Streaming {Expr,LazyFrame}.rolling support.
  • Support BYTE_ARRAY backed Decimals in Parquet.
  • Support ewm_var/std in streaming engine.
  • Support unique_counts for all datatypes.
  • Support additional forms of SQL "CREATE TABLE" statements.
  • Support arbitrary expressions in SQL JOIN constraints.
  • Support column-positional SQL "UNION" operations.
  • Support decimals in search_sorted.
  • Temporal quantile in rolling context.

🐛 Bug Fixes

  • Added .rolling_rank support for temporal types and pl.Boolean.
  • Addressed issues with SQL OVER clause behaviour for window functions.
  • Fixed aggregation with drop_nulls on literal.
  • Allowed Null dtype values in scatter.
  • Allowed broadcast in group_by for ApplyExpr and BinaryExpr.
  • Allowed empty list in sort_by in list.eval context.
  • Allowed for negative time in group_by_dynamic iterator.
  • Always respect return_dtype in map_elements and map_rows.
  • Fixed AnyValue::to_physical for categoricals.
  • Fixed CSV dict overrides applied by name only.
  • Blocked predicate pushdown when group_by key values are changed.
  • Fixed bugs in pl.from_repr with signed exponential floats and line wrapping.
  • Corrected drop_items for scalar input.
  • Corrected eq_missing for struct with nulls.
  • Corrected {first,last}_non_null if there are empty chunks.
  • Corrected handle requested stops in streaming shift.
  • Correctly pruned projected columns in hints.
  • DSL_SCHEMA_HASH should not changed by line endings.
  • Don't push down predicates passed inserted cache nodes.
  • Don't quietly allow unsupported SQL SELECT clauses.
  • Don't trigger DeprecationWarning from SQL "IN" constraints that use subqueries.
  • Enhanced column resolution/tracking through multi-way SQL joins.
  • Ensured SQL table alias resolution checks against CTE aliases on fallback.
  • Ensured out-of-range integers and other edge case values don't give wrong results for index_of.
  • Fixed CSV select(len) off by 1 with comment prefix.
  • Fixed arr.{eval,agg} in aggregation context.
  • Fixed format_str in case of multiple chunks.
  • Fixed groups update on slices with different offsets.
  • Fixed assertion panic on group_by.
  • Fixed building polars-expr without timezones feature.
  • Fixed building polars-mem-engine with the async feature.
  • Fixed building polars-plan with features lazy,concat_str (but no strings).
  • Fixed dictionary replacement error in write_ipc.
  • Fixed expr slice pushdown causing shape error on literals.
  • Fixed field metadata for nested categorical PyCapsule export.
  • Fixed group lengths check in sort_by with AggregatedScalar.
  • Fixed handling Null dtype in ApplyExpr on group_by.
  • Fixed incorrect .list.eval after slicing operations.
  • Fixed incorrect reshape on sliced lists.
  • Fixed length preserving check for eval expressions in streaming engine.
  • Fixed occurence of exact matches of .join_asof(strategy="nearest", allow_exact_matches=False, ...).
  • Fixed off-by-one bug in ColumnPredicates generation for inequalities operating on integer columns.
  • Fixed panic if scan predicate produces 0 length mask.
  • Fixed panic in dt.truncate for invalid duration strings.
  • Fixed panic in is_between support in streaming Parquet predicate push down.
  • Fixed panic when using struct field as join key.
  • Fixed serialization of lazyframes containing huge tables.
  • Fixed single-column CSV header duplication with leading empty lines.
  • Fixed small bug with PyExpr to PyObject conversion.
  • Fixed Group-By aggregation problems caused by AmortSeries.
  • Handled some unusual pl.col.<colname> edge-cases.
  • Fixed incorrect result in aggregated first/last with ignore_nulls.
  • Fixed incorrect results for aggregated {n_,}unique on bools.
  • Inverted drop_nans filtering.

🔧 Affected Symbols

LazyFrame.pivotitemExpr.explodeLazyFrame.explodeDataFrame.explodegroup_byfirstlastExpr.modestr.replace_manystr.find_manystr.extract_manypl.concatFloat16Schema.to_arrowExpr.rollingExpr.uniqueglimpsehashimplodesliceExpr.overpl.Objectrolling_rankdt.truncatepl.from_reprdrop_itemseq_missingscatterApplyExprBinaryExprsort_bylist.evalgroup_by_dynamicmap_elementsmap_rowsAnyValue::to_physicalwrite_ipcjoin_asofColumnPredicates