●Change8

5.0.1

📦 datasetsView on GitHub →
✨ 2 features🐛 29 fixes🔧 49 symbols

Summary

This release focuses on a wide range of bug fixes, including issues with data loading, Arrow processing, and dataset manipulation. It also introduces support for new trace formats and clarifies documentation for dataset creation workflows.

Migration Steps

  1. In image preprocessing documentation, replace AutoFeatureExtractor with AutoImageProcessor.

✨ New Features

  • Support for hermes and droid agent traces.
  • Support batched=True in Dataset.to_dict.

🐛 Bug Fixes

  • Fixed version string in __init__.py.
  • Fixed conda build issues.
  • Fixed JSON loader schema inference for files starting with a UTF-8 BOM.
  • Fixed batch(by_column=...) crashing after shard/shuffle/split.
  • Fixed traces streaming.
  • Fixed CI: commit operation equality and pytest parametrize collection error.
  • Fixed lance authentication.
  • Fixed symlink-following arbitrary file write in archive extraction.
  • Fixed resuming twice resetting the dataloader.
  • Bumped fsspec and simpler wds compression.
  • Validated Arrow IPC record batches.
  • Made dataset fingerprint independent of Arrow chunking.
  • Fixed casting a nullable LargeList to a different inner type.
  • Fixed column drop in Arrow path of axis=1 concatenation.
  • Raised on length mismatch in batched IterableDataset.map.
  • Fixed require_storage_embed recursing into require_storage_cast.
  • Fixed path traversal via metadata file_name in folder-based builders.
  • Fixed hdf5 external files.
  • Removed a bad require_storage test.
  • Ensured files are in the repository.
  • Kept flat numeric columns with nulls numeric in numpy format.
  • Fixed bucket dataset card handling and push metadata accounting.
  • Fixed CSV loader dropping on_bad_lines/encoding_errors on pandas 2.0-2.2.
  • Kept integers on the python read path for fixed-shape ArrayXD columns with nulls.
  • Rebatched arrow source before formatting in IterableDataset.filter to fix resume data loss.
  • Decoded Json() columns in Dataset.to_pandas().
  • Fixed buckets on Windows.
  • Fixed DatasetDict.push_to_hub leaving removed splits in the dataset card.
  • Preserved nullable integer columns in to_json/to_csv/to_sql.

Affected Symbols

__init__.pyJSON loaderbatch(by_column=...)traces streamingCIlance autharchive extractionDataloaderfsspecwds compressionArrow IPC record batchesdataset fingerprintArrow chunkingnullable LargeListcolumn dropArrow pathaxis=1 concatenationIterableDataset.maprequire_storage_embedrequire_storage_castpath traversalmetadata file_namefolder-based buildershdf5 external filesrequire_storage testfiles in repoflat numeric columnsnumpy formatbucket dataset card handlingpush metadata accountingCSV loaderon_bad_linesencoding_errorspandas 2.0-2.2integerspython read pathfixed-shape ArrayXD columnsIterableDataset.filterDataset.to_pandas()Json() columnsbuckets on windowsDatasetDict.push_to_hubdataset cardnullable integer columnsto_jsonto_csvto_sqlIterableDataset push-to-hub examplecreate_dataset tutorial