5.0.0
Breaking Changes📦 datasetsView on GitHub →
⚠ 1 breaking✨ 7 features🐛 10 fixes🔧 4 symbols
Summary
This release introduces significant features including parsing Agent traces, enhanced streaming shuffle using multiple shards, and new batching capabilities. It also adds support for several new data formats like Iceberg, TsFile, 3D Mesh, and CoNLL.
⚠️ Breaking Changes
- The default shuffling mechanism in streaming mode now uses multiple input shards, which is a change from the previous single-shard approach. To revert to the old mechanism, pass `max_buffer_input_shards=1` to `IterableDataset.shuffle()`.
Migration Steps
- If you rely on the exact behavior of shuffling in streaming mode before this release, update calls to `IterableDataset.shuffle()` to include `max_buffer_input_shards=1`.
✨ New Features
- Agent traces messages are now parsed using the optional `teich` dependency, allowing traces from sources like claude_code/pi/codex to be loaded and used for training via the `messages` field.
- Improved shuffling in streaming mode by using multiple input shards for the shuffle buffer, which can be configured via `buffer_size` and `max_buffer_input_shards` in `ds.shuffle()`.
- Added `by_column` argument to `Dataset.batch()` for grouping rows based on values in a specific column, useful for robotics datasets.
- Added support for loading datasets in Apache Iceberg format.
- Added support for loading datasets in TsFile (Apache IoTDB) format.
- Added support for 3D mesh datasets via the `MeshFolder` builder.
- Added support for loading `.conll` / `.conllu` dataset formats (CoNLL-2003 / 2000 / U).
🐛 Bug Fixes
- Fixed storage_options lookup for streaming Lance datasets.
- Added agent trace prompt, sent_at, and count fields.
- Added `num_proc` argument support to `Dataset.to_sql`.
- Fixed Parquet streaming hangs occurring at the end of the script.
- Fixed parquet resharding issues.
- Fixed issues with the parquet columns argument.
- Fixed progress bar exceeding total when using map with `load_from_cache_file=False`.
- Fixed embedding external files when using mesh support (`embed_external_files=True`).
- Fixed iterable skip operation over full Arrow blocks.
- Fixed Json() columns to keep `None` as a true null instead of the string "null".