●Change8

v3.9.0

Breaking Changes
📦 prometheusView on GitHub →
⚠ 1 breaking✨ 10 features🐛 16 fixes🔧 13 symbols

Summary

This release stabilizes Native Histograms, requiring configuration changes for collection, and introduces numerous enhancements across the API, Promtool, UI, and PromQL query engine. Numerous bug fixes address stability and correctness issues, particularly concerning histogram handling and PromQL operations.

⚠️ Breaking Changes

  • Native Histograms are no longer experimental. The feature flag `native-histogram` is now a no-op. Users must enable the configuration setting `scrape_native_histograms` to collect Native Histogram samples from exporters.

Migration Steps

  1. If you were using Native Histograms, you must now enable the configuration setting `scrape_native_histograms` in your configuration file to collect samples, as the `native-histogram` feature flag is ignored.
  2. Review PromQL queries that might have been affected by fixes related to delayed name removal or OR expressions.

✨ New Features

  • API: Added /api/v1/features endpoint for clients to determine supported features.
  • Promtool: Added `start_timestamp` field for unit tests.
  • Promtool: Added `--format seriesjson` option to `tsdb dump` to output only series labels in JSON format.
  • Added `--storage.tsdb.delay-compact-file.path` flag for improved interoperability with Thanos.
  • UI: Added an option on the query drop-down menu to duplicate a query panel.
  • PromQL, Rules, SD, Scraping: Native histograms are now fully supported to complement existing summaries.
  • Notifications: Added histogram `prometheus_notifications_latency_histogram_seconds` to complement the existing summary.
  • Remote-write: Added custom scope support for AzureAD authentication.
  • SD: Added a `config` label with the job name for most `prometheus_sd_refresh` metrics.
  • TSDB: New histogram `prometheus_tsdb_sample_ooo_delta` tracks the distribution of out-of-order samples in seconds.

🐛 Bug Fixes

  • UI: PromQL autocomplete now correctly shows the type and HELP text for OpenMetrics counters ending in `_total`.
  • UI: Fixed codemirror-promql incorrectly suggesting label completions after the closing curly brace of a vector selector.
  • UI: Query editor no longer suggests a duration unit if one is already present after a number.
  • PromQL: Fixed some "vector cannot contain metrics with the same labelset" errors when experimental delayed name removal is enabled.
  • PromQL: Fixed possible corruption of PromQL text if the query had an empty `ignoring()` and non-empty grouping.
  • PromQL: Fixed resets/changes to return empty results for anchored selectors when all samples are outside the range.
  • PromQL: Checked more consistently for many-to-one matching in filter binary operators.
  • PromQL: Fixed collision in unary negation with non-overlapping series.
  • PromQL: Fixed collision in label_join and label_replace with non-overlapping series.
  • PromQL: Fixed bug with inconsistent results for queries with OR expression when experimental delayed name removal is enabled.
  • PromQL: Ensured that `rate`/`increase`/`delta` of histograms results in a gauge histogram.
  • PromQL: Avoided panic while iterating over invalid histograms.
  • TSDB: Rejected chunk files whose encoded chunk length overflows int.
  • TSDB: Avoided panic during resolution reduction of invalid histograms.
  • Remote-write Receive: Avoided duplicate labels when the experimental type-and-unit-label feature is enabled.
  • OTLP Receiver: Only writes metadata to disk when the experimental metadata-wal-records feature is enabled.

🔧 Affected Symbols

scrape_native_histogramsnative-histogram/api/v1/featurestsdb dump--storage.tsdb.delay-compact-file.pathprometheus_notifications_latency_histogram_secondsprometheus_sd_refresh metricsprometheus_tsdb_sample_ooo_deltarate()increase()delta()label_join()label_replace()