●Change8

v3.7.0-rc.0

📦 prometheus
✨ 6 features🐛 18 fixes⚡ 1 deprecations🔧 11 symbols

Summary

This release introduces significant performance improvements across PromQL and TSDB, alongside new features like native histogram support in Federation and enhanced remote-write capabilities. Several legacy remote storage metrics have been deprecated in favor of more accurate replacements.

Migration Steps

  1. If you rely on the deprecated remote storage metrics, update your dashboards and alerts to use `prometheus_remote_storage_queue_highest_timestamp_seconds` instead of `prometheus_remote_storage_highest_timestamp_in_seconds`.

✨ New Features

  • Added support for experimental anchored and smoothed rate in PromQL behind the feature flag `promql-extended-range-selectors`.
  • Added support for native histograms with custom buckets (NHCB) in Federation.
  • Added `first_over_time(...)` and `ts_of_first_over_time(...)` functions to PromQL behind the feature flag `experimental-promql-functions`.
  • Added support for Azure Workload Identity as an authentication method for the remote-write receiver.
  • Added type and unit labels to outgoing time series in remote-write 2.0 when the `type-and-unit-labels` feature flag is enabled.
  • When `created-timestamp-zero-ingestion` feature flag is enabled, OTLP writes the start time of metrics as created time zero samples into TSDB.

🐛 Bug Fixes

  • Mutating alerts relabeling (using `replace` actions, etc.) within a `alertmanager_config.alert_relabel_configs` block is now scoped correctly and no longer yields altered alerts to subsequent blocks.
  • Configuration validation now correctly infers a valid escaping scheme when the scrape config validation scheme is set.
  • Correctly handle appending mixed-typed samples to the same series in TSDB.
  • Prevent sending unsupported native histograms with custom buckets (NHCB) over Remote-write 1.0; a warning is logged instead.
  • Fixed metadata entries handling on `metadata-wal-records` feature for Native Histograms with custom buckets in protobuf scraping.
  • Ignore Native Histograms with invalid schemas during WAL/WBL replay in TSDB.
  • Avoided empty metric names in annotations for `histogram_quantile()`.
  • Fixed the character position reported for errors in some aggregate PromQL expressions.
  • Fixed the `info()` function behavior on churning series.
  • Native histogram type is now correctly set to gauge when subtracting or multiplying/dividing with negative factors.
  • TSDB now rejects unsupported native histogram schemas when attempting to append; for scrape and remote-write, resolution is reduced to fit the maximum if the schema is within -9 to 52.
  • Fixed the remote-write HTTP handler to return after writing an error response for invalid compression.
  • Remote-write now returns HTTP error `400` instead of `5xx` for wrongly formatted Native Histograms.
  • Prevented staleness markers from generating unnecessary series during scraping.
  • Avoided misleading `Failed to calculate size of "wal" dir` error logs during WAL clean-up.
  • Prevented erroneously dropping series records during WAL checkpoints.
  • Fixed UI redirect to path of `-web.external-url` when `-web.route-prefix` is configured.
  • Remote-write 2.0 no longer panics on invalid symbol tables.

🔧 Affected Symbols

prometheus_remote_storage_samples_in_totalprometheus_remote_storage_exemplars_in_totalprometheus_remote_storage_histograms_in_totalprometheus_remote_storage_highest_timestamp_in_secondsprometheus_remote_storage_queue_highest_timestamp_secondsalertmanager_config.alert_relabel_configshistogram_quantile()info()ALERTSALERTS_FOR_STATEmetadata-wal-records

⚡ Deprecations

  • The metrics `prometheus_remote_storage_{samples,exemplars,histograms}_in_total` and `prometheus_remote_storage_highest_timestamp_in_seconds` are deprecated. Refer to their descriptions for alternatives, specifically `prometheus_remote_storage_queue_highest_timestamp_seconds` for the latter.