●Change8

v3.7.0

Breaking Changes
📦 prometheusView on GitHub →
⚠ 1 breaking✨ 6 features🐛 18 fixes⚡ 4 deprecations🔧 11 symbols

Summary

This release introduces significant performance improvements across PromQL and TSDB, adds support for new features like native histograms with custom buckets, and fixes numerous bugs, while also warning about a breaking change in OTLP attribute name handling.

⚠️ Breaking Changes

  • OTLP endpoint: OpenTelemetry attribute names starting with underscore (e.g., `_attrib`) are no longer automatically translated to `key_attrib`. Users relying on this automatic translation must update attribute names or use version 3.7.1 for OTLP ingestion.

Migration Steps

  1. If you rely on automatic translation of OpenTelemetry attribute names starting with an underscore in the OTLP endpoint, update your attribute names or use version 3.7.1 for OTLP ingestion.

✨ New Features

  • PromQL: Add support for experimental anchored and smoothed rate behind feature flag `promql-extended-range-selectors`.
  • Federation: Add support for native histograms with custom buckets (NHCB).
  • PromQL: Add `first_over_time(...)` and `ts_of_first_over_time(...)` behind feature flag `experimental-promql-functions`.
  • Remote-write: Add support for Azure Workload Identity as an authentication method for the receiver.
  • Remote-write: Add type and unit labels to outgoing time series in remote-write 2.0 when the `type-and-unit-labels` feature flag is enabled.
  • OTLP: Write start time of metrics as created time zero samples into TSDB when `created-timestamp-zero-ingestion` feature flag is enabled.

🐛 Bug Fixes

  • Alerting: Mutating alerts relabeling (using `replace` actions, etc.) within a `alertmanager_config.alert_relabel_configs` block is now scoped correctly and no longer yields altered alerts to subsequent blocks.
  • Config: Infer valid escaping scheme when scrape config validation scheme is set.
  • TSDB: Correctly handle appending mixed-typed samples to the same series.
  • Remote-write: Prevent sending unsupported native histograms with custom buckets (NHCB) over Remote-write 1.0, log warning.
  • TSDB: Fix metadata entries handling on `metadata-wal-records` experimental feature for native histograms with custom buckets (NHCB) in protobuf scraping.
  • TSDB: Ignore Native Histograms with invalid schemas during WAL/WBL replay.
  • PromQL: Avoid empty metric names in annotations for `histogram_quantile()`.
  • PromQL: Correct inaccurate character positions in errors for some aggregate expressions.
  • PromQL: Fix `info()` function on churning series.
  • PromQL: Set native histogram to gauge type when subtracting or multiplying/dividing with negative factors.
  • TSDB: Reject unsupported native histogram schemas when attempting to append to TSDB. For scrape and remote-write implement reducing the resolution to fit the maximum if the schema is within the -9 to 52.
  • Remote-write: Fix HTTP handler to return after writing error response for invalid compression.
  • Remote-write: Return HTTP error `400` instead of `5xx` for wrongly formatted Native Histograms.
  • Scrape: Prevent staleness markers from generating unnecessary series.
  • TSDB: Avoid misleading `Failed to calculate size of "wal" dir` error logs during WAL clean-up.
  • TSDB: Prevent erroneously dropping series records during WAL checkpoints.
  • UI: Fix redirect to path of `-web.external-url` if `-web.route-prefix` is configured.
  • Remote-write: Do not panic on invalid symbol table in remote-write 2.0.

🔧 Affected Symbols

prometheus_remote_storage_samples_in_totalprometheus_wal_watcher_records_read_totalprometheus_remote_storage_samples_dropped_totalprometheus_remote_storage_histograms_in_totalprometheus_remote_storage_histograms_dropped_totalprometheus_remote_storage_exemplars_in_totalprometheus_remote_storage_exemplars_dropped_totalprometheus_remote_storage_highest_timestamp_in_secondsprometheus_remote_storage_queue_highest_timestamp_secondsALERTSALERTS_FOR_STATE

⚡ Deprecations

  • Metric `prometheus_remote_storage_samples_in_total`: Use `prometheus_wal_watcher_records_read_total{type="samples"}` and `prometheus_remote_storage_samples_dropped_total` instead.
  • Metric `prometheus_remote_storage_histograms_in_total`: Use `prometheus_wal_watcher_records_read_total{type=~"\.\*histogram_samples"}` and `prometheus_remote_storage_histograms_dropped_total` instead.
  • Metric `prometheus_remote_storage_exemplars_in_total`: Use `prometheus_wal_watcher_records_read_total{type="exemplars"}` and `prometheus_remote_storage_exemplars_dropped_total` instead.
  • Metric `prometheus_remote_storage_highest_timestamp_in_seconds`: Use the more accurate `prometheus_remote_storage_queue_highest_timestamp_seconds` instead in dashboards and alerts.