●Change8

v3.14.0

Breaking Changes
📦 prometheusView on GitHub →
⚠ 3 breaking✨ 4 features🐛 21 fixes⚡ 1 deprecations🔧 28 symbols

Summary

This release introduces new PromQL functions and experimental features for timestamp handling and histograms, alongside numerous performance improvements and bug fixes across discovery, TSDB, and the UI. API changes include deprecating certain `stats` query parameter values and enhancing the `/api/v1/status/config` endpoint.

⚠️ Breaking Changes

  • API: The `stats` query parameter for `/api/v1/query` and `/api/v1/query_range` will reject values other than `true` and `all` in the next major release. Currently, these values return a deprecation warning but still enable basic statistics.
  • Discovery/Hetzner: The `__meta_hetzner_datacenter` label for `hcloud` targets has been removed, as it is no longer provided by the Hetzner Cloud API.
  • PromQL: Duration expressions are now enabled by default. The `promql-duration-expr` feature flag is now a no-op and can be removed from configurations.

Migration Steps

  1. Remove the `promql-duration-expr` feature flag from configurations as duration expressions are now enabled by default.

✨ New Features

  • Discovery: Added Oracle Cloud Infrastructure compute service discovery (`oci_sd_configs`).
  • PromQL: Added experimental `start_timestamp(instant-vector)` function returning the start timestamp of each sample in the given vector. Requires the `use-start-timestamps` feature flag.
  • PromQL: `rate()` and `increase()` can now use start timestamps for rate extrapolation. This is hidden behind the `use-start-timestamps` feature flag.
  • TSDB: Added experimental support for encoding start timestamps in histograms and float histograms. Hidden behind the `histograms-st-encoding` feature flag.

🐛 Bug Fixes

  • Alerting: Fixed 100% CPU usage on shutdown that could delay graceful shutdown and trigger timeout-based kills.
  • Discovery/AWS: `promtool check config` no longer makes AWS metadata service (IMDS) network calls when the `region` field is omitted in EC2, ECS, RDS, MSK, ElastiCache, and Lightsail service discovery configs.
  • Discovery/Docker: Set a request timeout for `docker_sd` and `dockerswarm_sd` on `unix`, `npipe`, and `tcp` hosts to prevent indefinite freezing of discovery.
  • Discovery/Docker: Fixed panic in Docker Swarm service discovery when a service runs as a plugin or network-attachment.
  • Discovery/Docker: Fixed discovery of IPv6-only containers.
  • PromQL: Fixed case-insensitive regex label matchers silently dropping matching values.
  • PromQL: Fixed `mad_over_time` returning 0 instead of `NaN` when the range contains a `NaN` sample.
  • Promtool: `check rules` now accepts `--enable-feature=promql-binop-fill-modifiers`, resolving issues with `fill()`/`fill_left()`/`fill_right()` expressions.
  • Remote write: Respects the `AZURE_FEDERATED_TOKEN_FILE` environment variable for workload identity authentication.
  • Rules: Cleaned up stale `rule_group_last_rule_duration_sum_seconds` and `rule_group_last_restore_duration_seconds` series when a rule group is removed or renamed on reload, preventing cardinality growth.
  • Scrape: Fixed scrape manager spinning at 100% CPU on shutdown.
  • TSDB: Fixed silent data loss and potential crash loop when `stale_series_compaction_threshold` is used.
  • TSDB: Fixed potential data loss on restart when out-of-order ingestion is enabled and blocks are compacted.
  • TSDB: Fixed `prometheus_tsdb_head_stale_series` over-counting and early eviction of series that change between float, integer histogram, and float histogram sample types.
  • TSDB: Fixed goroutine and file handle leaks when Prometheus fails to open a corrupt TSDB. On Windows, the leaked directory handle also prevented TSDB directory removal.
  • TSDB: Fixed out-of-order queries blocking compaction for hours, preventing memory usage growth.
  • TSDB: Fixed deleted series causing missing samples and errors after restart.
  • TSDB: Fixed native histogram data becoming incorrect after restart.
  • TSDB: Surfaced query errors that were previously silently discarded.
  • TSDB: Ensures configured float chunk encoding is honored when compaction rewrites chunks; chunks encoded with `--enable-feature=xor2-encoding` no longer silently revert to XOR after compaction.
  • UI: Shows the delete-series form on the TSDB Status page when `--web.enable-admin-api` is enabled.

Affected Symbols

/api/v1/query/api/v1/query_rangehcloudpromql-duration-exprfirst_over_timepromql-experimental-functionsstart_timestamp()use-start-timestampsrate()increase()histograms-st-encodingprometheus_api_otlp_translation_warnings_totalpromtool push metrics --remote-write.pathprometheus_tsdb_head_native_histogram_seriesprometheus_tsdb_head_native_histogram_bucketspromql duration expressionsstep()range()min_of()max_of()prometheus_tsdb_head_stale_seriesprometheus_tsdb_head_stale_seriesAZURE_FEDERATED_TOKEN_FILErule_group_last_rule_duration_sum_secondsrule_group_last_restore_duration_secondsstale_series_compaction_threshold--enable-feature=xor2-encoding--web.enable-admin-api

⚡ Deprecations

  • API: The `stats` query parameter of `/api/v1/query` and `/api/v1/query_range` for values other than `true` and `all` is deprecated and will be rejected in the next major release. Other values still enable basic statistics but now return a deprecation warning.