●Change8

v3.14.0-rc.0

Breaking Changes
📦 prometheusView on GitHub →
⚠ 3 breaking✨ 4 features🐛 21 fixes⚡ 1 deprecations🔧 61 symbols

Summary

This release introduces new PromQL functions like `start_timestamp`, enhances service discovery with OCI support, and includes numerous performance improvements and bug fixes across TSDB, scraping, and remote write functionalities. It also deprecates certain `stats` query parameter values for future removal.

⚠️ Breaking Changes

  • The `stats` query parameter of `/api/v1/query` and `/api/v1/query_range` will reject values other than `true` and `all` in the next major release. Currently, other values still enable basic statistics but return a deprecation warning.
  • The `promql-duration-expr` feature flag is now a no-op, meaning duration expressions are enabled by default.
  • The `promql-experimental-functions` feature flag is no longer required for `first_over_time` as it is now stable.

Migration Steps

  1. If using the `stats` query parameter with values other than `true` or `all`, update to use `true` or `all` to avoid errors in the next major release.

✨ New Features

  • Added Oracle Cloud Infrastructure compute service discovery (`oci_sd_configs`).
  • Added experimental `start_timestamp(instant-vector)` function to PromQL, returning the start timestamp of each sample in the given vector. Requires the `use-start-timestamps` feature flag.
  • Allow `rate()` and `increase()` in PromQL to use start timestamps for rate extrapolation. Hidden behind the `use-start-timestamps` feature flag.
  • Added experimental support for encoding start timestamps in histograms and float histograms in TSDB. Hidden behind the `histograms-st-encoding` feature flag.

🐛 Bug Fixes

  • Fixed 100% CPU usage on shutdown that could delay graceful shutdown and trigger timeout-based kills.
  • Stopped `promtool check config` from making AWS metadata service (IMDS) network calls when the `region` field is omitted in EC2, ECS, RDS, MSK, ElastiCache, and Lightsail service discovery configs.
  • Set a request timeout for `docker_sd` and `dockerswarm_sd` on `unix`, `npipe`, and `tcp` hosts to prevent indefinite freezing of discovery.
  • Fixed panic in Docker Swarm service discovery when a service runs as a plugin or network-attachment.
  • Fixed discovery of IPv6-only containers.
  • Fixed case-insensitive regex label matchers silently dropping matching values.
  • Fixed `mad_over_time` returning 0 instead of `NaN` when the range contains a `NaN` sample.
  • Promtool `check rules` now accepts `--enable-feature=promql-binop-fill-modifiers`, which previously rejected valid `fill()`/`fill_left()`/`fill_right()` expressions.
  • Remote write now respects the `AZURE_FEDERATED_TOKEN_FILE` environment variable for workload identity authentication instead of hardcoding the token file path.
  • Cleaned up stale `rule_group_last_rule_duration_sum_seconds` and `rule_group_last_restore_duration_seconds` series when a rule group is removed or renamed on reload, preventing cardinality growth.
  • Fixed scrape manager spinning at 100% CPU on shutdown.
  • Fixed silent data loss and potential crash loop when `stale_series_compaction_threshold` is used in the config file.
  • Fixed potential data loss on restart when out-of-order ingestion is enabled and blocks are compacted.
  • Fixed `prometheus_tsdb_head_stale_series` over-counting and early eviction of series that change between float, integer histogram, and float histogram sample types.
  • Fixed goroutine and file handle leaks when Prometheus fails to open a corrupt TSDB. On Windows, the leaked directory handle also prevented TSDB directory removal.
  • Fixed out-of-order queries blocking compaction for hours, causing memory usage to grow.
  • Fixed deleted series causing missing samples and errors after restart.
  • Fixed native histogram data becoming incorrect after restart.
  • Surfaced query errors that were previously silently discarded.
  • Honoured the configured float chunk encoding when compaction rewrites chunks; previously chunks encoded with `--enable-feature=xor2-encoding` could silently revert to XOR after compaction.
  • Show the delete-series form on the TSDB Status page when `--web.enable-admin-api` is enabled.

Affected Symbols

/api/v1/query/api/v1/query_range/api/v1/status/confighcloudpromql-duration-exprfirst_over_timepromql-experimental-functionsstart_timestamp()rate()increase()histograms-st-encodingprometheus_api_otlp_translation_warnings_totalpromtool push metrics--remote-write.pathremote write V2 protocolprometheus_tsdb_head_native_histogram_seriesprometheus_tsdb_head_native_histogram_bucketsUI PromQL syntax highlightingUI Rules page copy buttonUI Rules page title contrastregex label matchersremote readscrape text formatscrape OpenMetrics formatnative histogram scrape parsingTSDB in-memory chunksAlerting shutdownDiscovery/AWS metadata service callsEC2 service discoveryECS service discoveryRDS service discoveryMSK service discoveryElastiCache service discoveryLightsail service discoverydocker_sddockerswarm_sdDocker Swarm service discoveryIPv6 container discoveryPromQL case-insensitive regex label matchersmad_over_timepromtool check rules--enable-feature=promql-binop-fill-modifiersfill()fill_left()fill_right()AZURE_FEDERATED_TOKEN_FILErule_group_last_rule_duration_sum_secondsrule_group_last_restore_duration_secondsscrape manager shutdownstale_series_compaction_thresholdout-of-order ingestionprometheus_tsdb_head_stale_seriesTSDB corrupt file handlingout-of-order queriesTSDB compactionTSDB deleted seriesTSDB native histogram restartTSDB query errors--enable-feature=xor2-encodingTSDB Status page delete-series form--web.enable-admin-api

⚡ Deprecations

  • The `stats` query parameter of `/api/v1/query` and `/api/v1/query_range` is deprecated for values other than `true` and `all`. These values will be rejected in the next major release.