v3.11.0-rc.0
📦 prometheusView on GitHub →
✨ 12 features🐛 21 fixes⚡ 3 deprecations🔧 20 symbols
Summary
This release introduces several new features across Service Discovery modules (AWS, Azure, Kubernetes) and PromQL operators, alongside significant performance enhancements in PromQL and TSDB operations. Several legacy Hetzner Service Discovery labels are deprecated.
Migration Steps
- For Hetzner robot role, replace usage of the `__meta_hetzner_datacenter` label with `__meta_hetzner_robot_datacenter`.
- For Hetzner hcloud role, replace usage of the `__meta_hetzner_hcloud_datacenter_location` label with `__meta_hetzner_hcloud_location`.
- For Hetzner hcloud role, replace usage of the `__meta_hetzner_hcloud_datacenter_location_network_zone` label with `__meta_hetzner_hcloud_location_network_zone`.
✨ New Features
- AWS SD: Add Elasticache Role.
- AWS SD: Add RDS Role.
- Azure SD: Add support for Azure Workload Identity authentication method.
- Discovery: Introduce `prometheus_sd_last_update_timestamp_seconds` metric to track the last time a service discovery update was sent to consumers.
- Kubernetes SD: Add support for node role selectors for pod roles.
- Kubernetes SD: Introduce pod-based labels for deployment, cronjob, and job controller names: `__meta_kubernetes_pod_deployment_name`, `__meta_kubernetes_pod_cronjob_name` and `__meta_kubernetes_pod_job_name`, respectively.
- PromQL: Add `</` and `>/` operators for trimming observations from native histograms.
- PromQL: Add experimental `histogram_quantiles` variadic function for computing multiple quantiles at once.
- TSDB: Add `storage.tsdb.retention.percentage` configuration to configure the maximum percent of disk usable for TSDB storage.
- TSDB: Add an experimental `fast-startup` feature flag that writes a `series_state.json` file to the WAL directory to track active series state across restarts.
- TSDB: Add an experimental `st-storage` feature flag. When enabled, Prometheus stores ingested start timestamps (ST, previously called Created Timestamp) from scrape or OTLP in the TSDB and Agent WAL, and exposes them via Remote Write 2.
- TSDB: Add an experimental `xor2-encoding` feature flag for the new TSDB block float sample chunk encoding that is optimized for scraped data and allows encoding start timestamps.
🐛 Bug Fixes
- AWS SD: Fix EC2 SD ignoring the configured `endpoint` option, a regression from the AWS SDK v2 migration.
- AWS SD: Fix panic in EC2 SD when DescribeAvailabilityZones returns nil ZoneName or ZoneId.
- Agent: Fix memory leak caused by duplicate SeriesRefs being loaded as active series.
- Alerting: Fix alert state incorrectly resetting to pending when the FOR period is increased in the config file.
- Azure SD: Fix system-assigned managed identity not working when `client_id` is empty.
- Consul SD: Fix filter parameter not being applied to health service endpoint, causing Node and Node.Meta filters to be ignored.
- Kubernetes SD: Fix duplicate targets generated by `*DualStack` EndpointSlices policies.
- OTLP: Fix ErrTooOldSample being returned as HTTP 500 instead of 400 in PRW v2 histogram write paths, preventing infinite client retry loops.
- OTLP: Fix exemplars getting mixed between incorrect parts of a histogram.
- PromQL: Do not skip histogram buckets in queries where histogram trimming is used.
- Remote write: Fix `prometheus_remote_storage_sent_batch_duration_seconds` measuring before the request was sent.
- Rules: Fix alert state restoration when rule labels contain Go template expressions.
- Scrape: Fix panic when parsing bare label names without an equal sign in brace-only metric notation.
- TSDB: Fail early when `use-uncached-io` feature flag is set on unsupported environments.
- TSDB: Fall back to CLI flag values when retention is removed from config file.
- TSDB: Fix memory leaks in buffer pools by clearing reference fields before returning buffers to pools.
- TSDB: Fix missing mmap of histogram chunks during WAL replay.
- TSDB: Fix storage.tsdb.retention.time unit mismatch in file causing retention to be 1e6 times longer than configured.
- Tracing: Fix missing traceID in query log when tracing is enabled, previously only spanID was emitted.
- UI: Fix tooltip Y-offset drift when using multiple graph panels.
- UI: Update retention display in runtime info when config is reloaded.
Affected Symbols
__meta_hetzner_datacenter__meta_hetzner_robot_datacenter__meta_hetzner_hcloud_datacenter_location__meta_hetzner_hcloud_location__meta_hetzner_hcloud_datacenter_location_network_zone__meta_hetzner_hcloud_location_network_zoneprometheus_sd_last_update_timestamp_secondshistogram_quantilesstorage.tsdb.retention.percentagefast-startupst-storagexor2-encoding__meta_kubernetes_pod_deployment_name</>/external_id__meta_kubernetes_pod_cronjob_name__meta_kubernetes_pod_job_nameprometheus_remote_storage_sent_batch_duration_secondsseries_state.json
⚡ Deprecations
- Hetzner SD: The `__meta_hetzner_datacenter` label is deprecated for the role `robot`; use the `__meta_hetzner_robot_datacenter` label instead.
- Hetzner SD: The `__meta_hetzner_hcloud_datacenter_location` label is deprecated; use the `__meta_hetzner_hcloud_location` label instead.
- Hetzner SD: The `__meta_hetzner_hcloud_datacenter_location_network_zone` label is deprecated; use the `__meta_hetzner_hcloud_location_network_zone` label instead.