main / unreleased

[BUGFIX] Fix tempo-vulture ignoring -tempo-push-tls flag in normal operating mode. #6974 (@xaque208)
[CHANGE] BREAKING CHANGE Remove duplicate "compaction" prefix from CompactorConfig CLI flags. Affected flags: compaction.block-retention, compaction.max-objects-per-block, compaction.max-block-bytes, compaction.compaction-window. #6909 (@electron0zero)
[ENHANCEMENT] Support OR conditions for tag name and tag value autocomplete (search tags v2) #6827 (@ie-pham)
[ENHANCEMENT] Expose MinIO retry settings via S3 config #6561 (@rwhitty)
[ENHANCEMENT] Reduce default livestore WAL size and align query defaults: max_block_duration 1m to 30s, max_block_bytes 100MiB to 50MiB, complete_block_timeout 1h to 20m, metrics query_backend_after 30m to 15m. #6974 (@zhxiaogg)
[CHANGE] BREAKING CHANGE Centralize block and WAL config: block_builder and live_store now always use storage.trace.block settings; per-module block config fields are removed. #6647 (@stoewer)
[CHANGE] BREAKING CHANGE Remove Opencensus receiver #6523 (@javiermolinar)
[CHANGE] Upgrade Tempo to Go 1.26.0 #6443 (@stoewer)
[CHANGE] BREAKING CHANGE Remove legacy mem-ballast-size-mbs cli flag. #6403 (@orkhan-huseyn)
[CHANGE] Allow duplicate dimensions for span metrics and service graphs. This is a valid use case if using different instrumentation libraries, with spans having "deployment.environment" and others "deployment_environment", for example. #6288 (@carles-grafana)
[CHANGE] Updade default max duration for traceql metrics queries up to one day #6285 (@javiermolinar)
[CHANGE] Set traceQL query metrics checks by default in Vulture #6275 (@javiermolinar)
[CHANGE] TraceQL metrics - change default step intervals to align with new vParquet5 timestamp columns #6413 (@mdisibio)
[CHANGE] Remove all traces of ingesters from the dashboards #6352 (@javiermolinar)
[CHANGE] BREAKING CHANGE tempo-cli: Support relative time (now, now-1h) for start/end args and standardize on RFC3339 in all commands. #6458 (@electron0zero) query search command no longer accepts timestamps without timezone (e.g. 2024-01-01T00:00:00), use RFC3339 (e.g. 2024-01-01T00:00:00Z) or relative time instead.
[CHANGE] BREAKING CHANGE Consolidate read configuration for recent data cutoff. query_frontend.search.query_ingesters_until is removed in favor of only query_frontend.search.query_backend_after. #6507 (@mapno)
[CHANGE] jsonnet: Add emptyDir data volume to block-builder StatefulSet #6648 (@mapno)
[CHANGE] Add quick checks to tempo mixin runbook #6696 (@javiermolinar)
[CHANGE] Deprecate metrics-generator no-local-blocks #6707 (@javiermolinar)
[CHANGE] Own local block and partition ring helpers #6808 (@javiermolinar)
[CHANGE] Track invalid trace and span id discards #6799 (@javiermolinar)
[CHANGE] Deprecate query_frontend.rf1_after and query all blocks regardless of replication factor for non-metrics paths. Simplifies 2.x to 3.0 migration. #6969 (@mapno)
[FEATURE] Add automemlimit support for automatic GOMEMLIMIT configuration. Enable with memory.automemlimit_enabled: true. #6313 (@oleg-kozlyuk)
[FEATURE] Support comparison operators in TraceQL Metrics queries #6474 (@ruslan-mikhailov)
[FEATURE] Add new include_any filter policy for spanmetrics filter #6392 (@javiermolinar)
[FEATURE] Add span_multiplier_key to overrides. This allows tenants to specify the attribute key used for span multiplier values to compensate for head-based sampling. #6260 (@carles-grafana)
[FEATURE] BREAKING CHANGE Optimize TraceQL AST by rewriting conditions on the same attribute to their array equivalent #6353 (@stoewer) Slightly changes the array matching semantics of != and !~ operators and introduces stricter rules for regex literals.
[FEATURE] metrics-generator: Add per-label limiter to control cardinality #6414 (@electron0zero) Adds max_cardinality_per_label per per-tenant override and new metrics to estimate per label cardinality demand estimate.
[FEATURE] Add an extension mechanism for per-tenant overrides #6758 (@stoewer)
[FEATURE] Extend TraceRedactor interface to support hiding complete traces via ErrTraceHidden. #6811 (@stoewer)
[FEATURE] Single-binary mode: push distributor local ingest directly to live-store and metrics-generator without Kafka #6729 (@javiermolinar)
[ENHANCEMENT] Expose MinIO retry settings via S3 config #6561 (@rwhitty)
[ENHANCEMENT] Enable native histogram emission for all promauto-registered histograms, including tempo_request_duration_seconds. Both classic and native formats are emitted simultaneously; existing scrapers are unaffected. #6910 (@zalegrala)
[ENHANCEMENT] tempo-cli: Add --header flag to query api commands for custom headers #6768 (@Nouuu)
[ENHANCEMENT] tempo-cli: add redact command to submit trace redaction jobs to the backend scheduler #6832 (@zalegrala)
[ENHANCEMENT] Block builder: deduplicate spans within traces during block creation and track removed duplicates via tempo_block_builder_spans_deduped_total metric #6539 (@zhxiaogg)
[ENHANCEMENT] metrics-generator: Support extracting span multiplier from W3C tracestate OTel probability sampling threshold via enable_tracestate_span_multiplier config option #6684 (@cmarchbanks)
[ENHANCEMENT] Add new alerts and runbooks entries #6276 (@javiermolinar)
[ENHANCEMENT] Double the maximum number of dedicated string columns in vParquet5 and update tempo-cli to determine the optimum number for the data #6282 (@mdisibio)
[ENHANCEMENT] TraceQL metrics - experimental faster read path for most metrics queries, accessible behind the query hint spanonly_fetch=true when unsafe_query_hints is enabled #6359 (@mdisibio)
[ENHANCEMENT] TraceQL metrics - add new per-tenant override to opt-in or opt-out of the new experimental faster read path for most metrics queries. #6849 (@mdisibio)
[ENHANCEMENT] Vulture: extend data consistency checks to include more strings, integers, and blobs, at resource/span/event scopes, and perform deeper trace content check #6731 (@mdisibio)
[ENHANCEMENT] Improve attribute truncating observability #6400 (@javiermolinar)
[ENHANCEMENT] Log truncated oversized attributes #6467 (@carles-grafana)
[ENHANCEMENT] livestore: make trace_too_large log line an insight #6371 (@carles-grafana)
[ENHANCEMENT] Remove live-store partition owner from ring on shutdown to prevent stale owner entries #6409 (@oleg-kozlyuk-grafana)
[ENHANCEMENT] Improved live store readiness check and added readiness_target_lag and readiness_max_wait config parameters. Live store will now - if readiness_target_lag is set - not report /ready until Kafka lag is brought under the specified value #6238 #6405 (@oleg-kozlyuk-grafana, @ruslan-mikhailov)
[ENHANCEMENT] Expose a new histogram metric to track the jobs per query distribution #6343 (@javiermolinar)
[ENHANCEMENT] Do deep validation for filter policies in user configurable overrides API #6407 (@electron0zero)
[ENHANCEMENT] Allow span_name_sanitization to be set via user-configurable overrides API #6411 (@Logiraptor)
[ENHANCEMENT] Add fail_on_high_lag parameter to allow live-store to fail if it is lagged #6363 #6567 (@ruslan-mikhailov, @carles-grafana)
[ENHANCEMENT] Add support for per-tenant left-padding of trace IDs #6489 (@mapno)
[ENHANCEMENT] Add new metric for generator ring size: tempo_distributor_metrics_generator_tenant_ring_size #5686 (@zalegrala)
[ENHANCEMENT] Remove explicit runtime.GC() calls in vParquet5 compactor/block creation and CLI #6603 (@oleg-kozlyuk-grafana)
[ENHANCEMENT] Reduce allocations in extendReuseSlice growth path during WAL writes and block creation #6863 (@mapno)
[ENHANCEMENT] Implemented anti-affinity for pods in same livestore zone #6757 (@zhxiaogg)
[ENHANCEMENT] Livestore: skipped WAL complete op during shutdown #6839 (@zhxiaogg)
[ENHANCEMENT] Add metric to track livestore block cut reasons #6922 (@zhxiaogg)
[ENHANCEMENT] Enable async parquet read mode for WAL completion path #6967 (@zhxiaogg)
[BUGFIX] livestore: check readiness before lag for SearchRecent and QueryRange queries #6911 (@zhxiaogg)
[BUGFIX] Fix integer overflow in query parameters by using strconv.ParseUint instead of strconv.Atoi/strconv.ParseInt for unsigned integer fields. #6612 (@bejaratommy)
[BUGFIX] Fix live-store SearchTagValuesV2 disk cache never being populated on complete blocks #6858 (@mapno)
[BUGFIX] Fix dedicated columns fallback in block_builder and live_store to use storage.trace.block.parquet_dedicated_columns when not set via overrides. #6647 (@stoewer)
[BUGFIX] Force live-store to rehydrate from Kafka lookback period when local data is missing (e.g. PVC wipe, new node) instead of resuming from the committed consumer group offset #6428 (@oleg-kozlyuk-grafana)
[BUGFIX] fix: reload span_name_sanitization overrides during runtime #6435 (@electron0zero)
[BUGFIX] fix: live store honor the config options for block and WAL versions #6509 (@mdisibio)
[BUGFIX] fix: block builder honor the global storage block config for block and WAL versions #6451 (@Harry-kp)
[BUGFIX] fix: normalize allowlist headers when building the allowlist map #6481 (@javiermolinar)
[BUGFIX] fix: bug related to dedicated column filtering #6586 (@stoewer)
[BUGFIX] fix: compactor deduped spans metric uses wrong type (gauge instead of counter) #6558 (@bejaratommy)
[BUGFIX] metrics-generator: Fix active-series counter underflow in local series limiter when overflow series are deleted #6568 (@carles-grafana)
[BUGFIX] fix: skip per-label limiter and sanitizer for target_info and host_info metrics in metrics-generator #6660 (@electron0zero)
[BUGFIX] fix(traceql): err on division by zero #6580 (@Proximyst)
[BUGFIX] fix(traceql): stop intPow from hanging #6581 (@Proximyst)
[BUGFIX] fix(traceql): Fix incorrect search results for some queries on new blob columns #6815 (@mdisibio)
[BUGFIX] fix(vparquet5) Fix buffer-reuse bug where event attributes in dedicated columns could be persisted on additional spans and events #6914 (@mdisibio)
[BUGFIX] fix: race condition where remove_owner_on_shutdown flag was set too late — after context cancellation already triggered the lifecycler's shutdown, causing the partition owner to remain in the ring #6693 (@oleg-kozlyuk-grafana)
[BUGFIX] Return 400 instead of 500 when query_range or query_instant requests have unparseable start/end parameters #6694 (@ruslan-mikhailov)
[BUGFIX] fix: correct block-builder fetch metrics to use counters instead of gauges #6578 (@WinterCabbage)
[BUGFIX] Log tenant on receiver push errors #6780 (@javiermolinar)
[BUGFIX] Fix race conditions in WAL block #6773 (@ruslan-mikhailov)
[BUGFIX] metrics-generator: Fix target_info being skipped when resource attributes have empty values #6774 (@carles-grafana)
[BUGFIX] metrics-generator: Drain old series on metric replacement to prevent limiter leak and permanent overflow #6653 (@carles-grafana)
[BUGFIX] live-store: fixed unsuccessful deregistering from membership/partition rings during shutdown #6848 (@zhxiaogg)
[BUGFIX] fix: respect context cancellation when reading WAL block iterator #6928 (@zhxiaogg)
[BUGFIX] Complete lifecycler shutdown on errors #6906 (@javiermolinar)
[BUGFIX] livestore: fix concurrent WAL writes from periodic and shutdown flushes #6972 (@zhxiaogg)
[BUGFIX] live-store: fix race conditions for tag values endpoint #7000 (@ruslan-mikhailov)
[BUGFIX] live-store: correct backoff duration calculation #6999 (@ruslan-mikhailov)

3.0 Cleanup

[CHANGE] BREAKING CHANGE Remove partition ring livestore config #6981 (@javiermolinar)
[CHANGE] Flush blocks to backend storage from the Live store in single binary mode #6941 (@javiermolinar)
[CHANGE] BREAKING CHANGE Remove ingester module #6959 (@javiermolinar)
[CHANGE] Remove stale config from the examples #6980 (@javiermolinar)
[CHANGE] BREAKING CHANGE Remove ingest.enabled config #6873 (@javiermolinar)
[CHANGE] BREAKING CHANGE Disable legacy (flat, unscoped) overrides by default. Tempo will refuse to start if legacy overrides are detected. Set enable_legacy_overrides: true or -config.enable-legacy-overrides=true to opt back in temporarily. Legacy overrides will be removed in a future release. #6741 (@electron0zero)
[CHANGE] tempo-cli: Rewrite migrate overrides-config and add migrate overrides-per-tenant command to help migrate legacy flat overrides to the new scoped format. #6793 (@electron0zero)
[CHANGE] BREAKING CHANGE Remove remaining app ingester config #6667 (@javiermolinar)
[CHANGE] BREAKING CHANGE Remove span-metrics leftovers and lazy-init generator clients #6618 (@javiermolinar)
[CHANGE] BREAKING CHANGE Decommission livestore MetricsGenerator query service #6615 (@javiermolinar)
[CHANGE] BREAKING CHANGE Remove metrics-generator localblocks processor and related local block storage plumbing. #6555 (@javiermolinar)
[CHANGE] BREAKING CHANGE Remove ingesters #6504 (@javiermolinar)
[CHANGE] Decouple livestore from metrics-generator #6506 #6535 (@javiermolinar)
[CHANGE] BREAKING CHANGE Remove ingesters and compactor alerts #6369 (@javiermolinar)
[CHANGE] BREAKING CHANGE Removed v2 block encoding and compactor component. #6273 (@joe-elliott) This includes the removal of the following CLI commands which were v2 specific: list block, list index, view index, gen index, gen bloom.
[CHANGE] BREAKING CHANGE SpanMetricsSummary is removed and querier code simplified #6496 #6510 (@javiermolinar)
[CHANGE] BREAKING CHANGE Sets the all target to be 3.0 compatible and removes the scalable-single-binary target #6283 (@joe-elliott)
[CHANGE] BREAKING CHANGE Clean up enterprise jsonnet #6505 (@javiermolinar)
[CHANGE] Expose otlp http and grpc ports for Docker examples #6296 (@javiermolinar)

v2.10.3

[SECURITY] S3 SSE-C encryption_key is now treated as a secret to prevent it from being exposed in plaintext. Resolves CVE-2026-28377. #6711 (@mattdurham)

v2.10.2

[BUGFIX] Apply exemplars hint end-to-end and fix safety cap bypass in metrics queries. #6559 (@zhxiaogg)
[ENHANCEMENT] Used frontend MaxExemplars config as single source of truth for exemplar limits. Added a safety cap at the traceql engine entry points. #6515 (@zhxiaogg)
[CHANGE] Set default max_result_limit for search to 256*1024 #6525 (@zhxiaogg)

v2.10.1

[CHANGE] Upgrade Tempo to Go 1.25.7 #6449 (@zalegrala)
[BUGFIX] Fix query-frontend unable to convert dedicated column blob option #6377 (@stoewer)
[BUGFIX] Correct avg_over_time calculation #6252 (@ruslan-mikhailov)
[BUGFIX] Correct instant query calculation for rate() #6205 (@ruslan-mikhailov)
[BUGFIX] Fix issue with additional matched spans in queries with only resource attributes #6432 (@stoewer)
[BUGFIX] Fix live-store deadlock occurring after a complete block failure #6338 (@ruslan-mikhailov)
[BUGFIX] Fix query_end_cutoff param for query range #6360 (@ruslan-mikhailov)
[BUGFIX] generator: fix dimension_mappings and target_info_excluded_dimensions being unconditionally overwritten even when overrides were nil #6390 (@carles-grafana)
[BUGFIX] generator: fix panic when write_relabel_configs is configured on remote write endpoints #6396 (@carles-grafana)

v2.10.0

[CHANGE] BREAKING CHANGE Validate tenant ID in frontend and distributor #5786 (@carles-grafana)
[CHANGE] Remove vParquet2 encoding #6071 (@mdisibio) BREAKING CHANGE In the last release vParquet2 was deprecated and blocked from writes. Now, it's removed entirely. It will no longer be recognized as a valid encoding and cannot read any remaining vParquet2 blocks. Installations running with historical defaults should not require any changes as the default has been migrated for several releases. Installations with storage settings pinned to vParquet2 must run a previous release configured for vParquet3 or higher until all existing vParquet2 blocks have expired and been deleted from the backend, or else it will encounter read errors after upgrading to this release.
[CHANGE] BREAKING CHANGE Added scope and query params while rewriting integration tests to the Tempo HTTP client SearchTagsV2WithRange function. #6088 (@joe-elliott)
[CHANGE] BREAKING CHANGE TempoDB schemas vParquet3 and v2 are deprecated and will be removed in 3.0. #6198 (@joe-elliott)
[CHANGE] Add stricter validation to user-configurable overrides API, such as refusing duplicate labels #6008 #6104 (@carles-grafana, @mdisibio)
[CHANGE] Expose metrics_generator.dimension_mappings to user-configurable overrides API #5989 (@carles-grafana)
[CHANGE] Expose metrics_generator.intrinsic_dimensions to user-configurable overrides API #5974 (@carles-grafana)
[CHANGE] Expose metrics_generator.trace_id_label_name to user-configurable overrides API #5972 (@carles-grafana)
[CHANGE] Expose metrics_generator.ingestion_time_range_slack to user-configurable overrides API #5958 (@carles-grafana)
[CHANGE] Expose metrics_generator.native_histogram_bucket_factor and native_histogram_min_reset_duration to user-configurable overrides API #5973 (@carles-grafana)
[CHANGE] Remove remaining aws-sdk-go references and migrate tests to MinIO #5856 (@anglerfishlyy)
[CHANGE] Remove busybox from Tempo image to make it more minimal and prevent future vulnerabilities #5717 (@carles-grafana)
[CHANGE] Upgrade Tempo to Go 1.25.4 #5939 #6001 (@ruslan-mikhailov)
[CHANGE] Upgrade Tempo to Go 1.25.5 #6096 #6089 (@joe-elliott)
[FEATURE] New block encoding vParquet5 with dedicated attributes for integers and events, improved performance on high-cardinality/high-length attributes, faster TraceQL queries, and childCount intrinsic. See vParquet5 section below for full details. #6219 (@ie-pham @mdisibio @stoewer)
[FEATURE] TraceQL: Add minInt/maxInt. #5982 (@Syedowais312)
[FEATURE] TraceQL: Add support for attribute = nil #4905 (@ie-pham)
[FEATURE] Add support for Accept: application/vnd.grafana.llm to Tempo endpoints to improve MCP responses. Currently supported directly by trace by id and tag values #5962 (@joe-elliott) This response is subject to change and should not be relied on. It is intended for LLM consumption only. Even a fundamental change to its representation (yaml? markdown?) would not be considered breaking.
[FEATURE] Add entity-based limiting mode for metrics-generator as an alternative to series-based limiting. #5788 (@Logiraptor)
[FEATURE] Add tempo_metrics_generator_registry_active_series_demand_estimate that estimates metrics-generator active series demand even when the active series limit is reached #5710 (@carles-grafana)
[FEATURE] Metrics generator will now produce overflow series to capture new data once limits are hit. These series have the label metric_overflow="true". #5954 (@Logiraptor)
[FEATURE] Add experimental drain limiter to the metrics generator. This can reduce metrics cardinality by clustering similar span names. #6098 (@Logiraptor)
[FEATURE] Add support for external storage to trace by id endpoint #6185 (@Logiraptor)
[ENHANCEMENT] add database_name_attributes config to service graph processor #5398 (@KyriosGN0)
[ENHANCEMENT] Added validation mode and tests for tempo-vulture #5605 (@davidham)
[ENHANCEMENT] Add SSE-C encryption support to S3 backend #5789 (@steffsas)
[ENHANCEMENT] docs: Add explicit notes about authentication #5735 (@electron0zero)
[ENHANCEMENT] Add secure connection support to tempo-cli #5692 (@TheoBrigitte)
[ENHANCEMENT] Add config to enable instance label for spanmetrics series #5706 (@ie-pham)
[ENHANCEMENT] Unsafe search hints for frontend performance tests #5723 (@ruslan-mikhailov)
[ENHANCEMENT] TraceQL query performance improvement #5773 (@mdisibio)
[ENHANCEMENT] Performance improvement in quantile_over_time() #5996 (@mdisibio)
[ENHANCEMENT] Improve shutdown time in the first 30 seconds #5725 (@ldufr)
[ENHANCEMENT] Add metric for tracking added latency to write requests #5781 (@mapno)
[ENHANCEMENT] Update jsonnet vendor dependencies #6202 (@javiermolinar)
[ENHANCEMENT] Improve error message when overrides fail to parse #5787 (@mapno)
[ENHANCEMENT] Add default_spans_per_span_set configuration option to control default spans returned per trace in search results #5858 (@iamrajiv)
[ENHANCEMENT] Increase weight for heavy TraceQL queries #5782 (@ruslan-mikhailov)
[ENHANCEMENT] Improve observability of collection failures in the metrics generator with error categorization #5936 (@javiermolinar)
[ENHANCEMENT] Update tempo-mixin dashboards to use timeseries panels with explicit units #6210 (@javiermolinar)
[ENHANCEMENT] Remove Metrics Summary/GetMetrics panels from the tempo-operational dashboard #6221 (@javimolinar)
[ENHANCEMENT] Update list of intrinsics returned by search tags endpoint #5857 (@andreasgerstmayr)
[ENHANCEMENT] Add "Requests Executed" panel for querier metrics in the operational dashboard. #5848 (@anglerfishlyy)
[ENHANCEMENT] Drop and metric invalid utf8 for all metrics in metrics generator instead of sending invalid data. #5980 (@joe-elliott)
[ENHANCEMENT] Add support for application/protobuf in frontend endpoints #5865 (@oleg-kozliuk-grafana)
[ENHANCEMENT] Validate metrics-generator histogram buckets #5991 (@carles-grafana)
[ENHANCEMENT] TraceQL: Add support for with(exemplars=false) hint to disable exemplars in metrics queries #6204(@joe-elliott)
[ENHANCEMENT] Removed MustNewConstMetric to prevent panic and added validation for usage tracker config. Added tempo_distributor_usage_tracker_errors_total to surface errors in usage tracker. #5981 (@electron0zero)
[ENHANCEMENT] Simplify block-builder partition assignment with config parameter partitions_per_instance #6022 (@mapno)
[ENHANCEMENT] Allow RetryInfo to be disabled in per tenant overrides #5741 (@electron0zero)
[ENHANCEMENT] Set maxKeys to 1 for S3 confirm list operation #6114 (@cancub)
[BUGFIX] Fix leading zero handling in native histograms implementations #6033 (@zalegrala)
[BUGFIX] Prevent slice panic when truncating series after topk() by adding bounds check in metrics query-range combiner #6010 (@Syedowais312)
[BUGFIX] Fix compactor to properly consider SSE-KMS information during metadata copy #5774 (@steffsas)
[BUGFIX] Fix spss=0 parameter to properly mean unlimited spans instead of being rejected, and respect max_spans_per_span_set=0 configuration #5858 (@iamrajiv)
[BUGFIX] Fix incorrect results in TraceQL compare() caused by potential hash collision of string array attributes #5835 (@mdisibio)
[BUGFIX] Fix != nil checks broken in TraceQL Metrics #5998 (@mdisibio)
[BUGFIX] Fix = nil handling with intrinsic columns #5999 (@mdisibio)
[BUGFIX] Fix: handle collisions with job and instance labels when targetInfo is enabled #5774 (@javiermolinar)
[BUGFIX] Fix search by trace:id for short IDs with leading zeros #5587 (@ruslan-mikhailov)
[BUGFIX] Fixed rare bug where the TraceQL metrics could return an empty response when max series was hit. #6099 (@joe-elliott)
[BUGFIX] Fix S3 compactor multipart upload to ensure Cloudflare R2 compliance by using uniform chunk sizes #5838 (@constantins2001)
[BUGFIX] Make top/bottomk deterministic by breaking ties with label vals. #5846 (@joe-elliott)
[BUGFIX] Fix metrics streaming releases intermediate/incomplete results. #5937 (@joe-elliott)
[BUGFIX] Fix GetTrace() in tempo-query. #5864 (@andreasgerstmayr)
[BUGFIX] Fix unsupported nonexistence nil operator for tags lookup. Fixed issue where some tag values/names were not returned due to distinctAttrCollector optimization #5967 (@ie-pham) (@joe-elliott)
[BUGFIX] Fix delete implementation for s3/gcs/azure backends to account for prefix. #6011 (@kaustubhkurve)
[BUGFIX] Fix slow pod startup (~90s) in monolithic mode by returning false from isSharded() when kvstore is empty or inmemory. #6035 (@AryanBagade)
[BUGFIX] Fix response-too-large.md ingestion config example in document #6116 (@gamerslouis)
[BUGFIX] generator: back off when instance creation fails to avoid resource exhaustion #6142 (@carles-grafana)
[BUGFIX] Correct handle whitespace or invisible separators in filters in attribute values in tag value search #6124 (@mapno)
[BUGFIX] Fix special character handling in attribute value requests.#6212,#6223 (@joe-elliott)
[BUGFIX] Fix panic when trying to compact block with unsupported encodings such as vParquet previews. #6209 (@carles-grafana)
[BUGFIX] querier: allow external trace by id queries in validation #6217 (@Logiraptor)
[BUGFIX] querier: use v1 trace by id endpoint for external trace by id requests #6236 (@Logiraptor)

vParquet5

[CHANGE] Simplify column paths in vParquet5-preview7 by removing LIST meta #6134 (@mdisibio)
[FEATURE] New TraceQL intrinsic span:childCount which is the number of direct children of the span. Support in vParquet5 and later. #6126 (@mdisibio)
[FEATURE] New block encoding vParquet5-preview4 with array support for dedicated columns. This is a preview, breaking changes are expected. #5760 (@stoewer)
[FEATURE] New block encoding vParquet5-preview5 with virtual span row numbers. This is a preview, breaking changes are expected. #5943 (@stoewer)
[FEATURE] New block encoding vParquet5-preview6 with dedicated columns for events and improved support for high-cardinality/high-length "blob" attributes #5946 (@mdisibio)
[FEATURE] New block encoding vParquet5-preview3 replacing well-known attributes with dedicated column defaults. This format is in development and breaking changes are expected before final release. #5696 (@stoewer)
[BUGFIX] Fix issue with orphaned group nodes in vParquet5-preview6 #6095 (@stoewer)
[BUGFIX] Fix issues related to integer dedicated columns in vParquet5-preview2 #5716 (@stoewer)
[BUGFIX] Fix disappearing dedicated event attrs in trace view and dedicated blob column validation in vParquet5-preview6 #6100 (@stoewer)
[BUGFIX] Fix dedicated column array handling to better support attributes with mixed array and non-array values #6199 (@mdisibio)
[ENHANCEMENT] Add integer attribute analysis to the tempo-cli analyse block(s) commands #6103 (@mdisibio)
[ENHANCEMENT] Add suggest columns command to tempo-cli #6131 (@ie-pham)
[ENHANCEMENT] Convert ServiceStats data from map to list and update encoding/compression to enable sorted output and small reduction in file size #6216 (@mdisibio)

Tempo Rearchitecture [EXPERIMENTAL]

[CHANGE] Allow to configure WAL for livestore #6093 (@javiermolinar)
[CHANGE] Added a single binary 3.0 mode --target=all-3.0 to begin testing single binary 3.0 and updating integration tests. #6021 (@joe-elliott) This will be removed in 3.0 and become the standard single binary mode.
[CHANGE] Kafka KIP-714 telemetry is enabled by default. Added disable_kafka_telemetry config flag to opt-out #6046 (@oleg-kozlyuk-grafana)
[CHANGE] Add query_end_cutoff setting to clamp near-“now” queries (default 30s) for consistent Tempo results #5682 (@javiermolinar)
[ENHANCEMENT] Improve dashboards for livestore #5755 (@javiermolinar)
[ENHANCEMENT] On startup, first record for live store to consume is not older than two complete block timeouts #5693 (@ruslan-mikhailov)
[ENHANCEMENT] Add endpoints for partition downscaling for live-store #5600 #5738 #5805 #5829 (@ruslan-mikhailov, @mapno)
[ENHANCEMENT] Add new livestore alert to the tempo-mixin #5752 (@javiermolinar)
[ENHANCEMENT] Add partition ownership metric to live-store #5815 (@javiermolinar, @mapno)
[BUGFIX] Fix block-builder to more precisely validate block encoding on startup #6037 (@mdisibio)
[BUGFIX] Correctly track and reject too large traces in live stores. #5757 (@joe-elliott)
[BUGFIX] Jsonnet: Correctly add tempo-gossip-member: true labels to block-builders and live-stores #6125 (@joe-elliott)
[BUGFIX] Fix wrong sleep duration in block-builder if one of partitions is inactive #5855 (@ruslan-mikhailov)

v2.9.0

[CHANGE] BREAKING CHANGE Deprecating vParquet2 block format #5688 (@ie-pham)
[CHANGE] Upgrade Tempo to go 1.25.1 #5685 (@electron0zero)
[CHANGE] BREAKING CHANGE We are no longer publishing rpm and deb packages due to an internal change to the handling of signing keys. #5684 (@joe-elliott)
[CHANGE] Return Bad Request from all frontend endpoints if the tenant can't be extracted. #5480 (@carles-grafana)
[CHANGE] BREAKING CHANGE Migrated Tempo Vulture and Integration Tests from the deprecated Jaeger agent/exporter to the standard OTLP exporter. Vulture now pushes traces to the Tempo OTLP GRCP endpoint. #5058 (@iamrajiv, @javiermolinar)
[CHANGE] Do not count cached querier responses for SLO metrics such as inspected bytes. #5185 (@carles-grafana)
[CHANGE] Adjust the definition of tempo_metrics_generator_processor_service_graphs_expired_edges to exclude edges that are counted in the service graph. #5319 (@joe-elliott)
[CHANGE] Command tempo-cli analyse block(s) excludes attributes with array values. #5380 (@stoewer)
[CHANGE] Remove .005s and add a 1.5s bucket to all request duration histograms. #5492 (@joe-elliott)
[CHANGE] Improve tempo writes dashboard. #5500 (@javiermolinar)
[CHANGE] BREAKING CHANGE TraceQL Metrics buckets are calculated based on data in past. #5366 (@ruslan-mikhailov)
[CHANGE] Upgrade Tempo to go 1.25.0. #5548 (@javiermolinar)
[CHANGE] Drop tracing bridges in favor of OTEL only tracing. #5594 (@zalegrala)
[CHANGE] BREAKING CHANGE Fix incorrect TraceQL metrics results when series labels include strings and integers with same textural representation. This also changes the TraceQL metrics responses of /api/metrics/query_range and /api/metrics/query to remove the redundant prom_labels field which was the error source. There may be an interruption to TraceQL metrics queries during rollout while components are running the previous version. #5659 (@mdisibio)
[CHANGE] Enable HTTP writes in the multi-tenant example. #5297 (@carles-grafana)
[FEATURE] Add MCP Server support. #5212 (@joe-elliott)
[FEATURE] Add query hints sample=true and sample=0.xx which can speed up TraceQL metrics queries by sampling a subset of the data to provide an approximate result. #5469 (@mdisibio)
[FEATURE] New block encoding vParquet5-preview1 with low-resolution timestamp columns for better TraceQL metrics performance. This format is in development and breaking changes are expected before final release. #5495 (@mdisibio)
[FEATURE] New block encoding vParquet5-preview2 with dedicated attribute columns for integers. This format is in development and breaking changes are expected before final release. #5639 (@stoewer)
[ENHANCEMENT] Add counter query_frontend_bytes_inspected_total, which shows the total number of bytes read from disk and object storage #5310 (@carles-grafana)
[ENHANCEMENT] Add histograms spans_distance_in_future_seconds / spans_distance_in_past_seconds that count spans with end timestamp in the future / past. While spans in the future are accepted, they are invalid and may not be found using the Search API. #4936 (@carles-grafana)
[ENHANCEMENT] Add support for scope in cost-attribution usage tracker. #5646 (@electron0zero)
[ENHANCEMENT] Add alert for high error rate reported by vulture. #5206 (@ruslan-mikhailov)
[ENHANCEMENT] Support the new db.namespace attribute for service-graph DB nodes. #5602 (@gouthamve)
[ENHANCEMENT] TraceQL metrics performance increase for simple queries. #5247 (@mdisibio)
[ENHANCEMENT] TraceQL search and metrics performance increase. #5280 (@mdisibio)
[ENHANCEMENT] TraceQL performance improvement. #5218 (@mdisibio)
[ENHANCEMENT] TraceQL compare() performance improvement. #5419 (@mdisibio)
[ENHANCEMENT] Align traceql attribute struct for better performance. #5240 (@mdisibio)
[ENHANCEMENT] Drop invalid prometheus label names in the spanmetrics processor. #5122 (@KyriosGN0)
[ENHANCEMENT] Improve logging and tracing in the write path to include tenant info. #5436 (@javiermolinar)
[ENHANCEMENT] Added usage tracker example. #5356 (@javiermolinar)
[ENHANCEMENT] Add Stop method. #5293 (@stephanos)
[ENHANCEMENT] Use peer attributes to determine the name of a client service virtual node in the service graph. #5381 (@martenm)
[ENHANCEMENT] Put actual size for writing to backend. #5413 (@ruslan-mikhailov)
[ENHANCEMENT] Upgrade Azurite and Fake-gcs-server to latest version. #5512 (@javiermolinar)
[ENHANCEMENT] Make block ordering deterministic. #5411 (@iamrajiv)
[ENHANCEMENT] Improve exemplar selection in quantile_over_time(). #5278 (@zalegrala)
[ENHANCEMENT] Measure bytes received before limits and publish it as tempo_distributor_ingress_bytes_total. #5601 (@mapno)
[ENHANCEMENT] Add total size logging functionality to track trace #5625(@sienna011022)
[ENHANCEMENT] Enable histogram tuning in the user-configurable overrides API #5721(@zalegrala)
[BUGFIX] Fix Tempo configuration options that are always overrided with config overrides section. #5202 (@KyriosGN0)
[BUGFIX] Correctly apply trace idle period in ingesters and add the concept of trace live period. #5346 (@joe-elliott)
[BUGFIX] Fix invalid YAML output from /status/runtime_config endpoint by adding document separator. #5371 (@iamrajiv)
[BUGFIX] Fix panic in query_range HTTP handling that could be triggered by cancellations or other errors. #5667 (@mdisibio)
[BUGFIX] Do not allow very small steps. #5441 (@ruslan-mikhailov)
[BUGFIX] Fix incorrect TraceQL string comparison of strings starting with numbers. #5658 (@mdisibio)
[BUGFIX] Fix incorrect results in TraceQL compare() for spans with array attributes #5519 (@ruslan-mikhailov)
[BUGFIX] Fix cache collision for incomplete query in SearchTagValuesV2 #5549 (@ruslan-mikhailov)
[BUGFIX] Fix for structural operator with empty left-hand spanset. #5578 (@ruslan-mikhailov)
[BUGFIX] Deadlock on invalid query to api/v2/search/tags. (SearchTagsV2) #5607 (@ruslan-mikhailov)
[BUGFIX] Fixed incorrect root span detection when spans have a child_of link but no parent. #3634 (@mexirica)
[BUGFIX] Prevent metrics-generator WAL deletion when tenant is empty. #5586 (@sienna011022)
[BUGFIX] Fix docker-compose port configuration for Alloy gRPC (4319 → 4317). #5536
[BUGFIX] Fix panic error from empty span id. #5464
[BUGFIX] Return Bad Request from frontend if the provided tag is invalid in SearchTagValuesV2 endpoint. #5493 (@carles-grafana)

v2.9 Tempo Rearchitecture [EXPERIMENTAL]

[CHANGE] BREAKING CHANGE Drop unused backend_scheduler.tenant_measurement_interval, use backend_scheduler.compaction.measure_interval instead. #5328 (@zalegrala)
[CHANGE] Allow configuration of min/max input blocks for compaction provider. #5373 (@zalegrala)
[CHANGE] BREAKING CHANGE Add require minimum time between tenant sorting in backend-scheduler. #5410 (@zalegrala) The configuration for backend_scheduler.provider.compaction.backoff has been removed. Additionally, the compaction_tenant_backoff_total metric has been renamed to compaction_empty_tenant_cycle_total for clarity.
[CHANGE] Shard backend-scheduler work files locally and modify backend work structure to accommodate sharding approach. #5412 (@zalegrala)
[CHANGE] Change worker to shutdown after the current job, waiting 30s by default. #5460 (@zalegrala)
[CHANGE] Assert max live traces limits in the local-blocks processor. #5170 (@mapno)
[ENHANCEMENT] Add new header to vulture requests to query live stores. #5663 (@javiermolinar)
[ENHANCEMENT] Add endpoint for partition downscaling. #4913 (@mapno)
[ENHANCEMENT] Add backend scheduler and worker to the resources dashboard. #5206 (@javiermolinar)
[ENHANCEMENT] Allow configure group lag exporter update time. #5431 (@javiermolinar)
[ENHANCEMENT] Implement a listOffset by partition client. #5415 (@javiermolinar)
[ENHANCEMENT] Do not compact unfinished blocks. #5390 (@ruslan-mikhailov)
[ENHANCEMENT] Allow block-builder to operate over empty partitions. #5581 (@ruslan-mikhailov)
[ENHANCEMENT] Refactor method to wait for a healthy broker. #5618 (@javiermolinar)
[ENHANCEMENT] Remove duplicated metric to count the number of processed records. #5654 (@javiermolinar)
[ENHANCEMENT] Don't enqueue records to kafka when the context has been cancelled. #5499 (@javiermolinar)
[ENHANCEMENT] Include backendwork dashboard and include additional alert. #5159 (@zalegrala)
[ENHANCEMENT] Add live store to jsonnet lib. #5591 #5606 #5609 (@mapno)
[BUGFIX] Fix race condition between compaction provider and backend-scheduler. #5409 (@zalegrala)
[BUGFIX] Correctly support req.AllowPartialTrace and tenant limits in the live store when requesting trace by id. #5680 (@joe-elliott)

v2.8.2

[CHANGE] Update Go to version 1.24.4. #5323 (@stoewer)
[BUGFIX] Add nil check to partitionAssignmentVar. #5198 (@mapno)
[BUGFIX] Correct instant query calculation. #5252 (@ruslan-mikhailov)
[BUGFIX] Fix tracing context propagation in distributor HTTP write requests. #5312 (@mapno)
[BUGFIX] Fix search by trace:id with short trace ID. #5331 (@ruslan-mikhailov)
[BUGFIX] Fix bug where most_recent=true wouldn't return most recent results when query overlapped ingesters and few other blocks. #5438 (@joe-elliott)
[BUGFIX] Fix panic when counter series is missing during avg_over_time aggregation. #5300 (@ie-pham)

v2.8.1

[BUGFIX] Fix ingester issue where a hash collision could lead to spans stored incorrectly #5276 (@carles-grafana)

v2.8.0

[CHANGE] BREAKING CHANGE Change default http-listen-port from 80 to 3200 #4960 (@martialblog)
[CHANGE] BREAKING CHANGE Upgrade OTEL Collector to v0.122.1. The name dimension from tempo_receiver_accepted_span and tempo_receiver_refused_spans changes from tempo/jaeger_receiver to jaeger/jaeger_receiver. #4893 (@javiermolinar)
[CHANGE] BREAKING CHANGE Convert SLO metric query_frontend_bytes_processed_per_second from a histogram to a counter as it's more performant. #4748 (@carles-grafana)
[CHANGE] BREAKING CHANGE Remove tempo serverless. The following configuration options are no longer valid. If they are in your tempo config, remove them.
```
querier:
    search:
        prefer_self: <int>
        external_hedge_requests_at: <duration>
        external_hedge_requests_up_to:  <duration>
        external_backend: <string>
        google_cloud_run: <string>
        external_endpoints: <array>
```
The Tempo serverless-related metrics, tempo_querier_external_endpoint_duration_seconds, tempo_querier_external_endpoint_hedged_roundtrips_total, and tempo_feature_enabled, are also removed. #4599 (@electron0zero)
[CHANGE] BREAKING CHANGE Removed internal_error as a reason from tempo_discarded_spans_total. #4554 (@joe-elliott)
[CHANGE] BREAKING CHANGE Enforce max attribute size at event, link, and instrumentation scope. Make config per-tenant. Renamed max_span_attr_byte to max_attribute_bytes #4633 (@ie-pham)
[CHANGE] BREAKING CHANGE Removed otel jaeger exporter. #4926 (@javiermolinar)
[CHANGE] Update query range error message #4929 (@joey-grafana)
[CHANGE] Improve rate limit error message when traces size exceeds rate limit #4986 (@electron0zero)
[CHANGE] Tempo Query: replace opentracing-contrib/go-grpc by otelgrpc. #4958 (@javiermolinar)
[CHANGE] Improve parquet readers io.ReaderAt compatibility #4963 (@joe-elliott)
[CHANGE] Finish polling current tenants on poller shutdown #4897 (@zalegrala)
[CHANGE] Set querier default level to INFO #4943 (@javiermolinar)
[CHANGE] Change retention to honor compactor disablement #5044 (@zalegrala)
[CHANGE] Continue on error in tempo-cli rewrite-blocks #5043 (@zalegrala)
[CHANGE] Update jsonnet memcached and exporter image versions #5056 (@zalegrala)
[CHANGE] Enforce max series in response for metrics queries with new query-frontend configuration #4525 (@ie-pham)
[CHANGE] Update to go 1.24.3 #5110 (@stoewer)
[CHANGE] Update to go 1.24.2 #5057 #5082 (@carsontham)
[CHANGE] Update to go 1.24.1 #4704 (@ruslan-mikhailov) #4793 (@javiermolinar)
[FEATURE] Added most_recent=true query hint to TraceQL to return most recent results. #4238 (@joe-elliott)
[FEATURE] TraceQL metrics: sum_over_time #4786 (@javiermolinar)
[FEATURE] Add support for topk and bottomk functions for TraceQL metrics #4646 (@electron0zero)
[FEATURE] TraceQL: add support for querying by parent span id #4692 (@ie-pham)
[ENHANCEMENT] Add throughput SLO and metrics for the TraceByID endpoint. Configurable using the throughput_bytes_slo field. It populates the op="traces" label in slo and throughput metrics. #4668 (@carles-grafana)
[ENHANCEMENT] Add ability to add artificial delay to push requests #4716 #4899 #5035 (@yvrhdn, @mapno)
[ENHANCEMENT] tempo-vulture now generates spans with a parent, instead of only root spans #5154 (@carles-grafana)
[ENHANCEMENT] Add default mutex and blocking values. #4979 (@mattdurham)
[ENHANCEMENT] Improve Tempo build options #4755 (@stoewer)
[ENHANCEMENT] Rewrite traces using rebatching #4690 (@stoewer @joe-elliott)
[ENHANCEMENT] Reorder span iterators #4754 (@stoewer)
[ENHANCEMENT] Update minio to version #4341 (@javiermolinar)
[ENHANCEMENT] Prevent queries in the ingester from blocking flushing traces to disk and memory spikes. #4483 (@joe-elliott)
[ENHANCEMENT] Update tempo operational dashboard for new block-builder and v2 traces api #4559 (@mdisibio)
[ENHANCEMENT] Improve metrics-generator performance and stability by applying queue back pressure and concurrency #4721 (@mdisibio)
[ENHANCEMENT] Improve block-builder performance by flushing blocks concurrently #4565 (@mdisibio)
[ENHANCEMENT] Improve block-builder performance #4596 (@mdisibio)
[ENHANCEMENT] Improve block-builder performance by not using WAL stage #4647 #4671 (@mdisibio)
[ENHANCEMENT] Export new tempo_ingest_group_partition_lag metric from block-builders and metrics-generators #4571 (@mdisibio)
[ENHANCEMENT] Overall iterator performance improvement by using max definition level to ignore parts of the RowNumber while nexting. #4753 (@joe-elliott)
[ENHANCEMENT] Use distroless base container images for improved security #4556 (@carles-grafana)
[ENHANCEMENT] Rhythm: add block builder to resources dashboard #4556 (@javiermolinar)
[ENHANCEMENT] Upgrade prometheus to version 3.1.0 #4805 (@javiermolinar)
[ENHANCEMENT] Update dskit to latest version #4681 (@javiermolinar) #4865 (@javiermolinar)
[ENHANCEMENT] Increase query-frontend default batch size #4844 (@javiermolinar)
[ENHANCEMENT] Improve TraceQL perf by reverting EqualRowNumber to an inlineable function.#4705 (@joe-elliott)
[ENHANCEMENT] Rhythm: Implement MaxBytesPerCycle #4835 (@javiermolinar)
[ENHANCEMENT] Rhythm: fair partition consumption in blockbuilders #4655 (@javiermolinar)
[ENHANCEMENT] Rhythm: retry on commit error #4874 (@javiermolinar)
[ENHANCEMENT] Skip creating one span-traces for every pushed spans in metrics generator #4844 (@javiermolinar)
[ENHANCEMENT] Improve Tempo / Writes dashboard by adding a kafka panel #4947 (@javiermolinar)
[ENHANCEMENT] Improve memcached memory usage by pooling buffers #4970 (@joe-elliott)
[ENHANCEMENT] metrics-generator: allow skipping localblocks and consuming from a different source of data #4686 (@flxbk)
[ENHANCEMENT] compactor: restore dedicated columns logging for completed blocks #4832 (@edgarkz)
[ENHANCEMENT] Compactor: pooling changes to reduce memory usage #4985 (@mdisibio)
[ENHANCEMENT] distributor: add IPv6 support #4840 (@gjacquet)
[ENHANCEMENT] Support TraceQL Metrics checks in Vulture #4886 (@ruslan-mikhailov)
[ENHANCEMENT] Add memcached to the resources dashboard #5049 (@javiermolinar)
[ENHANCEMENT] Include partition owned metric for blockbuilders #5042 (@javiermolinar)
[ENHANCEMENT] Query-frontend: logs add msg to the log line #4975 (@jmichalek132)
[ENHANCEMENT] Host Info Processor: track host identifying resource attribute in metric #5152 (@rlankfo)
[ENHANCEMENT] Vulture checks recent traces #5157 (@ruslan-mikhailov)
[ENHANCEMENT] Add jitter in backendworker to avoid thundering herd from workers #5150 (@electron0zero)
[ENHANCEMENT] TraceQL Metrics: distribute exemplars over time #5158 (@ruslan-mikhailov)
[ENHANCEMENT] TraceQL Metrics: hard limit number of exemplars #5158 (@ruslan-mikhailov)
[BUGFIX] Choose a default step for a gRPC streaming query range request if none is provided. Correctly copy exemplars for metrics like | rate() when gRPC streaming. #4546 (@joe-elliott)
[BUGFIX] Make comparison to nil symmetric #4869 (@stoewer)
[BUGFIX] Fix behavior for queries like {.foo && true} and {.foo || false} #4855 (@stoewer)
[BUGFIX] Fix performance bottleneck and file cleanup in block builder #4550 (@mdisibio)
[BUGFIX] Add object name to cache key in ReadRange #4982 (@joe-elliott)
[BUGFIX] TraceQL incorrect results for additional spanset filters after a select operation #4600 (@mdisibio)
[BUGFIX] TraceQL metrics incorrect results for queries with multiple filters that reside in non-dedicated columns that also group by the same variable #4887 (@mdisibio)
[BUGFIX] TraceQL results caching bug for floats ending in .0 #4539 (@carles-grafana)
[BUGFIX] Rhythm: fix sorting order for partition consumption #4747 (@javiermolinar)
[BUGFIX] Rhythm: fix block builder to not reuse a block ID if it was already flushed, to prevent read errors #4872 (@mdisibio)
[BUGFIX] Rhythm: fix ingester to still clear old blocks when not flushing to object storage #5005 (@mdisibio)
[BUGFIX] Fix rare panic during compaction. #4915 (@joe-elliott)
[BUGFIX] Fix metrics streaming for all non-trivial metrics #4624 (@joe-elliott)
[BUGFIX] Fix starting consuming log #4539 (@javiermolinar)
[BUGFIX] TraceQL: Fix behavior of {} >> { span.attr-that-doesnt-exist != "foo" } #5007 (@joe-elliott)
[BUGFIX] Rhythm - fix adjustment of the start and end range for livetraces blocks #4746 (@javiermolinar)
[BUGFIX] Return the operand as the only value if the tag is already filtered in the query #4673 (@mapno)
[BUGFIX] Fix flaky test #4787 #4995 (@javiermolinar)
[BUGFIX] Fix flaky ingester test #4846 (@javiermolinar)
[BUGFIX] Include cost attribution when converting from default config to legacy one #4787 (@javiermolinar)
[BUGFIX] Fix memcached settings for docker compose example #4346 (@ruslan-mikhailov)
[BUGFIX] Update memcached to respect cancelled context to prevent panic #5041 (@joe-elliott)
[BUGFIX] Fix frontend cache key generation for TraceQL Metrics queries to prevent collisions. #5017 (@joe-elliott)
[BUGFIX] Fix setting processors in user configurations overrides via API #4741 (@ruslan-mikhailov)
[BUGFIX] Fix panic on startup #4744 (@ruslan-mikhailov)
[BUGFIX] Fix intrinsic tag lookups dropped when max tag lookup response size is exceeded #4784 (@mdisibio)
[BUGFIX] Use canonical bytesize when parsing partition.id in the distributor #5033 (@javiermolinar)
[BUGFIX] Measure the write latency on append errors #5034 (@javiermolinar)
[BUGFIX] Correctly cache frontend jobs for query range (TraceQL Metrics). #4771 (@joe-elliott)
[BUGFIX] Fix error propagation in the SyncIterator. #5045 (@joe-elliott)
[BUGFIX] Apply the artificial latency even on pushTrace errors #5153 (@javiermolinar)
[BUGFIX] Various edge case fixes for query range (TraceQL Metrics) #4962 (@ruslan-mikhailov)
[BUGFIX] Fix mixin to include otlp_v1_traces http write route #5072 (@mdisibio)
[BUGFIX] Fix TempoBlockListRisingQuickly alert grouping. #4876 (@mapno)
[BUGFIX] Reset SkipMetricsGeneration before reuse. #5117 (@flxbk)
[BUGFIX] Fix metrics generator host info processor overrides config. #5118 (@rlankfo)
[BUGFIX] Fix metrics generator target_info to skip attributes with no name to prevent downstream errors #5148 (@mdisibio)
[BUGFIX] Fix for queried number of exemplars (TraceQL Metrics) #5115 (@ruslan-mikhailov)
[BUGFIX] Excluded nestedSetParent and other values from compare() function #5196 (@mdisibio)
[BUGFIX] Fix distributor issue where a hash collision could lead to spans stored incorrectly #5186 (@mdisibio)
[BUGFIX] Fix structural metrics rate by aggregation #5204 (@zalegrala)
[BUGFIX] TraceQL Metrics: right exemplars for histogram and quantiles #5145 (@ruslan-mikhailov)

v2.7.2

[BUGFIX] Fix rare panic that occurred when a querier modified results from ingesters/generators while they were being marshalled to proto. #4790 (@joe-elliott) This bug also impacted query correctness on recent trace data by returning incomplete results before they were ready.

v2.7.1

[CHANGE] Default to snappy compression for all gRPC communications internal to Tempo. We feel this is a nice balance of resource usage and network traffic. For a discussion on alternatives see #4683. #4696 (@joe-elliott)

v2.7.0

[CHANGE] Disable gRPC compression in the querier and distributor for performance reasons #4429 (@carles-grafana) If you would like to re-enable it, we recommend 'snappy'. Use the following settings:

ingester_client:
    grpc_client_config:
        grpc_compression: "snappy"
metrics_generator_client:
    grpc_client_config:
        grpc_compression: "snappy"
querier:
    frontend_worker:
        grpc_client_config:
            grpc_compression: "snappy"

[CHANGE] slo: include request cancellations within SLO [#4355] (#4355) (@electron0zero) request cancellations are exposed under result label in tempo_query_frontend_queries_total and tempo_query_frontend_queries_within_slo_total with completed or canceled values to differentiate between completed and canceled requests.
[CHANGE] update default config values to better align with production workloads #4340 (@electron0zero)
[CHANGE] fix deprecation warning by switching to DoBatchWithOptions #4343 (@dastrobu)
[CHANGE] BREAKING CHANGE The Tempo serverless is now deprecated and will be removed in an upcoming release #4017 @electron0zero
[CHANGE] tempo-cli: add support for /api/v2/traces endpoint #4127 (@electron0zero) BREAKING CHANGE The tempo-cli now uses the /api/v2/traces endpoint by default, please use --v1 flag to use /api/traces endpoint, which was the default in previous versions.
[CHANGE] TraceByID: don't allow concurrent_shards greater than query_shards. #4074 (@electron0zero)
[CHANGE] BREAKING CHANGE The dynamic injection of X-Scope-OrgID header for metrics generator remote-writes is changed. If the header is aleady set in per-tenant overrides or global tempo configuration, then it is honored and not overwritten. #4021 (@mdisibio)
[CHANGE] BREAKING CHANGE Migrate from OpenTracing to OpenTelemetry instrumentation. Removed the use_otel_tracer configuration option. Use the OpenTelemetry environment variables to configure the span exporter #4028,#3646 (@andreasgerstmayr) To continue using the Jaeger exporter, use the following environment variable: OTEL_TRACES_EXPORTER=jaeger.
[CHANGE] No longer send the final diff in GRPC streaming. Instead we rely on the streamed intermediate results. #4062 (@joe-elliott)
[CHANGE] Update Go to 1.23.3 #4146 #4147 #4380 (@javiermolinar @mdisibio)
[CHANGE] Return 422 for TRACE_TOO_LARGE queries #4160 (@zalegrala)
[CHANGE] Upgrade OTEL sdk to reduce allocs #4243 (@joe-elliott)
[CHANGE] Update Alpine image version to 3.21 #4504 (@mdisibio)
[CHANGE] Tighten file permissions #4251 (@zalegrala)
[CHANGE] Drop max live traces log message and rate limit trace too large. #4418 (@joe-elliott)
[CHANGE] Update the Open-Telemetry dependencies to v0.116.0 #4466 (@yvrhdn) BREAKING CHANGE After this update the Open-Telemetry Collector receiver will connect to localhost instead of all interfaces 0.0.0.0. Due to this, Tempo installations running inside Docker have to update the address they listen. For more details on this change, see #4465 For more information about the security risk this change addresses, see https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/security-best-practices.md#safeguards-against-denial-of-service-attacks
[CHANGE] BREAKING CHANGE By default, attributes are truncated to 2KB. #4335 (@ie-pham)
[CHANGE] Replace cespare/xxhash with cespare/xxhash/v2 #4485 (@Juneezee)
[CHANGE] chore: remove gofakeit dependency #4274 (@javiermolinar)
[CHANGE] Chore: delete spanlogger. 4312 (@javiermolinar)
[CHANGE] Update Azurite image. #4298 (@javiermolinar)
[CHANGE] Update golang.org/x/crypto #4474 (@javiermolinar)
[CHANGE] Distributor shim: add test verifying receiver works (including metrics) #4477 (@yvrhdn)
[CHANGE] Replace Grafana Agent example by Grafana Alloy#4030 (@javiermolinar)
[CHANGE] Update to the latest dskit #4341 (@dastrobu)
[FEATURE] tempo-cli: support dropping multiple traces in a single operation #4266 (@ndk)
[FEATURE] Discarded span logging log_discarded_spans #3957 (@dastrobu)
[FEATURE] TraceQL support for instrumentation scope #3967 (@ie-pham)
[FEATURE] Export cost attribution usage metrics from distributor #4162 (@mdisibio)
[FEATURE] TraceQL metrics: avg_over_time #4073 (@javiermolinar)
[FEATURE] Limit tags and tag values search #4320 (@javiermolinar)
[FEATURE] TraceQL metrics queries: add min_over_time #3975 (@javiermolinar)
[FEATURE] TraceQL metrics queries: add max_over_time #4065 (@javiermolinar)
[ENHANCEMENT] TraceQL: Add range condition for byte predicates #4198 (@ie-pham)
[ENHANCEMENT] Add throughput and SLO metrics in the tags and tag values endpoints #4148 (@electron0zero)
[ENHANCEMENT] BREAKING CHANGE Add maximum spans per span set. Users can set max_spans_per_span_set to 0 to obtain the old behavior. #4275 (@carles-grafana)
[ENHANCEMENT] Add query-frontend limit for max length of query expression #4397 (@electron0zero)
[ENHANCEMENT] distributor: return trace id length when it is invalid #4407 (@carles-grafana)
[ENHANCEMENT] Changed log level from INFO to DEBUG for the TempoDB Find operation using traceId to reduce excessive/unwanted logs in log search. #4179 (@Aki0x137)
[ENHANCEMENT] Pushdown collection of results from generators in the querier #4119 (@electron0zero)
[ENHANCEMENT] The span multiplier now also sources its value from the resource attributes. #4210 (@JunhoeKim)
[ENHANCEMENT] TraceQL: Attribute iterators collect matched array values #3867 (@electron0zero, @stoewer)
[ENHANCEMENT] Allow returning partial traces that exceed the MaxBytes limit for V2 #3941 (@javiermolinar)
[ENHANCEMENT] Added new middleware to validate request query values #3993 (@javiermolinar)
[ENHANCEMENT] Prevent massive allocations in the frontend if there is not sufficient pressure from the query pipeline. #3996 (@joe-elliott) BREAKING CHANGE Removed querier_forget_delay setting from the frontend. This configuration option did nothing.
[ENHANCEMENT] Update metrics-generator config in Tempo distributed docker compose example to serve TraceQL metrics #4003 (@javiermolinar)
[ENHANCEMENT] Reduce allocs related to marshalling dedicated columns repeatedly in the query frontend. #4007 (@joe-elliott)
[ENHANCEMENT] Improve performance of TraceQL queries #4114 (@mdisibio)
[ENHANCEMENT] Improve performance of TraceQL queries #4163 (@mdisibio)
[ENHANCEMENT] Improve performance of some TraceQL queries using select() operation #4438 (@mdisibio)
[ENHANCEMENT] Reduce memory usage of classic histograms in the span-metrics and service-graphs processors #4232 (@mdisibio)
[ENHANCEMENT] Implement simple Fetch by key for cache items #4032 (@javiermolinar)
[ENHANCEMENT] Write tenantindex as proto and json with a preference for proto #4072 (@zalegrala)
[ENHANCEMENT] Pool zstd encoding/decoding for tempodb/backend #4208 (@zalegrala)
[ENHANCEMENT] Send semver version in api/stattus/buildinfo for cloud deployments #4110 [@Aki0x137]
[ENHANCEMENT] Add completed block validation on startup.#4256 (@joe-elliott)
[ENHANCEMENT] Speedup DistinctString and ScopedDistinctString collectors #4109 (@electron0zero)
[ENHANCEMENT] Speedup collection of results from ingesters in the querier #4100 (@electron0zero)
[ENHANCEMENT] Speedup DistinctValue collector and exit early for ingesters #4104 (@electron0zero)
[ENHANCEMENT] Add disk caching in ingester SearchTagValuesV2 for completed blocks #4069 (@electron0zero)
[ENHANCEMENT] Add a max flush attempts and metric to the metrics generator #4254 (@joe-elliott)
[ENHANCEMENT] Collection of query-frontend changes to reduce allocs. #4242 (@joe-elliott)
[ENHANCEMENT] Added insecure-skip-verify option in tempo-cli to skip SSL certificate validation when connecting to the S3 backend. #4259 (@faridtmammadov)
[ENHANCEMENT] Add invalid_utf8 to reasons spanmetrics will discard spans. #4293 (@zalegrala)
[ENHANCEMENT] Reduce frontend and querier allocations by dropping HTTP headers early in the pipeline. #4298 (@joe-elliott)
[ENHANCEMENT] Reduce ingester working set by improving prelloc behavior. #4344,#4369 (@joe-elliott) Added tunable prealloc env vars PREALLOC_BKT_SIZE, PREALLOC_NUM_BUCKETS, PREALLOC_MIN_BUCKET and metric tempo_ingester_prealloc_miss_bytes_total to observe and tune prealloc behavior.
[ENHANCEMENT] Use Prometheus fast regexp for TraceQL regular expression matchers. #4329 (@joe-elliott) BREAKING CHANGE All regular expression matchers will now be fully anchored. span.foo =~ "bar" will now be evaluated as span.foo =~ "^bar$"
[ENHANCEMENT] Reuse generator code to better refuse "too large" traces. #4365 (@joe-elliott) This will cause the ingester to more aggressively and correctly refuse traces. Also added two metrics to better track bytes consumed per tenant in the ingester. tempo_metrics_generator_live_trace_bytes and tempo_ingester_live_trace_bytes.
[ENHANCEMENT] Reduce goroutines in all non-querier components. #4484 (@joe-elliott)
[ENHANCEMENT] Add option to enforce max span attribute size #4335 (@ie-pham)
[BUGFIX] Handle invalid TraceQL query filter in tag values v2 disk cache #4392 (@electron0zero)
[BUGFIX] Replace hedged requests roundtrips total with a counter. #4063 #4078 (@galalen)
[BUGFIX] Metrics generators: Correctly drop from the ring before stopping ingestion to reduce drops during a rollout. #4101 (@joe-elliott)
[BUGFIX] Correctly handle 400 Bad Request and 404 Not Found in gRPC streaming #4144 (@mapno)
[BUGFIX] Correctly handle Authorization header in gRPC streaming #4419 (@mdisibio)
[BUGFIX] Pushes a 0 to classic histogram's counter when the series is new to allow Prometheus to start from a non-null value. #4140 (@mapno)
[BUGFIX] Fix counter samples being downsampled by backdate to the previous minute the initial sample when the series is new #4236 (@javiermolinar)
[BUGFIX] Fix traceql metrics returning incorrect data for falsey queries and unscoped attributes #4409 (@mdisibio)
[BUGFIX] Fix traceql metrics time range handling at the cutoff between recent and backend data #4257 (@mdisibio)
[BUGFIX] Fix several issues with exemplar values for traceql metrics #4366 #4404 (@mdisibio)
[BUGFIX] Skip computing exemplars for instant queries. #4204 (@javiermolinar)
[BUGFIX] Gave context to orphaned spans related to various maintenance processes. #4260 (@joe-elliott)
[BUGFIX] Initialize histogram buckets to 0 to avoid downsampling. #4366 (@javiermolinar)
[BUGFIX] Utilize S3Pass and S3User parameters in tempo-cli options, which were previously unused in the code. #4259 (@faridtmammadov)
[BUGFIX] Fixed an issue in the generator where the first batch was counted 2x against a traces size. #4365 (@joe-elliott)
[BUGFIX] Fix compaction bug in SingleBinaryMode that could lead to 2x, 3x, etc TraceQL metrics results #4446 (@mdisibio)
[BUGFIX] Unstable compactors can occasionally duplicate data. Check for job ownership during compaction and cancel a job if ownership changes. #4420 (@joe-elliott)

v2.6.1

[CHANGE] BREAKING CHANGE tempo-query is no longer a Jaeger instance with grpcPlugin. It's now a standalone server. Serving a gRPC API for Jaeger on 0.0.0.0:7777 by default. #3840 (@frzifus)
[ENHANCEMENT] Register gRPC health server to tempo-query. #4178 (@frzifus)
[ENHANCEMENT] Support Tempo on IBM s390x. #4175 (@pavolloffay)
[ENHANCEMENT] tempo-query: Separate TLS settings for server and client. #4177 (@frzifus)
[ENHANCEMENT] Speedup tempo-query trace search by allowing parallel queries. #4159 (@pavolloffay)
[BUGFIX] Bring back application-json content-type header. #4123 (@javiermolinar)

v2.6.0

[CHANGE] BREAKING CHANGE Remove autocomplete_filtering_enabled feature flag #3729 (@mapno)
[CHANGE] Bump opentelemetry-collector to 0.102.1 #3784 (@debasishbsws)
[CHANGE] Bump Jaeger query docker image to 1.57.0 #3652 (@iblancasa)
[CHANGE] Update Go to 1.22.4 #3757 #3793 (@joe-elliott, @mapno)
[CHANGE] Make vParquet4 the default block encoding #3810 (@ie-pham)
[CHANGE] Azure v2 backend becomes the only and primary Azure backend #3875 (@zalegrala) BREAKING CHANGE The use_v2_sdk configuration option has been removed.
[FEATURE] TraceQL support for link scope and link:traceID and link:spanID #3741 (@stoewer)
[FEATURE] TraceQL support for link attribute querying #3814 (@ie-pham)
[FEATURE] TraceQL support for event scope and event:name intrinsic #3708 (@stoewer)
[FEATURE] TraecQL support for event attributes #3708 (@ie-pham)
[FEATURE] TraceQL support for event:timeSinceStart #3908 (@ie-pham)
[FEATURE] Autocomplete support for events and links #3846 (@ie-pham)
[FEATURE] Add new compare() metrics function #3695 (@mdisibio)
[FEATURE] Add new api /api/metrics/query for instant metrics queries #3859 (@mdisibio)
[FEATURE] Add a q parameter to /api/v2/serach/tags for tag name filtering #3822 (@joe-elliott)
[FEATURE] Add exemplars to TraceQL metrics #3824 (@mapno)
[FEATURE] Add support for generating native histograms from metrics-generator #3789 (@zalegrala, @kvrhdn)
[FEATURE] Flush and query RF1 blocks for TraceQL metric queries #3628 #3691 #3723 (@mapno) #3995 (@mdisibio)
[FEATURE] Added new middleware to block matching URLs #3963 (@javiermolinar)
[ENHANCEMENT] Implement arrays for traceql.Static with reused fields #3827 (@stoewer)
[ENHANCEMENT] Tag value lookup use protobuf internally for improved latency #3731 (@mdisibio)
[ENHANCEMENT] TraceQL metrics queries use protobuf internally for improved latency #3745 (@mdisibio)
[ENHANCEMENT] TraceQL search and other endpoints use protobuf internally for improved latency and resource usage #3944 (@mdisibio)
[ENHANCEMENT] Add local disk caching of metrics queries in local-blocks processor #3799 (@mdisibio)
[ENHANCEMENT] Improve use of OTEL semantic conventions on the service graph #3711 (@zalegrala)
[ENHANCEMENT] Performance improvement for rate() by () queries #3719 (@mapno)
[ENHANCEMENT] Performance improvement for queries using trace-level intrinsics #3920 (@mdisibio)
[ENHANCEMENT] Use multiple goroutines to unmarshal responses in parallel in the query frontend. #3713 (@joe-elliott)
[ENHANCEMENT] Protect ingesters from panics by adding defer/recover to all read path methods. #3790 (@joe-elliott)
[ENHANCEMENT] Added a boolean flag to enable or disable dualstack mode on Storage block config for S3 #3721 (@sid-jar, @mapno)
[ENHANCEMENT] Add caching to query range queries #3796 (@mapno)
[ENHANCEMENT] Only stream diffs on metrics queries #3808 (@joe-elliott)
[ENHANCEMENT] Add data quality metric to measure traces without a root #3812 (@mapno)
[ENHANCEMENT] Added an example for running Tempo vulture #3829 (@javiermolinar)
[ENHANCEMENT] Add a new helper method to allow debugging e2e tests #3836 (@javiermolinar)
[ENHANCEMENT] Refactor Tempo Vulture to reduce code complexity #3850 (@javiermolinar)
[ENHANCEMENT] Self document makefile #3844 (@javiermolinar)
[ENHANCEMENT] Added a Tempo CLI command to drop traces by id by rewriting blocks. #3856 (@joe-elliott)
[ENHANCEMENT] Mixin, make recording rule range interval configurable and increase range interval in alert to support scrape interval of 1 minute #3851 (@jmichalek132)
[ENHANCEMENT] Add vParquet4 support to the tempo-cli analyse blocks command #3868 (@stoewer)
[ENHANCEMENT] Improve trace id lookup from Tempo Vulture by selecting a date range #3874 (@javiermolinar)
[ENHANCEMENT] Add native histograms for internal metrics#3870 (@zalegrala)
[ENHANCEMENT] Expose availability-zone as a cli flag in ingester #3881 (@KyriosGN0)
[ENHANCEMENT] Rename batches property of Trace to ResourceSpans to be OTEL compatible #3895
[ENHANCEMENT] Reduce memory consumption of query-frontend#3888 (@joe-elliott)
[ENHANCEMENT] Reduce log level verbosity for e2e tests#3900 (@javiermolinar)
[ENHANCEMENT] Added new Traces api V2#3912 (@javiermolinar)
[ENHANCEMENT] Reduce allocs building queriers sharded requests #3932 (@javiermolinar)
[ENHANCEMENT] Allow compaction disablement per-tenant #3965 (@zalegrala)
[ENHANCEMENT] Implement polling tenants concurrently #3647 (@zalegrala)
[ENHANCEMENT] Reduce allocs of caching middleware #3976 (@joe-elliott)
[ENHANCEMENT] BlockMeta improvements to reduce the size #3950 #3951 #3952(@zalegrala)
[ENHANCEMENT] Add bytes and spans received to usage stats #3983 (@joe-elliott)
[BUGFIX] Fix panic in certain metrics queries using rate() with by #3847 (@stoewer)
[BUGFIX] Fix double appending the primary iterator on second pass with event iterator #3903 (@ie-pham)
[BUGFIX] Fix metrics queries when grouping by attributes that may not exist #3734 (@mdisibio)
[BUGFIX] Fix frontend parsing error on cached responses #3759 (@mdisibio)
[BUGFIX] Fix autocomplete of a query using scoped instrinsics #3865 (@mdisibio)
[BUGFIX] Fix metrics query histograms and quantiles on traceDuration #3879 (@mdisibio)
[BUGFIX] Fix divide by 0 bug in query frontend exemplar calculations #3936 (@mdisibio)
[BUGFIX] max_global_traces_per_user: take into account ingestion.tenant_shard_size when converting to local limit #3618 (@kvrhdn)
[BUGFIX] Fix http connection reuse on GCP and AWS by reading io.EOF through the http body. #3760 (@bmteller)
[BUGFIX] Improved handling of complete blocks in localblocks processor after enabling flusing #3805 (@mapno)
[BUGFIX] Handle out of boundaries spans kinds #3861 (@javiermolinar)
[BUGFIX] Maintain previous tenant blocklist on tenant errors #3860 (@zalegrala)
[BUGFIX] Fix prefix handling in Azure backend Find() call #3875 (@zalegrala)
[BUGFIX] BREAKING CHANGE Remove unused properties from the WAL configuration #3911 (@javiermolinar)
[BUGFIX] Bring back OTEL receiver metrics #3917 (@javiermolinar)
[BUGFIX] Correct block end time when the ingested traces are outside the ingestion slack #3954 (@javiermolinar)
[BUGFIX] Fix race condition where a streaming response could be marshalled while being modified in the combiner resulting in a panic. #3961 (@joe-elliott)
[BUGFIX] Pass search options to the backend for SearchTagValuesBlocksV2 requests #3971 (@javiermolinar)

v2.5.0

[CHANGE] Align metrics query time ranges to the step parameter #3490 (@mdisibio)
[CHANGE] Change the UID and GID of the tempo user to avoid root #2265 (@zalegrala) BREAKING CHANGE Ownership of /var/tempo is changing. Historically, this has been owned by root:root. With this change, it will now be owned by tempo:tempo with the UID/GID of 10001. The ingester and metrics-generator statefulsets may need to be chown'd in order to start properly. A jsonnet example of an init container is included with the PR. This impacts all users of the grafana/tempo Docker image.
[CHANGE] Remove vParquet encoding #3663 (@mdisibio) BREAKING CHANGE In the last release vParquet (the first version) was deprecated and blocked from writes. Now, it's removed entirely. It will no longer be recognized as a valid encoding and cannot read any remaining vParquet blocks. Installations running with historical defaults should not require any changes as the default has been migrated for several releases. Installations with storage settings pinned to vParquet must run a previous release configured for vParquet2 or higher until all existing vParquet (1) blocks have expired and been deleted from the backend, or else will encounter read errors after upgrading to this release.
[CHANGE] Return a less confusing error message to the client when refusing spans due to ingestion rates. #3485 (@ie-pham)
[CHANGE] Clean Metrics Generator's Prometheus wal before creating instance #3548 (@ie-pham)
[CHANGE] Update docker examples for permissions, deprecations, and clean-up #3603 (@zalegrala)
[CHANGE] Update debian and rpm packages to grant required permissions to default storage path after installation #3657 (@mdisibio)
[CHANGE] Delete any remaining objects for empty tenants after a configurable duration, requires config enable #3611 (@zalegrala)
[CHANGE] Add golangci to the tools image and update lint make target #3610 (@zalegrala)
[CHANGE] Update Alpine image version to 3.20 #3710 (@joe-elliott)
[FEATURE] Add TLS support for Memcached Client #3585 (@sonisr)
[FEATURE] TraceQL metrics queries: add quantile_over_time #3605 #3633 (@mdisibio)
[FEATURE] TraceQL metrics queries: add histogram_over_time #3644 (@mdisibio)
[FEATURE] Added gRPC streaming endpoints for Tempo APIs.
- Added gRPC streaming endpoints for all tag queries. #3460 (@joe-elliott)
- Added gRPC streaming endpoints for metrics. #3584 (@joe-elliott)
- Reduced memory consumption in the frontend for large traces. #3522 (@joe-elliott)
- Breaking Change Remove trace by id hedging from the frontend. #3522 (@joe-elliott)
- Breaking Change Dropped meta-tag for tenant from trace by id multitenant. #3522 (@joe-elliott)
[FEATURE] New block encoding vParquet4 with support for links, events, and arrays #3368 (@stoewer @ie-pham @andreasgerstmayr)
[ENHANCEMENT] Remove hardcoded delay in distributor shutdown #3687 (@chodges15)
[ENHANCEMENT] Tempo CLI - add percentage support for query blocks #3697 #3697 (@edgarkz)
[ENHANCEMENT] Update OTLP and add attributes to instrumentation scope in vParquet4 #3649 (@stoewer) Breaking Change The update to OTLP 1.3.0 removes the deprecated InstrumentationLibrary and InstrumentationLibrarySpan from the OTLP receivers
[ENHANCEMENT] Surface new labels for uninstrumented services and systems #3543 (@t00mas)
[ENHANCEMENT] Add querier metrics for requests executed #3524 (@electron0zero)
[ENHANCEMENT] Add messaging-system latency histogram to service-graph #3453 (@adirmatzkin)
[ENHANCEMENT] Add string interning to TraceQL queries #3411 (@mapno)
[ENHANCEMENT] Add new (unsafe) query hints for metrics queries #3396 (@mdisibio)
[ENHANCEMENT] Add nestedSetLeft/Right/Parent instrinsics to TraceQL. #3497 (@joe-elliott)
[ENHANCEMENT] Add tenant to frontend job cache key. #3527 (@joe-elliott)
[ENHANCEMENT] Better compaction throughput and memory usage #3579 (@mdisibio)
[ENHANCEMENT] Add support for sharded ingester queries #3574 (@zalegrala)
[ENHANCEMENT] TraceQL - Add support for scoped intrinsics using : #3629 (@ie-pham) available scoped intrinsics: trace:duration, trace:rootName, trace:rootService, span:duration, span:kind, span:name, span:status, span:statusMessage
[ENHANCEMENT] Performance improvements on TraceQL and tag value search. #3650,#3667 (@joe-elliott)
[ENHANCEMENT] TraceQL - Add support for trace:id and span:id #3670 (@ie-pham)
[ENHANCEMENT] Add toggle to inject the tenant ID to generated metrics #3638 (@kvrhdn)
[BUGFIX] Fix handling of regex matchers in autocomplete endpoints #3641 (@sd2k)
[BUGFIX] Update golang.org/x/net package to 0.24.0 to fix CVE-2023-45288 #3613 (@pavolloffay)
[BUGFIX] Fix metrics query results when filtering and rating on the same attribute #3428 (@mdisibio)
[BUGFIX] Fix metrics query results when series contain empty strings or nil values #3429 (@mdisibio)
[BUGFIX] Fix metrics query duration check, add per-tenant override for max metrics query duration #3479 (@mdisibio)
[BUGFIX] Fix metrics query panic "index out of range [-1]" when a trace has zero-length ID #3668 (@mdisibio)
[BUGFIX] Return unfiltered results when a bad TraceQL query is provided in autocomplete. #3426 (@mapno)
[BUGFIX] Add support for dashes, quotes and spaces in attribute names in autocomplete #3458 (@mapno)
[BUGFIX] Correctly handle 429s in GRPC search streaming. #3469 (@joe-ellitot)
[BUGFIX] Correctly cancel GRPC and HTTP contexts in the frontend to prevent having to rely on http write timeout. #3443 (@joe-elliott)
[BUGFIX] Add spss and limit to the frontend cache key to prevent the return of incorrect results. #3557 (@joe-elliott)
[BUGFIX] Use os path separator to split blocks path. #3552 (@teyyubismayil)
[BUGFIX] Correctly parse traceql queries with > 1024 character attribute names or static values. #3571 (@joe-elliott)
[BUGFIX] Fix span-metrics' subprocessors bug that applied wrong configs when running multiple tenants. #3612 (@mapno)
[BUGFIX] Fix panic in query-frontend when combining results #3683 (@mapno)
[BUGFIX] Fix panic in metrics-generator when starting with a partial completed block #3704 (@zalegrala)
[BUGFIX] Fix TraceQL queries involving non boolean operations between statics and attributes. #3698 (@joe-elliott)

v2.4.2

[BUGFIX] Update golang.org/x/net package to 0.24.0 to fix CVE-2023-45288 #3613 (@pavolloffay)

v2.4.1

[BUGFIX] Fix compaction/retention in AWS S3 and GCS when a prefix is configured. #3465 (@bpfoster)

v2.4.0

[CHANGE] Merge the processors overrides set through runtime overrides and user-configurable overrides #3125 (@kvrhdn)
[CHANGE] Make vParquet3 the default block encoding #2526 (@stoewer)
[CHANGE] Set autocomplete_filtering_enabled to true by default #3178 (@mapno)
[CHANGE] Update Alpine image version to 3.19 #3289 (@zalegrala)
[CHANGE] Breaking Change Fix issue where tempo drops the entire batch if one trace triggers an error #2571 (@ie-pham) Distributor now returns 200 for any batch containing only trace_too_large and max_live_traces errors The number of discarded spans are still reflected in the tempo_discarded_spans_total metrics
[CHANGE] Remove experimental websockets support for search streaming. GRPC is the supported method of streaming results #3307 (@joe-elliott)
[CHANGE] Breaking Change Deprecating vParquet v1 #3377 (@ie-pham)
[FEATURE] TraceQL metrics queries #3227 #3252 #3258 (@mdisibio @zalegrala)
[FEATURE] Add support for multi-tenant queries. #3087 (@electron0zero)
[FEATURE] Major cache refactor to allow multiple role based caches to be configured #3166. BREAKING CHANGE Deprecate the following fields. These have all been migrated to a top level "cache:" field.
```
storage:
  trace:
    cache:
    search:
      cache_control:
    background_cache:
    memcached:
    redis:
```
[ENHANCEMENT] Add support for multi-tenant queries in streaming search #3262 (@electron0zero)
[ENHANCEMENT] Add configuration on tempo-query plugin for fetch services older than complete_block_timeout #3262 (@rubenvp8510)
[ENHANCEMENT] Add tracing integration to profiling endpoints #3276 (@cyriltovena)
[ENHANCEMENT] Introduced AttributePolicyMatch & IntrinsicPolicyMatch structures to match span attributes based on strongly typed values & precompiled regexp #3025 (@andriusluk)
[ENHANCEMENT] Make the trace ID label name configurable for remote written exemplars #3074
[ENHANCEMENT] Update poller to make use of previous results and reduce backend load. #2652 (@zalegrala)
[ENHANCEMENT] Improve TraceQL regex performance in certain queries. #3139 (@joe-elliott)
[ENHANCEMENT] Improve TraceQL performance in complex queries. #3113 (@joe-elliott)
[ENHANCEMENT] Added a frontend-search cache role for job search caching. #3225 (@joe-elliott)
[ENHANCEMENT] Added a parquet-page cache role for page level caching. #3196 (@joe-elliott)
[ENHANCEMENT] Update opentelemetry-collector-contrib dependency to the latest version, v0.89.0 #3148 (@gebn)
[ENHANCEMENT] Add --max-start-time and --min-start-time flag to tempo-cli command analyse blocks #3250 (@mapno)
[ENHANCEMENT] Add per-tenant configurable remote_write headers to metrics-generator #3175 (@mapno)
[ENHANCEMENT] Add variable expansion support to overrides configuration #3175 (@mapno)
[ENHANCEMENT] Update memcached default image in jsonnet for multiple CVE #3310 (@zalegrala)
[ENHANCEMENT] Add HTML pages /status/overrides and /status/overrides/{tenant} #3244 #3332 (@kvrhdn)
[ENHANCEMENT] Precalculate and reuse the vParquet3 schema before opening blocks #3367 (@stoewer)
[ENHANCEMENT] Config: Adds query-frontend.log-query-request-headers to enable logging of request headers in query logs. #3383 (@jmichalek132)
[ENHANCEMENT] Add --shutdown-delay to allow Tempo to cleanly drain connections. #3395 (@joe-elliott)
[ENHANCEMENT] Introduce localblocks process config option to select only server spans 3303https://github.com/grafana/tempo/pull/3303 (@zalegrala)
[ENHANCEMENT] TraceQL/Structural operators performance improvement. #3088 (@joe-elliott)
[ENHANCEMENT] Localblocks processor honor tenant max trace size limit 3305 (@mdisibio)
[ENHANCEMENT] Introduce list_blocks_concurrency on GCS and S3 backends to control backend load and performance. #2652 (@zalegrala)
[ENHANCEMENT] Add per-tenant compaction window #3129 (@zalegrala)
[BUGFIX] Fix parsing of span.resource.xyz attributes in TraceQL. #3284 (@mghildiy)
[BUGFIX] Change exit code if config is successfully verified #3174 (@am3o @agrib-01)
[BUGFIX] The tempo-cli analyse blocks command no longer fails on compacted blocks #3183 (@stoewer)
[BUGFIX] Move waitgroup handling for poller error condition #3224 (@zalegrala)
[BUGFIX] Fix head block excessive locking in ingester search #3328 (@mdisibio)
[BUGFIX] Fix issue with ingester failed to cut traces no such file or directory #3346 (@mdisibio)
[BUGFIX] Restore tempo_request_duration_seconds metrics for querier_api_* requests #3403 (@kvrhdn)
[BUGFIX] Prevent building parquet iterators that would loop forever. #3159 (@mapno)
[BUGFIX] Sanitize name in mapped dimensions in span-metrics processor #3171 (@mapno)
[BUGFIX] Fixed an issue where cached footers were requested then ignored. #3196 (@joe-elliott)
[BUGFIX] Fix panic in autocomplete when query condition had wrong type #3277 (@mapno)
[BUGFIX] Fix TLS when GRPC is enabled on HTTP #3300 (@joe-elliott)
[BUGFIX] Correctly return 400 when max limit is requested on search. #3340 (@joe-elliott)
[BUGFIX] Fix autocomplete filters sometimes returning erroneous results. #3339 (@joe-elliott)
[BUGFIX] Fixes trace context propagation between query-frontend and querier. #3387 (@mapno)
[BUGFIX] Fix some instances where spanmetrics histograms could be inconsistent #3412 (@mdisibio)

v2.3.1 / 2023-11-28

[BUGFIX] Include statusMessage intrinsic attribute in tag search. #3084 (@rcrowe)
[BUGFIX] Fix compactor ignore configured S3 headers #3149 (@Batkilin)
[BUGFIX] Readd session token to s3 credentials. #3144 (@farodin91)

v2.3.0 / 2023-10-30

[CHANGE] Update Go to 1.21 #2486 (@zalegrala)
[CHANGE] Moved the tempo_ingester_traces_created_total metric to be incremented when a trace is cut to the wal #2884 (@joe-elliott)
[CHANGE] Upgrade from deprecated azure-storage-blob-go SDK to azure-sdk-for-go #2835 (@LasseHels)
[CHANGE] Metrics summary API validate the requested time range #2902 (@mdisibio)
[CHANGE] Restructure Azure backends into versioned backends. Introduce use_v2_sdk config option for switching. #2952 (@zalegrala) v1: azure-storage-blob-go original (now deprecated) SDK v2: azure-sdk-for-go
[CHANGE] Adjust trace size estimation to better honor row group size settings. #3038 (@joe-elliott)
[CHANGE] Update alpine image version to 3.18 to patch CVE-2022-48174. #3046 (@joe-elliott)
[CHANGE] Overrides module refactor #2688 (@mapno) Added new defaults block to the overrides' module. Overrides change to indented syntax. Old config:

overrides:
  ingestion_rate_strategy: local
  ingestion_rate_limit_bytes: 12345
  ingestion_burst_size_bytes: 67890
  max_search_duration: 17s
  forwarders: ['foo']
  metrics_generator_processors: [service-graphs, span-metrics]

New config:

overrides:
defaults:
  ingestion:
    rate_strategy: local
    rate_limit_bytes: 12345
    burst_size_bytes: 67890
  read:
    max_search_duration: 17s
  forwarders: ['foo']
  metrics_generator:
    processors: [service-graphs, span-metrics]

[CHANGE] Bump Jaeger query docker image to 1.50.0 #2998 (@pavolloffay)
[FEATURE] New TraceQL structural operators ancestor (<<), parent (<) #2877 (@kousikmitra)
[FEATURE] Add the /api/status/buildinfo endpoint #2702 (@fabrizio-grafana)
[FEATURE] New encoding vParquet3 with support for dedicated attribute columns (@mapno, @stoewer) #2649
[FEATURE] Add filtering support to Generic Forwarding #2742 (@Blinkuu)
[FEATURE] Add cli command to print out summary of large traces #2775 (@ie-pham)
[FEATURE] Added not structural operators to TraceQL: !>, !<, and !~ #2993 (@joe-elliott)
[ENHANCEMENT] Make metrics-generator ingestion slack per tenant #2589 (@ie-pham)
[ENHANCEMENT] Support quoted attribute name in TraceQL #3004 (@kousikmitra)
[ENHANCEMENT] Add support for searching by span status message using statusMessage keyword #2848 (@kousikmitra)
[ENHANCEMENT] Add block indexes to vParquet2 and vParquet3 to improve trace by ID lookup #2697 (@mdisibio)
[ENHANCEMENT] Assert ingestion rate limits as early as possible #2640 (@mghildiy)
[ENHANCEMENT] Add several metrics-generator fields to user-configurable overrides #2711 (@kvrhdn)
[ENHANCEMENT] Update /api/metrics/summary to correctly handle missing attributes and improve performance of TraceQL select() queries. #2765 (@mdisibio)
[ENHANCEMENT] Tempo CLI command to convert from vParquet2 -> 3. #2828 (@joe-elliott)
[ENHANCEMENT] Add TempoUserConfigurableOverridesReloadFailing alert #2784 (@kvrhdn)
[ENHANCEMENT] Add RootSpanName and RootServiceName to log about discarded spans #2816 (@marcinginszt)
[ENHANCEMENT] Add UserID to log message about rate limiting #2850 (@lshippy)
[ENHANCEMENT] Requests to Azure Blob Storage will now be retried once instead of zero times #2835 (@LasseHels)
[ENHANCEMENT] Add span metrics filter policies to user-configurable overrides #2906 (@rlankfo)
[ENHANCEMENT] Add collection-interval to metrics-generator config in user-configurable overrides #2899 (@rlankfo)
[ENHANCEMENT] Enforce max trace size on the trace by id path. #2935 (@joe-elliott)
[ENHANCEMENT] Add target_info_excluded_dimensions to user-config api #2945 (@ie-pham)
[ENHANCEMENT] User-configurable overrides: add scope query parameter to return merged overrides for tenant #2915 #3018 (@kvrhdn)
[ENHANCEMENT] Add histogram buckets to metrics-generator config in user-configurable overrides #2928 (@mar4uk)
[ENHANCEMENT] Adds websocket support for search streaming. #2971 (@joe-elliott)
[ENHANCEMENT] Add new config block to distributors to produce debug metrics. #3008 (@joe-elliott) Breaking Change Removed deprecated config option: distributor.log_received_traces
[ENHANCEMENT] added a metrics generator config option to enable/disable X-Scope-OrgID headers on remote write. #2974 (@vineetjp)
[ENHANCEMENT] Correctly return RetryInfo to Otel Collector/Grafana Agent on ResourceExhausted. This allows the agents to honor their own retry settings. #3019 (@joe-elliott)
[BUGFIX] Unescape tag names #2894 (@fabrizio-grafana)
[BUGFIX] Load defaults for the internal server #3041 (@rubenvp8510)
[BUGFIX] Fix pass-through to runtime overrides for FilterPolicies and TargetInfoExcludedDimensions #3012 (@electron0zero)
[BUGFIX] Fix panic in metrics summary api #2738 (@mdisibio)
[BUGFIX] Fix rare deadlock when uploading blocks to Azure Blob Storage #2129 (@LasseHels)
[BUGFIX] Only search ingester blocks that fall within the request time range. #2783 (@joe-elliott)
[BUGFIX] Align tempo_query_frontend_queries_total and tempo_query_frontend_queries_within_slo_total. #2840 (@joe-elliott) This query will now correctly tell you %age of requests that are within SLO:
```
sum(rate(tempo_query_frontend_queries_within_slo_total{}[1m])) by (op)
/
sum(rate(tempo_query_frontend_queries_total{}[1m])) by (op)
```
BREAKING CHANGE Removed: tempo_query_frontend_queries_total{op="searchtags|metrics"}.
[BUGFIX] To support blob storage in Azure Stack Hub as backend. #2853 (@chlislb)
[BUGFIX] Respect spss on GRPC streaming. #2971 (@joe-elliott)
[BUGFIX] Moved empty root span substitution from querier to query-frontend. #2671 (@galalen)
[BUGFIX] Correctly propagate ingester errors on the query path #2935 (@joe-elliott)
[BUGFIX] Fix issue where ingester doesn't stop query after timeout #3031 (@mdisibio)
[BUGFIX] Fix cases where empty filter {} wouldn't return expected results #2498 (@mdisibio)
[BUGFIX] Reorder S3 credential chain and upgrade minio-go. native_aws_auth_enabled is deprecated #3006 (@ekristen, @mapno)
[BUGFIX] Update parquet-go dependency including a bugfix that prevents corrupted blocks from being written #3068 (@stoewer)

v2.2.4 / 2023-10-25

[CHANGE] Update alpine image version to 3.18 to patch CVE-2022-48174. #3046 (@joe-elliott)
[CHANGE] Bump Jaeger query docker image to 1.50.0 #2998 (@pavolloffay)

v2.2.3 / 2023-09-13

[BUGFIX] Fix S3 credentials providers configuration #2889 (@mapno)

v2.2.2 / 2023-08-30

[BUGFIX] Fix node role auth IDMSv1 #2760 (@coufalja)

v2.2.1 / 2023-08-21

[BUGFIX] Fix incorrect metrics for index failures #2781 (@zalegrala)
[BUGFIX] Fix panic in the metrics-generator when using multiple tenants with default overrides #2786 (@kvrhdn)
[BUGFIX] Restore tenant_header_key removed in #2414. #2795 (@joe-elliott)
[BUGFIX] Disable streaming over http by default. #2803 (@joe-elliott)

v2.2.0 / 2023-07-31

[CHANGE] Make vParquet2 the default block format #2526 (@stoewer)
[CHANGE] Disable tempo-query by default in Jsonnet libs. #2462 (@electron0zero)
[CHANGE] Integrate gofumpt into CI for formatting requirements 2584 (@zalegrala)
[CHANGE] Change log level of two compactor messages from debug to info. #2443 (@dylanguedes)
[CHANGE] Remove tenant_header_key option from tempo-query config #2414 (@kousikmitra)
[CHANGE] Breaking Change Remove support tolerate_failed_blocks. #2416 (@joe-elliott) Removed config option:
```
query_frontend:
  tolerate_failed_blocks: <int>
```
[CHANGE] Upgrade memcached version in jsonnet microservices #2466 (@zalegrala)
[CHANGE] Prefix service graph extra dimensions labels with server_ and client_ if enable_client_server_prefix is enabled #2335 (@domasx2)
[CHANGE] Breaking Change Rename s3.insecure_skip_verify #2407 (@zalegrala)

storage:
  trace:
    s3:
      insecure_skip_verify: true   // renamed to tls_insecure_skip_verify

[CHANGE] Ignore context canceled errors in the queriers #2440 (@joe-elliott)
[CHANGE] Start flush queue worker after wal replay and block rediscovery #2456 (@ie-pham)
[CHANGE] Update Go to 1.20.4 #2486 (@ie-pham)
[CHANGE] Breaking Change Convert metrics generator from deployment to a statefulset in jsonnet. Refer to the PR for seamless migration instructions. #2533 #2467 (@zalegrala)
[FEATURE] New experimental API to derive on-demand RED metrics grouped by any attribute, and new metrics generator processor #2368 #2418 #2424 #2442 #2480 #2481 #2501 #2579 #2582 (@mdisibio @zalegrala)
[FEATURE] New TraceQL structural operators descendant (>>), child (>), and sibling (~) #2625 #2660 (@mdisibio)
[FEATURE] Add user-configurable overrides module #2543 #2682 #2681 (@electron0zero @kvrhdn)
[FEATURE] Add support for q query param in /api/v2/search/<tag.name>/values to filter results based on a TraceQL query #2253 (@mapno) To make use of filtering, configure autocomplete_filtering_enabled.
[FEATURE] Add support for by() and coalesce() to TraceQL. #2490
[FEATURE] Add a GRPC streaming endpoint for traceql search #2366 (@joe-elliott)
[FEATURE] Add new API to summarize span metrics from generators #2481 (@zalegrala)
[FEATURE] Add select() to TraceQL #2494 (@joe-elliott)
[FEATURE] Add traceDuration, rootName and rootServiceName intrinsics to TraceQL #2503 (@joe-elliott)
[ENHANCEMENT] Add support for query batching between frontend and queriers to improve throughput #2677 (@joe-elliott)
[ENHANCEMENT] Add initial RBAC support for serverless backend queries, limited to Google CloudRun #2487 (@modulitos)
[ENHANCEMENT] Add capability to flush all remaining traces to backend when ingester is stopped #2538
[ENHANCEMENT] Fill parent ID column and nested set columns #2487 (@stoewer)
[ENHANCEMENT] Add metrics generator config option to allow customizable ring port #2399 (@mdisibio)
[ENHANCEMENT] Improve performance of TraceQL regex #2484 (@mdisibio)
[ENHANCEMENT] log client ip to help identify which client is no org id #2436
[ENHANCEMENT] Add spss parameter to /api/search/tags[#2308] to configure the spans per span set in response
[ENHANCEMENT] Continue polling tenants on error with configurable threshold #2540 (@mdisibio)
[ENHANCEMENT] Fully skip over parquet row groups with no matches in the column dictionaries #2676 (@mdisibio)
[ENHANCEMENT] Add prefix configuration option to storage.trace.azure and storage.trace.gcs #2362 (@kousikmitra)
[ENHANCEMENT] Add support to filter using negated regex operator !~ #2410 (@kousikmitra)
[ENHANCEMENT] Add prefix configuration option to storage.trace.azure and storage.trace.gcs #2386 (@kousikmitra)
[ENHANCEMENT] Add prefix configuration option to storage.trace.s3 #2362 (@kousikmitra)
[ENHANCEMENT] Add support for concurrent_shards under trace_by_id #2416 (@joe-elliott)
```
query_frontend:
  trace_by_id:
    concurrent_shards: 3
```

[ENHANCEMENT] Enable cross cluster querying by adding two config options. #2598 (@joe-elliott)

querier:
  secondary_ingester_ring: <string>
metrics_generator:
  override_ring_key: <string>

[ENHANCEMENT] Add scope parameter to /api/search/tags #2282 (@joe-elliott) Create new endpoint /api/v2/search/tags that returns all tags organized by scope.
[ENHANCEMENT] Ability to toggle off latency or count metrics in metrics-generator #2070 (@AlexDHoffer)
[ENHANCEMENT] Extend /flush to support flushing a single tenant #2260 (@kvrhdn)
[ENHANCEMENT] Add override to limit number of blocks inspected in tag value search #2358 (@mapno)
[ENHANCEMENT] New synchronous read mode for vParquet and vParquet2 #2165 #2535 (@mdisibio)
[ENHANCEMENT] Add option to override metrics-generator ring port #2399 (@mdisibio)
[ENHANCEMENT] Add support for IPv6 #1555 (@zalegrala)
[ENHANCEMENT] Add span filtering to spanmetrics processor #2274 (@zalegrala)
[ENHANCEMENT] Add ability to detect virtual nodes in the servicegraph processor #2365 (@mapno)
[ENHANCEMENT] Introduce overrides.Interface to decouple implementation from usage #2482 (@kvrhdn)
[ENHANCEMENT] Improve TraceQL throughput by asynchronously creating jobs #2530 (@joe-elliott)
[BUGFIX] Fix Search SLO by routing tags to a new handler. #2468 (@electron0zero)
[BUGFIX] tempodb integer divide by zero error #2167 (@kroksys)
[BUGFIX] metrics-generator: ensure Prometheus will scale up shards when remote write is lagging behind #2463 (@kvrhdn)
[BUGFIX] Fixes issue where matches and other spanset level attributes were not persisted to the TraceQL results. #2490
[BUGFIX] Fixes issue where ingester search could occasionally fail with file does not exist error #2534 (@mdisibio)
[BUGFIX] Tempo failed to find meta.json path after adding prefix in S3/GCS/Azure configuration. #2585 (@WildCatFish)
[BUGFIX] Delay logging config warnings until the logger has been initialized #2645 (@kvrhdn)
[BUGFIX] Fix issue where metrics-generator was setting wrong labels for traces_target_info #2546 (@ie-pham)
[FEATURE] Add tempo-cli commands analyse block and analyse blocks to analyse parquet blocks and output summaries of generic attribute columns #2622 (@mapno)

v2.1.1 / 2023-04-28

[BUGFIX] Fix issue where Tempo sometimes flips booleans from false->true at storage time. #2400 (@joe-elliott)

v2.1.0 / 2023-04-26

[CHANGE] Capture and update search metrics for TraceQL #2087 (@electron0zero)
[CHANGE] tempo-mixin: disable auto refresh every 10 seconds #2290 (@electron0zero)
[CHANGE] Update tempo-mixin to show request in Resources dashboard #2281 (@electron0zero)
[CHANGE] Add support for s3 session token in static config #2093 (@farodin91)
[CHANGE] Breaking Change Remove support for search on v2 blocks. #2159 (@joe-elliott) Removed config options:
```
overrides:
  max_search_bytes_per_trace:
  search_tags_allow_list:
  search_tags_deny_list:
```
Removed metrics: tempo_ingester_trace_search_bytes_discarded_total
[CHANGE] Stop caching parquet files for search #2164 (@mapno)
[CHANGE] Update Go to 1.20 #2079 (@scalalang2)
[CHANGE] BREAKING CHANGE Change metrics prefixed with cortex_ to tempo_ #2204 (@mapno)
[CHANGE] Upgrade OTel to v0.74.0 #2317 (@mapno)
[FEATURE] New parquet based block format vParquet2 #2244 (@stoewer)
[FEATURE] Add support for Azure Workload Identity authentication #2195 (@LambArchie)
[FEATURE] Add flag to check configuration #2131 (@robertscherbarth @agrib-01)
[FEATURE] Add flag to optionally enable all available Go runtime metrics #2005 (@andreasgerstmayr)
[FEATURE] Add support for span kind to TraceQL #2217 (@joe-elliott)
[FEATURE] Add support for min/max/avg aggregates to TraceQL#2255 (@joe-elliott)
[ENHANCEMENT] Add Throughput and SLO Metrics with SLOConfig in Query Frontend #2008 (@electron0zero)
- BREAKING CHANGE query_frontend_result_metrics_inspected_bytes metric removed in favour of query_frontend_bytes_processed_per_second
[ENHANCEMENT] Metrics generator to make use of counters earlier #2068 (@zalegrala)
[ENHANCEMENT] Log when a trace is too large to compact #2105 (@scalalang2)
[ENHANCEMENT] Add support for arbitrary arithemtic to TraceQL queries #2146 (@joe-elliott)
[ENHANCEMENT] tempo-cli: add command to migrate a tenant #2130 (@kvrhdn)
[ENHANCEMENT] Added the ability to multiple span metrics by an attribute such as X-SampleRatio #2172 (@altanozlu)
[BUGFIX] Correctly connect context during compaction #2220 (@ie-pham)
[BUGFIX] Apply rate() to bytes/s panel in tenant's dashboard. #2081 (@mapno)
[BUGFIX] Retry copy operations during compaction in GCS backend #2111 (@mapno)
[BUGFIX] Fix float/int comparisons in TraceQL. #2139 (@joe-elliott)
[BUGFIX] Improve locking and search head block in SearchTagValuesV2 #2164 (@mapno)
[BUGFIX] Fix not closing WAL block file before attempting to delete the folder. #2139 (@kostya9)
[BUGFIX] Stop searching for virtual tags if there are any hits. This prevents invalid values from showing up for intrinsics like status #2219 (@joe-elliott)
[BUGFIX] Correctly return unique spans when &&ing and ||ing spansets. #2254 (@joe-elliott)
[BUGFIX] Support negative values on aggregate filters like count() > -1. #2289 (@joe-elliott)
[BUGFIX] Support float as duration like {duration > 1.5s} #2304 (@ie-pham)
[ENHANCEMENT] Supports range operators for strings in TraceQL #2321 (@ie-pham)
[ENHANCEMENT] Supports TraceQL in Vulture #2321 (@ie-pham)
[FEATURE] Add job & instance labels to span metrics, a new target_info metrics, and custom dimension label mapping #2261 (@ie-pham)

v2.0.1 / 2023-03-03

[CHANGE] No longer return status.code from /api/search/tags unless it is an attribute present in the data #2059 (@mdisibio)
[BUGFIX] Suppress logspam in single binary mode when metrics generator is disabled. #2058 (@joe-elliott)
[BUGFIX] Error more gracefully while reading some blocks written by an interim commit between 1.5 and 2.0 #2055 (@mdisibio)
[BUGFIX] Correctly coalesce trace level data when combining Parquet traces. #2095 (@joe-elliott)
[BUGFIX] Unescape query parameters in AWS Lambda to allow TraceQL queries to work. #2114 (@joe-elliott)
[CHANGE] Pad leading zeroes in span id to always be 16 chars #2062 (@ie-pham)

v2.0.0 / 2023-01-31

[CHANGE] BREAKING CHANGE Use snake case on Azure Storage config #1879 (@faustodavid) Example of using snake case on Azure Storage config:

# config.yaml
storage:
  azure:
    storage_account_name:
    storage_account_key:
    container_name:

[CHANGE] Increase default values for server.grpc_server_max_recv_msg_size and server.grpc_server_max_send_msg_size from 4MB to 16MB #1688 (@mapno)
[CHANGE] Propagate Ingesters search errors correctly #2023 (@electron0zero)
[CHANGE] BREAKING CHANGE Use storage.trace.block.row_group_size_bytes to cut rows during compaction instead of compactor.compaction.flush_size_bytes. #1696 (@joe-elliott)
[CHANGE] Update Go to 1.19 #1665 (@ie-pham)
[CHANGE] Remove unsued scheduler frontend code #1734 (@mapno)
[CHANGE] Deprecated query-frontend.query_shards in favor of query_frontend.trace_by_id.query_shards. Old config will still work but will be removed in a future release. #1735 (@mapno)
[CHANGE] Update alpine image version to 3.16. #1784 (@zalegrala)
[CHANGE] Delete TempoRequestErrors alert from mixin #1810 (@zalegrala)
- BREAKING CHANGE Any jsonnet users relying on this alert should copy this into their own environment.
[CHANGE] Update and replace a few go modules #1945 (@zalegrala)
- Replace github.com/thanos-io/thanos/pkg/discovery/dns use with github.com/grafana/dskit/dns
- Upgrade github.com/grafana/dskit
- Upgrade github.com/grafana/e2e
- Upgrade github.com/minio/minio-go/v7

[CHANGE] Config updates to prepare for Tempo 2.0. #1978 (@joe-elliott) Defaults updated:

query_frontend:
  max_oustanding_per_tenant: 2000
  search:
      concurrent_jobs: 1000
      target_bytes_per_job: 104857600
      max_duration: 168h
      query_ingesters_until: 30m
  trace_by_id:
      query_shards: 50
querier:
    max_concurrent_queries: 20
    search:
        prefer_self: 10
ingester:
    concurrent_flushes: 4
    max_block_duration: 30m
    max_block_bytes: 524288000
storage:
    trace:
        pool:
            max_workers: 400
            queue_depth: 20000
        search:
            read_buffer_count: 32
            read_buffer_size_bytes: 1048576

BREAKING CHANGE Renamed/removed/moved

query_frontend:
  query_shards:                  // removed. use trace_by_id.query_shards
querier:
    query_timeout:               // removed. use trace_by_id.query_timeout
compactor:
    compaction:
        chunk_size_bytes:        // renamed to v2_in_buffer_bytes
        flush_size_bytes:        // renamed to v2_out_buffer_bytes
        iterator_buffer_size:    // renamed to v2_prefetch_traces_count
ingester:
    use_flatbuffer_search:       // removed. automatically set based on block type
storage:
    wal:
        encoding:                // renamed to v2_encoding
        version:                 // removed and pinned to block.version
    block:
        index_downsample_bytes:  // renamed to v2_index_downsample_bytes
        index_page_size_bytes:   // renamed to v2_index_page_size_bytes
        encoding:                // renamed to v2_encoding
        row_group_size_bytes:    // renamed to parquet_row_group_size_bytes

[CHANGE] BREAKING CHANGE Remove search_enabled and metrics_generator_enabled. Both default to true. #2004 (@joe-elliott)
[CHANGE] Update OTel collector to v0.57.2 #1757 (@mapno)
[FEATURE] TraceQL support https://grafana.com/docs/tempo/latest/traceql/
[FEATURE] Parquet backend is GA and default
[FEATURE] Add generic forwarder and implement otlpgrpc forwarder #1775 (@Blinkuu) New config options and example configuration:

# config.yaml
distributor:
  forwarders:
    - name: "otel-forwarder"
      backend: "otlpgrpc"
      otlpgrpc:
        endpoints: ['otelcol:4317']
        tls:
          insecure: true

# overrides.yaml
overrides:
  "example-tenant-1":
    forwarders: ['otel-forwarder']
  "example-tenant-2":
    forwarders: ['otel-forwarder']

[ENHANCEMENT] Add support for TraceQL in Parquet WAL and Local Blocks. #1966 (@electron0zero)
[ENHANCEMENT] Add /status/usage-stats endpoint to show usage stats data #1782 (@electron0zero)
[ENHANCEMENT] Add TLS support to jaeger query plugin. #1999 (@rubenvp8510)
[ENHANCEMENT] Collect inspectedBytes from SearchMetrics #1975 (@electron0zero)
[ENHANCEMENT] Add zone awareness replication for ingesters. #1936 (@manohar-koukuntla)

# use the following fields in _config field of jsonnet config, to enable zone aware ingester
    multi_zone_ingester_enabled: false,
    multi_zone_ingester_migration_enabled: false,
    multi_zone_ingester_replicas: 0,
    multi_zone_ingester_max_unavailable: 25,

[ENHANCEMENT] Support global and wildcard overrides in generic forwarder feature #1871 (@Blinkuu)
[ENHANCEMENT] Add new data-type aware searchtagvalues v2 api #1956 (@mdisibio)
[ENHANCEMENT] Refactor queueManager into generic queue.Queue #1796 (@Blinkuu)
- BREAKING CHANGE Rename tempo_distributor_forwarder_queue_length metric to tempo_distributor_queue_length. New metric has two custom labels: name and tenant.
- Deprecated tempo_distributor_forwarder_pushes_total metric in favor of tempo_distributor_queue_pushes_total.
- Deprecated tempo_distributor_forwarder_pushes_failures_total metric in favor of tempo_distributor_queue_pushes_failures_total.
[ENHANCEMENT] Filter namespace by cluster in tempo dashboards variables #1771 (@electron0zero)
[ENHANCEMENT] Exit early from sharded search requests #1742 (@electron0zero)
[ENHANCEMENT] Upgrade prometheus/prometheus to 51a44e6657c3 #1829 (@mapno)
[ENHANCEMENT] Avoid running tempodb pool jobs with a cancelled context #1852 (@zalegrala)
[ENHANCEMENT] Add config flag to allow for compactor disablement for debug purposes #1850 (@zalegrala)
[ENHANCEMENT] Identify bloom that could not be retrieved from backend block #1737 (@AlexDHoffer)
[ENHANCEMENT] tempo: check configuration returns now a list of warnings #1663 (@frzifus)
[ENHANCEMENT] Make DNS address fully qualified to reduce DNS lookups in Kubernetes #1687 (@electron0zero)
[ENHANCEMENT] Improve parquet compaction memory profile when dropping spans #1692 (@joe-elliott)
[ENHANCEMENT] Use Parquet for local block search, tag search and tag value search instead of flatbuffers. A configuration value (ingester.use_flatbuffer_search) is provided to continue using flatbuffers.
- BREAKING CHANGE Makes Parquet the default encoding.
[ENHANCEMENT] Return 200 instead of 206 when blocks failed is < tolerate_failed_blocks. #1725 (@joe-elliott)
[ENHANCEMENT] Add GOMEMLIMIT variable to compactor jsonnet and set the value to equal compactor memory limit. #1758 (@ie-pham)
[ENHANCEMENT] Add capability to configure the used S3 Storage Class #1697 (@amitsetty)
[ENHANCEMENT] cache: expose username and sentinel_username redis configuration options for ACL-based Redis Auth support #1708 (@jsievenpiper)
[ENHANCEMENT] metrics-generator: expose span size as a metric #1662 (@ie-pham)
[ENHANCEMENT] Set Max Idle connections to 100 for Azure, should reduce DNS errors in Azure #1632 (@electron0zero)
[ENHANCEMENT] Add PodDisruptionBudget to ingesters in jsonnet #1691 (@joe-elliott)
[ENHANCEMENT] Add cli command an existing file to tempodb's current parquet schema. #1706 (@joe-elliott)
[ENHANCEMENT] Add query parameter to search API for traceQL queries #1729 (@kvrhdn)
[ENHANCEMENT] metrics-generator: filter out older spans before metrics are aggregated #1612 (@ie-pham)
[ENHANCEMENT] Add hedging to trace by ID lookups created by the frontend. #1735 (@mapno) New config options and defaults:

query_frontend:
  trace_by_id:
    hedge_requests_at: 5s
    hedge_requests_up_to: 3

[ENHANCEMENT] Vulture now has improved distribution of the random traces it searches. #1763 (@rfratto)
[ENHANCEMENT] Upgrade opentelemetry-proto submodule to v0.18.0 Internal types are updated to use scope instead of instrumentation_library. This is a breaking change in trace by ID queries if JSON is requested. #1754 (@mapno)
[ENHANCEMENT] Add TLS support to the vulture #1874 (@zalegrala)
[ENHANCEMENT] metrics-generator: extract status_message field from spans #1786, #1794 (@stoewer)
[ENHANCEMENT] metrics-generator: handle collisions between user defined and default dimensions #1794 (@stoewer) BREAKING CHANGE Custom dimensions colliding with intrinsic dimensions will be prefixed with __.
[ENHANCEMENT] metrics-generator: make intrinsic dimensions configurable and disable status_message by default #1960 (@stoewer)
[ENHANCEMENT] distributor: Log span names when distributor.log_received_spans.include_all_attributes is on #1790 (@suraciii)
[ENHANCEMENT] metrics-generator: truncate label names and values exceeding a configurable length #1897 (@kvrhdn)
[ENHANCEMENT] Add parquet WAL #1878 (@joe-elliott, @mdisibio)
[ENHANCEMENT] Convert last few Jsonnet alerts with per_cluster_label #2000 (@Whyeasy)
[ENHANCEMENT] New tenant dashboard #1901 (@mapno)
[BUGFIX] Stop distributors on Otel receiver fatal error#1887 (@rdooley)
[BUGFIX] New wal file separator '+' for the NTFS filesystem and backward compatibility with the old separator ':' #1700 (@kilian-kier)
[BUGFIX] Honor caching and buffering settings when finding traces by id #1697 (@joe-elliott)
[BUGFIX] Correctly propagate errors from the iterator layer up through the queriers #1723 (@joe-elliott)
[BUGFIX] Make multitenancy work with HTTP #1781 (@gouthamve)
[BUGFIX] Fix parquet search bug fix on http.status_code that may cause incorrect results to be returned #1799 (@mdisibio)
[BUGFIX] Fix failing SearchTagValues endpoint after startup #1813 (@stoewer)
[BUGFIX] tempo-mixin: tweak dashboards to support metrics without cluster label present #1913 (@kvrhdn)
[BUGFIX] Fix docker-compose examples not running on Apple M1 hardware #1920 (@stoewer)
[BUGFIX] Fix traceql parsing of most binary operations to not require spacing #1939 (@mdisibio)
[BUGFIX] Don't persist tenants without blocks in the ingester#1947 (@joe-elliott)
[BUGFIX] TraceQL: span scope not working with ranges #1948 (@mdisibio)
[BUGFIX] TraceQL: skip live traces search #1997 (@mapno)
[BUGFIX] Return more consistent search results by combining partial traces #2003 (@mapno)

v1.5.0 / 2022-08-17

[CHANGE] metrics-generator: Changed added metric label instance to __metrics_gen_instance to reduce collisions with custom dimensions. #1439 (@joe-elliott)
[CHANGE] Don't enforce max_bytes_per_tag_values_query when set to 0. #1447 (@joe-elliott)
[CHANGE] Add new querier service in deployment jsonnet to serve /status endpoint. #1474 (@annanay25)
[CHANGE] Swapped out Google Cloud Functions serverless docs and build for Google Cloud Run. #1483 (@joe-elliott)
[CHANGE] BREAKING CHANGE Change spanmetrics metric names and labels to match OTel conventions. #1478 (@mapno)
[FEATURE] Add support for time picker in jaeger query plugin. #1631 (@rubenvp8510) Old metric names:

traces_spanmetrics_duration_seconds_{sum,count,bucket}

New metric names:

traces_spanmetrics_latency_{sum,count,bucket}

Additionally, default label span_status is renamed to status_code.

[CHANGE] Update to Go 1.18 #1504 (@annanay25)
[CHANGE] Change tag/value lookups to return partial results when reaching response size limit instead of failing #1517 (@mdisibio)
[CHANGE] Change search to be case-sensitive #1547 (@mdisibio)

[CHANGE] Relax Hedged request defaults for external endpoints. #1566 (@joe-elliott)

querier:
  search:
    external_hedge_requests_at: 4s    -> 8s
    external_hedge_requests_up_to: 3  -> 2

[CHANGE] BREAKING CHANGE Include emptyDir for metrics generator wal storage in jsonnet #1556 (@zalegrala) Jsonnet users will now need to specify a storage request and limit for the generator wal. _config+:: { metrics_generator+: { ephemeral_storage_request_size: '10Gi', ephemeral_storage_limit_size: '11Gi', }, }
[CHANGE] Two additional latency buckets added to the default settings for generated spanmetrics. Note that this will increase cardinality when using the defaults. #1593 (@fredr)
[CHANGE] Mark log_received_traces as deprecated. New flag is log_received_spans. Extend distributor spans logger with optional features to include span attributes and a filter by error status. #1465 (@faustodavid)
[FEATURE] Add parquet block format #1479 #1531 #1564 (@annanay25, @mdisibio)
[FEATURE] Add anonymous usage reporting, enabled by default. #1481 (@zalegrala) BREAKING CHANGE As part of the usage stats inclusion, the distributor will also require access to the store. This is required so the distirbutor can know which cluster it should be reporting membership of.
[FEATURE] Include messaging systems and databases in service graphs. #1576 (@kvrhdn)
[ENHANCEMENT] Added the ability to have a per tenant max search duration. #1421 (@joe-elliott)
[ENHANCEMENT] metrics-generator: expose max_active_series as a metric #1471 (@kvrhdn)
[ENHANCEMENT] Azure Backend: Add support for authentication with Managed Identities. #1457 (@joe-elliott)
[ENHANCEMENT] Add metric to track feature enablement #1459 (@zalegrala)
[ENHANCEMENT] Added s3 config option insecure_skip_verify #1470 (@zalegrala)
[ENHANCEMENT] Added polling option to reduce issues in Azure blocklist_poll_jitter_ms #1518 (@joe-elliott)
[ENHANCEMENT] Add a config to query single ingester instance based on trace id hash for Trace By ID API. 1484 (@sagarwala, @bikashmishra100, @ashwinidulams)
[ENHANCEMENT] Add blocklist metrics for total backend objects and total backend bytes #1519 (@ie-pham)
[ENHANCEMENT] Adds tempo_querier_external_endpoint_hedged_roundtrips_total to count the total hedged requests #1558 (@joe-elliott) BREAKING CHANGE Removed deprecated metrics tempodb_(gcs|s3|azure)_request_duration_seconds in favor of tempodb_backend_request_duration_seconds. These metrics have been deprecated since v1.1.
[ENHANCEMENT] Add tags option for s3 backends. This allows new objects to be written with the configured tags. #1442 (@stevenbrookes)
[ENHANCEMENT] metrics-generator: support per-tenant processor configuration #1434 (@kvrhdn)
[ENHANCEMENT] Include rollout dashboard #1456 (@zalegrala)
[ENHANCEMENT] Add SentinelPassword configuration for Redis #1463 (@zalegrala)
[BUGFIX] Fix nil pointer panic when the trace by id path errors. #1441 (@joe-elliott)
[BUGFIX] Update tempo microservices Helm values example which missed the 'enabled' key for thriftHttp. #1472 (@hajowieland)
[BUGFIX] Fix race condition in forwarder overrides loop. 1468 (@mapno)
[BUGFIX] Fix v2 backend check on span name to be substring #1538 (@mdisibio)
[BUGFIX] Fix wal check on span name to be substring #1548 (@mdisibio)
[BUGFIX] Prevent ingester panic "cannot grow buffer" #1258 (@mdisibio)
[BUGFIX] metrics-generator: do not remove x-scope-orgid header in single tenant modus #1554 (@kvrhdn)
[BUGFIX] Fixed issue where backend does not support root.name and root.service.name #1589 (@kvrhdn)
[BUGFIX] Fixed ingester to continue starting up after block replay error #1603 (@mdisibio)
[BUGFIX] Fix issue relating to usage stats and GCS returning empty strings as tenantID #1625 (@ie-pham)

v1.4.1 / 2022-05-05

[BUGFIX] metrics-generator: don't inject X-Scope-OrgID header for single-tenant setups 1417 (@kvrhdn)
[BUGFIX] compactor: populate compaction_objects_combined_total and tempo_discarded_spans_total{reason="trace_too_large_to_compact"} metrics again 1420 (@mdisibio)
[BUGFIX] distributor: prevent panics when concurrently calling shutdown to forwarder's queueManager 1422 (@mapno)

v1.4.0 / 2022-04-28

[CHANGE] Vulture now exercises search at any point during the block retention to test full backend search. BREAKING CHANGE Dropped tempo-search-retention-duration parameter. #1297 (@joe-elliott)
[CHANGE] Updated storage.trace.pool.queue_depth default from 200->10000. #1345 (@joe-elliott)
[CHANGE] Update alpine images to 3.15 #1330 (@zalegrala)
[CHANGE] Updated flags -storage.trace.azure.storage-account-name and -storage.trace.s3.access_key to no longer to be considered as secrets #1356 (@simonswine)
[CHANGE] Add warning threshold for TempoIngesterFlushes and adjust critical threshold #1354 (@zalegrala)
[CHANGE] Include lambda in serverless e2e tests #1357 (@zalegrala)
[CHANGE] Replace mixin TempoIngesterFlushes metric to only look at retries #1354 (@zalegrala)
[CHANGE] Update the jsonnet for single-binary to include clustering #1391 (@zalegrala) BREAKING CHANGE After this change, the port specification has moved under $._config.tempo to avoid global port spec.
[FEATURE]: v2 object encoding added. This encoding adds a start/end timestamp to every record to reduce proto marshalling and increase search speed. BREAKING CHANGE After this rollout the distributors will use a new API on the ingesters. As such you must rollout all ingesters before rolling the distributors. Also, during this period, the ingesters will use considerably more resources and as such should be scaled up (or incoming traffic should be heavily throttled). Once all distributors and ingesters have rolled performance will return to normal. Internally we have observed ~1.5x CPU load on the ingesters during the rollout. #1227 (@joe-elliott)
[FEATURE] Added metrics-generator: an optional components to generate metrics from ingested traces #1282 (@mapno, @kvrhdn)
[FEATURE] Allow the compaction cycle to be configurable with a default of 30 seconds #1335 (@willdot)
[FEATURE] Add new config options for setting GCS metadata on new objects (@zalegrala)
[ENHANCEMENT] Enterprise jsonnet: add config to create tokengen job explicitly #1256 (@kvrhdn)
[ENHANCEMENT] Add new scaling alerts to the tempo-mixin #1292 (@mapno)
[ENHANCEMENT] Improve serverless handler error messages #1305 (@joe-elliott)
[ENHANCEMENT] Added a configuration option search_prefer_self to allow the queriers to do some work while also leveraging serverless in search. #1307 (@joe-elliott)
[ENHANCEMENT] Make trace combination/compaction more efficient #1291 (@mdisibio)
[ENHANCEMENT] Add Content-Type headers to query-frontend paths #1306 (@wperron)
[ENHANCEMENT] Partially persist traces that exceed max_bytes_per_trace during compaction #1317 (@joe-elliott)
[ENHANCEMENT] Make search respect per tenant max_bytes_per_trace and added skippedTraces to returned search metrics. #1318 (@joe-elliott)
[ENHANCEMENT] Improve serverless consistency by forcing a GC before returning. #1324 (@joe-elliott)
[ENHANCEMENT] Add forwarding queue from distributor to metrics-generator. #1331 (@mapno)

[ENHANCEMENT] Add hedging to queries to external endpoints. #1350 (@joe-elliott) New config options and defaults:

querier:
  search:
    external_hedge_requests_at: 5s
    external_hedge_requests_up_to: 3

BREAKING CHANGE Querier options related to search have moved under a search block:

querier:
 search_query_timeout: 30s
 search_external_endpoints: []
 search_prefer_self: 2

becomes

querier:
  search:
    query_timeout: 30s
    prefer_self: 2
    external_endpoints: []

[ENHANCEMENT] Added tenant ID (instance ID) to trace too large message. #1385 (@cristiangsp)
[ENHANCEMENT] Add a startTime and endTime parameter to the Trace by ID Tempo Query API to improve query performance #1388 (@sagarwala, @bikashmishra100, @ashwinidulams)
[BUGFIX] Correct issue where Azure "Blob Not Found" errors were sometimes not handled correctly #1390 (@joe-elliott)
[BUGFIX]: Enable compaction and retention in Tanka single-binary #1352
[BUGFIX]: Remove unnecessary PersistentVolumeClaim #1245
[BUGFIX] Fixed issue when query-frontend doesn't log request details when request is cancelled #1136 (@adityapwr)
[BUGFIX] Update OTLP port in examples (docker-compose & kubernetes) from legacy ports (55680/55681) to new ports (4317/4318) #1294 (@mapno)
[BUGFIX] Fixes min/max time on blocks to be based on span times instead of ingestion time. #1314 (@joe-elliott)
- Includes new configuration option to restrict the amount of slack around now to update the block start/end time. #1332 (@joe-elliott)
```
storage:
  trace:
    wal:
      ingestion_time_range_slack: 2m0s
```
- Includes a new metric to determine how often this range is exceeded: tempo_warnings_total{reason="outside_ingestion_time_slack"}
[BUGFIX] Prevent data race / ingester crash during searching by trace id by using xxhash instance as a local variable. #1387 (@bikashmishra100, @sagarwala, @ashwinidulams)
[BUGFIX] Fix spurious "failed to mark block compacted during retention" errors #1372 (@mdisibio)
[BUGFIX] Fix error message "Writer is closed" by resetting compression writer correctly on the error path. #1379 (@annanay25)

v1.3.2 / 2022-02-23

[BUGFIX] Fixed an issue where the query-frontend would corrupt start/end time ranges on searches which included the ingesters [#1295] (@joe-elliott)

v1.3.1 / 2022-02-02

[BUGFIX] Fixed panic when using etcd as ring's kvstore #1260 (@mapno)

v1.3.0 / 2022-01-24

[FEATURE]: Add support for inline environments. #1184 @irizzant
[CHANGE] Search: Add new per-tenant limit max_bytes_per_tag_values_query to limit the size of tag-values response. #1068 (@annanay25)
[CHANGE] Reduce MaxSearchBytesPerTrace ingester.max-search-bytes-per-trace default to 5KB #1129 @annanay25
[CHANGE] BREAKING CHANGE The OTEL GRPC receiver's default port changed from 55680 to 4317. #1142 (@tete17)
[CHANGE] Remove deprecated method Push from tempopb.Pusher #1173 (@kvrhdn)
[CHANGE] Upgrade cristalhq/hedgedhttp from v0.6.0 to v0.7.0 #1159 (@cristaloleg)
[CHANGE] Export trace id constant in api package #1176
[CHANGE] GRPC 1.33.3 => 1.38.0 broke compatibility with gogoproto.customtype. Enforce the use of gogoproto marshalling/unmarshalling for Tempo, Cortex & Jaeger structs. #1186 (@annanay25)
[CHANGE] BREAKING CHANGE Remove deprecated ingester gRPC endpoint and data encoding. The current data encoding was introduced in v1.0. If running earlier versions, first upgrade to v1.0 through v1.2 and allow time for all blocks to be switched to the "v1" data encoding. #1215 (@mdisibio)
[FEATURE] Added support for full backend search. #1174 (@joe-elliott) BREAKING CHANGE Moved querier.search_max_result_limit and querier.search_default_result_limit to query_frontend.search.max_result_limit and query_frontend.search.default_result_limit
[ENHANCEMENT]: Improve variables expansion support #1212 @irizzant
[ENHANCEMENT] Expose upto parameter on hedged requests for each backend with hedge_requests_up_to. #1085 (@joe-elliott)
[ENHANCEMENT] Search: drop use of TagCache, extract tags and tag values on-demand #1068 (@kvrhdn)
[ENHANCEMENT] Jsonnet: add $._config.namespace to filter by namespace in cortex metrics #1098 (@mapno)
[ENHANCEMENT] Add middleware to compress frontend HTTP responses with gzip if requested #1080 (@kvrhdn, @zalegrala)
[ENHANCEMENT] Allow query disablement in vulture #1117 (@zalegrala)
[ENHANCEMENT] Improve memory efficiency of compaction and block cutting. #1121 #1130 (@joe-elliott)
[ENHANCEMENT] Include metrics for configured limit overrides and defaults: tempo_limits_overrides, tempo_limits_defaults #1089 (@zalegrala)
[ENHANCEMENT] Add Envoy Proxy panel to Tempo / Writes dashboard #1137 (@kvrhdn)
[ENHANCEMENT] Reduce compactionCycle to improve performance in large multitenant environments #1145 (@joe-elliott)
[ENHANCEMENT] Added max_time_per_tenant to allow for independently configuring polling and compaction cycle. #1145 (@joe-elliott)
[ENHANCEMENT] Add tempodb_compaction_outstanding_blocks metric to measure compaction load #1144 (@mapno)
[ENHANCEMENT] Update mixin to use new backend metric #1151 (@zalegrala)
[ENHANCEMENT] Make TempoIngesterFlushesFailing alert more actionable #1157 (@dannykopping)
[ENHANCEMENT] Switch open-telemetry/opentelemetry-collector to grafana/opentelemetry-collectorl fork, update it to 0.40.0 and add missing dependencies due to the change #1142 (@tete17)
[ENHANCEMENT] Allow environment variables for Azure storage credentials #1147 (@zalegrala)
[ENHANCEMENT] jsonnet: set rollingUpdate.maxSurge to 3 for distributor, frontend and queriers #1164 (@kvrhdn)
[ENHANCEMENT] Reduce search data file sizes by optimizing contents #1165 (@mdisibio)
[ENHANCEMENT] Add tempo_ingester_live_traces metric #1170 (@mdisibio)
[ENHANCEMENT] Update compactor ring to automatically forget unhealthy entries #1178 (@mdisibio)
[ENHANCEMENT] Added the ability to pass ISO8601 date/times for start/end date to tempo-cli query api search #1208 (@joe-elliott)
[ENHANCEMENT] Prevent writes to large traces even after flushing to disk #1199 (@mdisibio)
[BUGFIX] Add process name to vulture traces to work around display issues #1127 (@mdisibio)
[BUGFIX] Fixed issue where compaction sometimes dropped spans. #1130 (@joe-elliott)
[BUGFIX] Ensure that the admin client jsonnet has correct S3 bucket property. (@hedss)
[BUGFIX] Publish tenant index age correctly for tenant index writers. #1146 (@joe-elliott)
[BUGFIX] Ingester startup panic slice bounds out of range #1195 (@mdisibio)
[BUGFIX] Update goreleaser install method to go install. #1222 (@mapno)
[BUGFIX] tempo-mixin: remove TempoDB Access panel from Tempo / Reads, metrics don't exist anymore #1218 (@kvrhdn)

v1.2.1 / 2021-11-15

[BUGFIX] Fix defaults for MaxBytesPerTrace (ingester.max-bytes-per-trace) and MaxSearchBytesPerTrace (ingester.max-search-bytes-per-trace) #1109 (@bitprocessor)
[BUGFIX] Ignore empty objects during compaction #1113 (@mdisibio)

v1.2.0 / 2021-11-05

[CHANGE] BREAKING CHANGE Drop support for v0 and v1 blocks. See 1.1 changelog for details #919 (@joe-elliott)
[CHANGE] Renamed CLI flag from --storage.trace.maintenance-cycle to --storage.trace.blocklist_poll. This is a breaking change #897 (@mritunjaysharma394)
[CHANGE] update jsonnet alerts and recording rules to use job_selectors and cluster_selectors for configurable unique identifier labels #935 (@kevinschoonover)
[CHANGE] Modify generated tag keys in Vulture for easier filtering #934 (@zalegrala)
[CHANGE] BREAKING CHANGE Consolidate status information onto /status endpoint #952 @zalegrala) The following endpoints moved. /runtime_config moved to /status/runtime_config /config moved to /status/config /services moved to /status/services
[CHANGE] BREAKING CHANGE Change ingester metric ingester_bytes_metric_total in favor of ingester_bytes_received_total #979 (@mapno)
[CHANGE] Add troubleshooting language to config for server.grpc_server_max_recv_msg_size and server.grpc_server_max_send_msg_size when handling large traces #1023 (@thejosephstevens)
[CHANGE] Parse search query tags from tags query parameter #1055 (@kvrhdn)
[FEATURE] Add ability to search ingesters for traces #806 (@mdisibio)
[FEATURE] Add runtime config handler #936 (@mapno)
[FEATURE] Search WAL reload and compression(versioned encoding) support #1000 (@annanay25, @mdisibio)
[FEATURE] Added ability to add a middleware to the OTel receivers' consume function #1015 (@chaudum)
[FEATURE] Add ScalableSingleBinary operational run mode #1004 (@zalegrala)
[FEATURE] Added a jsonnet library for Grafana Enterprise Traces (GET) deployments #1096
[ENHANCEMENT] Added "query blocks" cli option. #876 (@joe-elliott)
[ENHANCEMENT] Added "search blocks" cli option. #972 (@joe-elliott)
[ENHANCEMENT] Added traceid to trace too large message. #888 (@mritunjaysharma394)
[ENHANCEMENT] Add support to tempo workloads to overrides from single configmap in microservice mode. #896 (@kavirajk)
[ENHANCEMENT] Make overrides_config block name consistent with Loki and Cortex in microservice mode. #906 (@kavirajk)
[ENHANCEMENT] Changes the metrics name from cortex_runtime_config_last_reload_successful to tempo_runtime_config_last_reload_successful #945 (@kavirajk)

[ENHANCEMENT] Updated config defaults to reflect better capture operational knowledge. #913 (@joe-elliott)

ingester:
  trace_idle_period: 30s => 10s  # reduce ingester memory requirements with little impact on querying
  flush_check_period: 30s => 10s
query_frontend:
  query_shards: 2 => 20          # will massively improve performance on large installs
storage:
  trace:
    wal:
      encoding: none => snappy   # snappy has been tested thoroughly and ready for production use
    block:
      bloom_filter_false_positive: .05 => .01          # will increase total bloom filter size but improve query performance
      bloom_filter_shard_size_bytes: 256KiB => 100 KiB # will improve query performance
compactor:
  compaction:
    chunk_size_bytes: 10 MiB => 5 MiB  # will reduce compactor memory needs
    compaction_window: 4h => 1h        # will allow more compactors to participate in compaction without substantially increasing blocks

[ENHANCEMENT] Make s3 backend readError logic more robust #905 (@wei840222)
[ENHANCEMENT] Include additional detail when searching for traces #916 (@zalegrala)
[ENHANCEMENT] Add gen index and gen bloom commands to tempo-cli. #903 (@annanay25)
[ENHANCEMENT] Implement trace comparison in Vulture #904 (@zalegrala)
[ENHANCEMENT] Improve zstd read throughput using zstd.Decoder #948 (@joe-elliott)
[ENHANCEMENT] Dedupe search records while replaying WAL #940 (@annanay25)
[ENHANCEMENT] Add status endpoint to list the available endpoints #938 (@zalegrala)
[ENHANCEMENT] Compression updates: Added s2, improved snappy performance #961 (@joe-elliott)
[ENHANCEMENT] Add search block headers #943 (@mdisibio)
[ENHANCEMENT] Add search block headers for wal blocks #963 (@mdisibio)
[ENHANCEMENT] Add support for vulture sending long running traces #951 (@zalegrala)
[ENHANCEMENT] Support global denylist and per-tenant allowlist of tags for search data. #960 (@annanay25)
[ENHANCEMENT] Add search_query_timeout to querier config. #984 (@kvrhdn)
[ENHANCEMENT] Include simple e2e test to test searching #978 (@zalegrala)
[ENHANCEMENT] Jsonnet: add $._config.memcached.memory_limit_mb #987 (@kvrhdn)
[ENHANCEMENT] Upgrade jsonnet-libs to 1.19 and update tk examples #1001 (@mapno)
[ENHANCEMENT] Shard tenant index creation by tenant and add functionality to handle stale indexes. #1005 (@joe-elliott)
[ENHANCEMENT] BREAKING CHANGE Support partial results from failed block queries #1007 (@mapno) Querier GET /querier/api/traces/<traceid> response's body has been modified to return tempopb.TraceByIDResponse instead of simply tempopb.Trace. This will cause a disruption of the read path during rollout of the change.
[ENHANCEMENT] Add search_default_limit and search_max_result_limit to querier config. #1022 #1044 (@kvrhdn)
[ENHANCEMENT] Add new metric tempo_distributor_push_duration_seconds #1027 (@zalegrala)
[ENHANCEMENT] Add query parameter to show the default config values and the difference between the current values and the defaults. #1045 (@MichelHollands)
[ENHANCEMENT] Adding metrics around ingester flush retries #1049 (@dannykopping)
[ENHANCEMENT] Performance: More efficient distributor batching #1075 (@joe-elliott)
[ENHANCEMENT] Allow search disablement in vulture #1069 (@zalegrala)
[ENHANCEMENT] Jsonnet: add $._config.search_enabled, correctly set http_api_prefix in config #1072 (@kvrhdn)
[ENHANCEMENT] Performance: Remove WAL contention between ingest and searches #1076 (@mdisibio)
[ENHANCEMENT] Include tempo-cli in the release #1086 (@zalegrala)
[ENHANCEMENT] Add search on span status #1093 (@mdisibio)
[ENHANCEMENT] Slightly improved compression performance #1094 (@bboreham)
[BUGFIX] Update port spec for GCS docker-compose example #869 (@zalegrala)
[BUGFIX] Fix "magic number" errors and other block mishandling when an ingester forcefully shuts down #937 (@mdisibio)
[BUGFIX] Fix compactor memory leak #806 (@mdisibio)
[BUGFIX] Fix an issue with WAL replay of zero-length search data when search is disabled. #968 (@annanay25)
[BUGFIX] Set span's tag span.kind to client in query-frontend #975 (@mapno)
[BUGFIX] Nil check overrides module in the /status handler #994 (@mapno)
[BUGFIX] Several bug fixes for search contention and panics #1033 (@mdisibio)
[BUGFIX] Fixes tempodb_backend_hedged_roundtrips_total to correctly count hedged roundtrips. #1079 (@joe-elliott)
[BUGFIX] Update go-kit logger package to remove spurious debug logs #1094 (@bboreham)

v1.1.0 / 2021-08-26

[CHANGE] Upgrade Cortex from v1.9.0 to v1.9.0-131-ga4bf10354 #841 (@aknuds1)
[CHANGE] Change default tempo port from 3100 to 3200 #770 (@MurzNN)
[CHANGE] Jsonnet: use dedicated configmaps for distributors and ingesters #775 (@kvrhdn)
[CHANGE] Docker images are now prefixed by their branch name #828 (@jvrplmlmn)
[CHANGE] Update to Go 1.17 #953
[FEATURE] Added the ability to hedge requests with all backends #750 (@joe-elliott)
[FEATURE] Added a tenant index to reduce bucket polling. #834 (@joe-elliott)
[ENHANCEMENT] Added hedged request metric tempodb_backend_hedged_roundtrips_total and a new storage agnostic tempodb_backend_request_duration_seconds metric that supersedes the soon-to-be deprecated storage specific metrics (tempodb_azure_request_duration_seconds, tempodb_s3_request_duration_seconds and tempodb_gcs_request_duration_seconds). #790 (@JosephWoodward)
[ENHANCEMENT] Performance: improve compaction speed with concurrent reads and writes #754 (@mdisibio)
[ENHANCEMENT] Improve readability of cpu and memory metrics on operational dashboard #764 (@bboreham)
[ENHANCEMENT] Add azure_request_duration_seconds metric. #767 (@JosephWoodward)
[ENHANCEMENT] Add s3_request_duration_seconds metric. #776 (@JosephWoodward)
[ENHANCEMENT] Add tempo_ingester_flush_size_bytes metric. #777 (@bboreham)
[ENHANCEMENT] Microservices jsonnet: resource requests and limits can be set in $._config. #793 (@kvrhdn)
[ENHANCEMENT] Add -config.expand-env cli flag to support environment variables expansion in config file. #796 (@Ashmita152)
[ENHANCEMENT] Add ability to control bloom filter caching based on age and/or compaction level. Add new cli command list cache-summary. #805 (@annanay25)
[ENHANCEMENT] Emit traces for ingester flush operations. #812 (@bboreham)
[ENHANCEMENT] Add retry middleware in query-frontend. #814 (@kvrhdn)
[ENHANCEMENT] Add -use-otel-tracer to use the OpenTelemetry tracer, this will also capture traces emitted by the gcs sdk. Experimental: not all features are supported (i.e. remote sampling). #842 (@kvrhdn)
[ENHANCEMENT] Add /services endpoint. #863 (@kvrhdn)
[ENHANCEMENT] Include distributed docker-compose example #859 (@zalegrala)
[ENHANCEMENT] Added "query blocks" cli option. #876 (@joe-elliott)
[ENHANCEMENT] Add e2e integration test for GCS. #883 (@annanay25)
[ENHANCEMENT] Added traceid to trace too large message. #888 (@mritunjaysharma394)
[ENHANCEMENT] Add support to tempo workloads to overrides from single configmap in microservice mode. #896 (@kavirajk)
[ENHANCEMENT] Make overrides_config block name consistent with Loki and Cortex in microservice mode. #906 (@kavirajk)
[ENHANCEMENT] Make overrides_config mount name static tempo-overrides in the tempo workloads in microservice mode. #906 (@kavirajk)
[ENHANCEMENT] Reduce compactor memory usage by forcing garbage collection. #915 (@joe-elliott)
[ENHANCEMENT] Implement search in vulture. #944 (@zalegrala)
[BUGFIX] Allow only valid trace ID characters when decoding #854 (@zalegrala)
[BUGFIX] Queriers complete one polling cycle before finishing startup. #834 (@joe-elliott)
[BUGFIX] Update port spec for GCS docker-compose example #869 (@zalegrala)
[BUGFIX] Cortex upgrade to fix an issue where unhealthy compactors can't be forgotten #878 (@joe-elliott)

v1.0.1 / 2021-06-14

[BUGFIX] Guard against negative dataLength #763 (@joe-elliott)

v1.0.0 / 2021-06-08

[CHANGE] Mark -auth.enabled as deprecated. New flag is -multitenancy.enabled and is set to false by default. This is a breaking change if you were relying on auth/multitenancy being enabled by default. #646
[ENHANCEMENT] Performance: Improve Ingester Record Insertion. #681
[ENHANCEMENT] Improve WAL Replay by not rebuilding the WAL. #668
[ENHANCEMENT] Add config option to disable write extension to the ingesters. #677
[ENHANCEMENT] Preallocate byte slices on ingester request unmarshal. #679
[ENHANCEMENT] Reduce marshalling in the ingesters to improve performance. #694 This change requires a specific rollout process to prevent dropped spans. First, rollout everything except distributors. After all ingesters have updated you can then rollout distributors to the latest version. This is due to changes in the communication between ingesters <-> distributors.
[ENHANCEMENT] Allow setting the bloom filter shard size with support dynamic shard count.#644
[ENHANCEMENT] GCS SDK update v1.12.0 => v.15.0, ReadAllWithEstimate used in GCS/S3 backends. #693
[ENHANCEMENT] Add a new endpoint /api/echo to test the query frontend is reachable. #714
[BUGFIX] Fix Query Frontend grpc settings to avoid noisy error log. #690
[BUGFIX] Zipkin Support - CombineTraces. #688
[BUGFIX] Zipkin support - Dedupe span IDs based on span.Kind (client/server) in Query Frontend. #687
[BUGFIX] Azure Backend - Fix an issue with the append method on the Azure backend. #736

v0.7.0 / 2021-04-22

License Change v0.7.0 and future versions are licensed under AGPLv3 #660

[CHANGE] Add json struct tags to overrides' Limits struct in addition to yaml tags. #656
[CHANGE] Update to Go 1.16, latest OpenTelemetry proto definition and collector #546
[CHANGE] max_spans_per_trace limit override has been removed in favour of max_bytes_per_trace. This is a breaking change to the overrides config section. #612
[CHANGE] Add new flag -ingester.lifecycler.ID to manually override the ingester ID with which to register in the ring. #625
[CHANGE] ingestion_rate_limit limit override has been removed in favour of ingestion_rate_limit_bytes. ingestion_burst_size limit override has been removed in favour of ingestion_burst_size_bytes. This is a breaking change to the overrides config section. #630
[FEATURE] Add page based access to the index file. #557
[FEATURE] (Experimental) WAL Compression/checksums. #638
[ENHANCEMENT] Add a Shutdown handler to flush data to backend, at "/shutdown". #526
[ENHANCEMENT] Queriers now query all (healthy) ingesters for a trace to mitigate 404s on ingester rollouts/scaleups. This is a breaking change and will likely result in query errors on rollout as the query signature b/n QueryFrontend & Querier has changed. #557
[ENHANCEMENT] Add list compaction-summary command to tempo-cli #588
[ENHANCEMENT] Add list and view index commands to tempo-cli #611
[ENHANCEMENT] Add a configurable prefix for HTTP endpoints. #631
[ENHANCEMENT] Add kafka receiver. #613
[ENHANCEMENT] Upgrade OTel collector to v0.21.0. #613
[ENHANCEMENT] Add support for Cortex Background Cache. #640
[BUGFIX] Fixes permissions errors on startup in GCS. #554
[BUGFIX] Fixes error where Dell ECS cannot list objects. #561
[BUGFIX] Fixes listing blocks in S3 when the list is truncated. #567
[BUGFIX] Fixes where ingester may leave file open #570
[BUGFIX] Fixes a bug where some blocks were not searched due to query sharding and randomness in blocklist poll. #583
[BUGFIX] Fixes issue where wal was deleted before successful flush and adds exponential backoff for flush errors #593
[BUGFIX] Fixes issue where Tempo would not parse odd length trace ids #605
[BUGFIX] Sort traces on flush to reduce unexpected recombination work by compactors #606
[BUGFIX] Ingester fully persists blocks locally to reduce amount of work done after restart #628

v0.6.0 / 2021-02-18

[CHANGE] Fixed ingester latency spikes on read #461
[CHANGE] Ingester cut blocks based on size instead of trace count. Replace ingester traces_per_block setting with max_block_bytes. This is a breaking change. #474
[CHANGE] Refactor cache section in tempodb. This is a breaking change b/c the cache config section has changed. #485
[CHANGE] New compactor setting for max block size data instead of traces. #520
[CHANGE] Change default ingester_client compression from gzip to snappy. #522
[CHANGE/BUGFIX] Rename tempodb_compaction_objects_written and tempodb_compaction_bytes_written metrics to tempodb_compaction_objects_written_total and tempodb_compaction_bytes_written_total. #524
[CHANGE] Replace tempo-cli list block --check-dupes option with --scan and collect additional stats #534
[FEATURE] Added block compression. This is a breaking change b/c some configuration fields moved. #504
[CHANGE] Drop Vulture Loki dependency. This is a breaking change. #509
[ENHANCEMENT] Serve config at the "/config" endpoint. #446
[ENHANCEMENT] Switch blocklist polling and retention to different concurrency mechanism, add configuration options. #475
[ENHANCEMENT] Add S3 options region and forcepathstyle #431
[ENHANCEMENT] Add exhaustive search to combine traces from all blocks in the backend. #489
[ENHANCEMENT] Add per-tenant block retention #77
[ENHANCEMENT] Change index-downsample to index-downsample-bytes. This is a breaking change #519
[BUGFIX] Upgrade cortex dependency to v1.7.0-rc.0+ to address issue with forgetting ring membership #442 #512
[BUGFIX] No longer raise the tempodb_blocklist_poll_errors_total metric if a block doesn't have meta or compacted meta. #481]
[BUGFIX] Replay wal completely before ingesting new spans. #525

v0.5.0 / 2021-01-15

[CHANGE] Redo tempo-cli with basic command structure and improvements #385
[CHANGE] Add content negotiation support and sharding parameters to Querier #375
[CHANGE] Remove S3 automatic bucket creation #404
[CHANGE] Compactors should round robin tenants instead of choosing randomly #420
[CHANGE] Switch distributor->ingester communication to more efficient PushBytes method. This is a breaking change when running in microservices mode with separate distributors and ingesters. To prevent errors ingesters must be fully upgraded first, then distributors.
[CHANGE] Removed disk_cache. This is a breaking change b/c there is no disk cache. Please use redis or memcached. #441
[CHANGE] Rename IngestionMaxBatchSize to IngestionBurstSize. This is a breaking change. #445
[ENHANCEMENT] Add docker-compose example for GCS along with new backend options #397
[ENHANCEMENT] tempo-cli list blocks usability improvements #403
[ENHANCEMENT] Reduce active traces locking time. #449
[ENHANCEMENT] Added tempo_distributor_bytes_received_total as a per tenant counter of uncompressed bytes received. #453
[BUGFIX] Compactor without GCS permissions fail silently #379
[BUGFIX] Prevent race conditions between querier polling and ingesters clearing complete blocks #421
[BUGFIX] Exclude blocks in last active window from compaction #411
[BUGFIX] Mixin: Ignore metrics and query-frontend route when checking for TempoRequestLatency alert. #440
[FEATURE] Add support for Azure Blob Storage backend #340
[FEATURE] Add Query Frontend module to allow scaling the query path #400

v0.4.0 / 2020-12-03

[CHANGE] From path.Join to filepath.Join #338
[CHANGE] Upgrade Cortex from v1.3.0 to v.1.4.0 #341
[CHANGE] Compact more than 2 blocks at a time #348
[CHANGE] Remove tempodb_compaction_duration_seconds metric. #360
[ENHANCEMENT] Add tempodb_compaction_objects_combined metric. #339
[ENHANCEMENT] Added OpenMetrics exemplar support. #359
[ENHANCEMENT] Add tempodb_compaction_objects_written metric. #360
[ENHANCEMENT] Add tempodb_compaction_bytes_written metric. #360
[ENHANCEMENT] Add tempodb_compaction_blocks_total metric. #360
[ENHANCEMENT] Add support for S3 V2 signatures. #352
[ENHANCEMENT] Add support for Redis caching. #354
[BUGFIX] Frequent errors logged by compactor regarding meta not found #327
[BUGFIX] Fix distributors panicking on rollout #343
[BUGFIX] Fix ingesters occassionally double flushing #364
[BUGFIX] Fix S3 backend logs "unsupported value type" #381

v0.3.0 / 2020-11-10

[CHANGE] Bloom filters are now sharded to reduce size and improve caching, as blocks grow. This is a breaking change and all data stored before this change will not be queryable. #192
[CHANGE] Rename maintenance cycle to blocklist poll. #315
[ENHANCEMENT] CI checks for vendored dependencies using make vendor-check. Update CONTRIBUTING.md to reflect the same before checking in files in a PR. #274
[ENHANCEMENT] Add warnings for suspect configs. #294
[ENHANCEMENT] Add command line flags for s3 credentials. #308
[ENHANCEMENT] Support multiple authentication methods for S3 (IRSA, IAM role, static). #320
[ENHANCEMENT] Add per tenant bytes counter. #331
[BUGFIX] S3 multi-part upload errors #306
[BUGFIX] Increase Prometheus notfound metric on tempo-vulture. #301
[BUGFIX] Return 404 if searching for a tenant id that does not exist in the backend. #321
[BUGFIX] Prune in-memory blocks from missing tenants. #314

FilesExpand file tree

CHANGELOG.md

Latest commit

History