From 71a3ed4c6b887a795827f374244c8aed2902eea4 Mon Sep 17 00:00:00 2001 From: Aleksandra Spilkowska Date: Thu, 27 Aug 2026 11:51:49 +0200 Subject: [PATCH 1/3] clarify why observed sampling rate can differ in Storage Explorer Expands the single-sentence explanation into a dedicated subsection covering: errors always being stored, head-based and tail-based sampling edge cases, APM Server buffer overflow behavior, and third-party tracing header overrides --- .../observability/apm/storage-explorer.md | 30 ++++++++++++++++++- 1 file changed, 29 insertions(+), 1 deletion(-) diff --git a/solutions/observability/apm/storage-explorer.md b/solutions/observability/apm/storage-explorer.md index 514b3735aa..72d7d9d960 100644 --- a/solutions/observability/apm/storage-explorer.md +++ b/solutions/observability/apm/storage-explorer.md @@ -38,7 +38,7 @@ The service size chart displays the estimated size of each service over time. Ex The service statistics table provides detailed information on each service: * A list of **service environments**. -* The **sampling rate**. This value is calculated by dividing the number of sampled transactions by total throughput. It might differ from the configured sampling rate for two reasons: with head-based sampling, the initial service makes the sampling decision, and with tail-based sampling, granular policies allow you to set multiple sample rates. +* The **sampling rate**. This is the observed sampling rate — the number of sampled transactions divided by total throughput — based on documents that actually reached {{es}}. It can differ from your configured rate. See [Why the observed sampling rate might differ](#why-the-observed-sampling-rate-might-differ). * The estimated **size on disk**. This storage size includes both primary and replica shards and is calculated by prorating the total size of your indices by the service’s document count divided by the total number of documents. * Number of **transactions**, **spans**, **errors**, and **metrics** — doc count and size on disk. @@ -65,6 +65,34 @@ To reduce the number of system, runtime, and application metrics, tune the APM a To reduce the number of errors a service generate, work with your developers to change how exceptions are handled in your code. +### Why the observed sampling rate might differ [why-the-observed-sampling-rate-might-differ] + +The observed sampling rate reflects what actually reached {{es}}, not what you configured. Several factors can cause a difference. + +**Errors are never sampled** + +Errors are always stored regardless of your sampling configuration. If a service generates a high volume of errors, the observed rate will be higher than the configured sampling rate. + +**Head-based sampling** + +With [head-based sampling](/solutions/observability/apm/transaction-sampling.md), the sampling decision happens at the root service and is propagated to downstream services through distributed tracing headers. Configuring a sampling rate on a non-root service has no effect — it will always respect the upstream decision. + +Non-sampled transactions are not sent to {{apm-server}} at all, so they don't appear in storage. The exception is the RUM JS Agent, which always sends transaction events regardless of whether they are sampled. + +On low-throughput services, the observed rate may be a less precise approximation of the configured rate because the representative value used to scale sampled transactions can introduce statistical error. + +**Tail-based sampling** + +With [tail-based sampling](/solutions/observability/apm/apm-server/tail-based-sampling.md), {{apm-server}} buffers events locally before making the sampling decision. If the local buffer reaches its storage limit, {{apm-server}} either sends all buffered events (raising the observed rate) or drops them (lowering the observed rate), depending on how overflow is configured. + +**Third-party tracing headers** + +If a gateway or third-party service injects distributed tracing headers that include a sampling decision before the request reaches your root service, {{apm-agent}}s respect those headers instead of the configured rate. You can override this behavior using the [trace continuation strategies](/solutions/observability/apm/transaction-sampling.md#_trace_continuation_strategies_with_distributed_tracing) settings in your {{apm-agent}}. + +**Tips for a more accurate comparison** + +For the best approximation of your configured sampling rate, filter the time range to a period when your sampling settings were stable and your service was not experiencing an unusually high error rate. + ## Privileges [_privileges] Storage Explorer requires expanded privileges to view. See [Create a storage explorer user](/solutions/observability/apm/ui-user-storage-explorer.md) for more information. From 50b4bfcb4b6251d6512d83e485acdb2ddb6cf9c1 Mon Sep 17 00:00:00 2001 From: Aleksandra Spilkowska Date: Fri, 28 Aug 2026 09:45:24 +0200 Subject: [PATCH 2/3] minor style fixes --- .../observability/apm/storage-explorer.md | 27 +++++++++---------- 1 file changed, 13 insertions(+), 14 deletions(-) diff --git a/solutions/observability/apm/storage-explorer.md b/solutions/observability/apm/storage-explorer.md index 72d7d9d960..aad250e271 100644 --- a/solutions/observability/apm/storage-explorer.md +++ b/solutions/observability/apm/storage-explorer.md @@ -10,7 +10,7 @@ products: # Storage Explorer [apm-storage-explorer] -Analyze your APM data and manage costs with **storage explorer**. For example, analyze the storage footprint of each of your services to see which are producing large amounts of data—then change the sample rate of a service to lower the amount of data ingested. Or, expand the time filter to visualize data trends over time so that you can better forecast and prepare for future storage needs. +Analyze your APM data and manage costs with **storage explorer**. For example, analyze the storage footprint of each of your services to see which are producing large amounts of data, then change the sample rate of a service to lower the amount of data ingested. Or, expand the time filter to visualize data trends over time so that you can better forecast and prepare for future storage needs. :::{image} /solutions/images/observability-storage-explorer-overview.png :alt: APM Storage Explorer @@ -38,7 +38,7 @@ The service size chart displays the estimated size of each service over time. Ex The service statistics table provides detailed information on each service: * A list of **service environments**. -* The **sampling rate**. This is the observed sampling rate — the number of sampled transactions divided by total throughput — based on documents that actually reached {{es}}. It can differ from your configured rate. See [Why the observed sampling rate might differ](#why-the-observed-sampling-rate-might-differ). +* The **sampling rate**. This is the observed sampling rate, which is the number of sampled transactions divided by total throughput. It is based on documents that actually reached {{es}}. It can differ from your configured rate. Refer to [Why the observed sampling rate might differ](#why-the-observed-sampling-rate-might-differ) for more information. * The estimated **size on disk**. This storage size includes both primary and replica shards and is calculated by prorating the total size of your indices by the service’s document count divided by the total number of documents. * Number of **transactions**, **spans**, **errors**, and **metrics** — doc count and size on disk. @@ -67,29 +67,28 @@ To reduce the number of errors a service generate, work with your developers to ### Why the observed sampling rate might differ [why-the-observed-sampling-rate-might-differ] -The observed sampling rate reflects what actually reached {{es}}, not what you configured. Several factors can cause a difference. +The observed sampling rate reflects what actually reached {{es}}, and not necessarily what you configured. Several factors can cause a difference: -**Errors are never sampled** +* **Errors are never sampled** -Errors are always stored regardless of your sampling configuration. If a service generates a high volume of errors, the observed rate will be higher than the configured sampling rate. + Errors are always stored regardless of your sampling configuration. If a service generates a high volume of errors, the observed rate will be higher than the configured sampling rate. -**Head-based sampling** +* **Head-based sampling** -With [head-based sampling](/solutions/observability/apm/transaction-sampling.md), the sampling decision happens at the root service and is propagated to downstream services through distributed tracing headers. Configuring a sampling rate on a non-root service has no effect — it will always respect the upstream decision. + With [head-based sampling](/solutions/observability/apm/transaction-sampling.md), the sampling decision happens at the root service and is propagated to downstream services through distributed tracing headers. Configuring a sampling rate on a non-root service has no effect, as it will always respect the upstream decision. -Non-sampled transactions are not sent to {{apm-server}} at all, so they don't appear in storage. The exception is the RUM JS Agent, which always sends transaction events regardless of whether they are sampled. + Non-sampled transactions are not sent to {{apm-server}} at all, so they don't appear in storage. The exception is the RUM JS Agent, which always sends transaction events regardless of whether they are sampled. -On low-throughput services, the observed rate may be a less precise approximation of the configured rate because the representative value used to scale sampled transactions can introduce statistical error. + On low-throughput services, the observed rate might be a less precise approximation of the configured rate because the representative value used to scale sampled transactions can introduce statistical error. -**Tail-based sampling** +* **Tail-based sampling** -With [tail-based sampling](/solutions/observability/apm/apm-server/tail-based-sampling.md), {{apm-server}} buffers events locally before making the sampling decision. If the local buffer reaches its storage limit, {{apm-server}} either sends all buffered events (raising the observed rate) or drops them (lowering the observed rate), depending on how overflow is configured. + With [tail-based sampling](/solutions/observability/apm/apm-server/tail-based-sampling.md), {{apm-server}} buffers events locally before making the sampling decision. If the local buffer reaches its storage limit, {{apm-server}} either sends all buffered events (raising the observed rate) or drops them (lowering the observed rate), depending on how overflow is configured. -**Third-party tracing headers** +* **Third-party tracing headers** -If a gateway or third-party service injects distributed tracing headers that include a sampling decision before the request reaches your root service, {{apm-agent}}s respect those headers instead of the configured rate. You can override this behavior using the [trace continuation strategies](/solutions/observability/apm/transaction-sampling.md#_trace_continuation_strategies_with_distributed_tracing) settings in your {{apm-agent}}. + If a gateway or third-party service injects distributed tracing headers that include a sampling decision before the request reaches your root service, {{apm-agent}}s respect those headers instead of the configured rate. You can override this behavior using the [trace continuation strategies](/solutions/observability/apm/transaction-sampling.md#_trace_continuation_strategies_with_distributed_tracing) settings in your {{apm-agent}}. -**Tips for a more accurate comparison** For the best approximation of your configured sampling rate, filter the time range to a period when your sampling settings were stable and your service was not experiencing an unusually high error rate. From 61b332c664e5e4c81844d342c363e4bbedfa8458 Mon Sep 17 00:00:00 2001 From: Aleksandra Spilkowska Date: Mon, 31 Aug 2026 15:02:02 +0200 Subject: [PATCH 3/3] apply MIke's comment --- solutions/observability/apm/storage-explorer.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/solutions/observability/apm/storage-explorer.md b/solutions/observability/apm/storage-explorer.md index aad250e271..c3148d0001 100644 --- a/solutions/observability/apm/storage-explorer.md +++ b/solutions/observability/apm/storage-explorer.md @@ -77,7 +77,7 @@ The observed sampling rate reflects what actually reached {{es}}, and not necess With [head-based sampling](/solutions/observability/apm/transaction-sampling.md), the sampling decision happens at the root service and is propagated to downstream services through distributed tracing headers. Configuring a sampling rate on a non-root service has no effect, as it will always respect the upstream decision. - Non-sampled transactions are not sent to {{apm-server}} at all, so they don't appear in storage. The exception is the RUM JS Agent, which always sends transaction events regardless of whether they are sampled. + Non-sampled transactions are not sent to {{apm-server}}, so they don't appear in storage. The exception is the RUM JS Agent, which always sends transaction events regardless of whether they are sampled. On low-throughput services, the observed rate might be a less precise approximation of the configured rate because the representative value used to scale sampled transactions can introduce statistical error.