Skip to content

feat(container-insights): drop duplicate cAdvisor and node-exporter series #2322

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
petruanica wants to merge 2 commits into from
Closed

feat(container-insights): drop duplicate cAdvisor and node-exporter series #2322

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
9b077ca
feat(container-insights): drop duplicate cadvisor series and two coll…
petruanica Oct 5, 2026
370eea9
Merge branch 'main' into otel-ci-m1-drop-duplicate-series
petruanica Oct 6, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 7 additions & 1 deletion translator/tocwconfig/sampleConfig/opentelemetry/combined_v1_v2_eks_config.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -619,6 +619,11 @@ processors:
metrics:
datapoint:
- attributes["container"] == "POD"
filter/cw_k8s_ci_v0_cadvisor_rollup:
error_mode: ignore
metrics:
datapoint:
- (attributes["container"] == nil or attributes["container"] == "") and attributes["pod"] != nil and attributes["pod"] != "" and IsMatch(metric.name, "^container_network_") != true
filter/cw_k8s_ci_v0_neuron:
error_mode: ignore
metrics:
Expand All @@ -628,7 +633,7 @@ processors:
error_mode: ignore
metrics:
metric:
- IsMatch(name, "^(up|scrape_duration_seconds|scrape_samples_scraped|scrape_samples_post_metric_relabeling|scrape_series_added)$")
- IsMatch(name, "^(up|scrape_duration_seconds|scrape_samples_scraped|scrape_samples_post_metric_relabeling|scrape_series_added|node_scrape_collector_duration_seconds|node_scrape_collector_success)$")
filter/dbi_exclude_monitor_0:
error_mode: propagate
logs:
Expand Down Expand Up @@ -2878,6 +2883,7 @@ service:
- transform/cw_k8s_ci_v0_set_cluster_name
- filter/cw_k8s_ci_v0_cadvisor_empty
- filter/cw_k8s_ci_v0_cadvisor_pod
- filter/cw_k8s_ci_v0_cadvisor_rollup
- groupbyattrs/cw_k8s_ci_v0_cadvisor
- transform/cw_k8s_ci_v0_cadvisor_promote
- transform/cw_k8s_ci_v0_set_node_name
Expand Down
2 changes: 1 addition & 1 deletion translator/tocwconfig/sampleConfig/opentelemetry/container_insights_cluster_config.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -90,7 +90,7 @@ processors:
error_mode: ignore
metrics:
metric:
- IsMatch(name, "^(up|scrape_duration_seconds|scrape_samples_scraped|scrape_samples_post_metric_relabeling|scrape_series_added)$")
- IsMatch(name, "^(up|scrape_duration_seconds|scrape_samples_scraped|scrape_samples_post_metric_relabeling|scrape_series_added|node_scrape_collector_duration_seconds|node_scrape_collector_success)$")
groupbyattrs/cw_k8s_ci_v0_ksm:
keys:
- pod
Expand Down
2 changes: 1 addition & 1 deletion translator/tocwconfig/sampleConfig/opentelemetry/container_insights_cluster_config_aks.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -90,7 +90,7 @@ processors:
error_mode: ignore
metrics:
metric:
- IsMatch(name, "^(up|scrape_duration_seconds|scrape_samples_scraped|scrape_samples_post_metric_relabeling|scrape_series_added)$")
- IsMatch(name, "^(up|scrape_duration_seconds|scrape_samples_scraped|scrape_samples_post_metric_relabeling|scrape_series_added|node_scrape_collector_duration_seconds|node_scrape_collector_success)$")
groupbyattrs/cw_k8s_ci_v0_ksm:
keys:
- pod
Expand Down
2 changes: 1 addition & 1 deletion translator/tocwconfig/sampleConfig/opentelemetry/container_insights_cluster_config_gke.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -90,7 +90,7 @@ processors:
error_mode: ignore
metrics:
metric:
- IsMatch(name, "^(up|scrape_duration_seconds|scrape_samples_scraped|scrape_samples_post_metric_relabeling|scrape_series_added)$")
- IsMatch(name, "^(up|scrape_duration_seconds|scrape_samples_scraped|scrape_samples_post_metric_relabeling|scrape_series_added|node_scrape_collector_duration_seconds|node_scrape_collector_success)$")
groupbyattrs/cw_k8s_ci_v0_ksm:
keys:
- pod
Expand Down
8 changes: 7 additions & 1 deletion translator/tocwconfig/sampleConfig/opentelemetry/container_insights_node_config.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -172,6 +172,11 @@ processors:
metrics:
datapoint:
- attributes["container"] == "POD"
filter/cw_k8s_ci_v0_cadvisor_rollup:
error_mode: ignore
metrics:
datapoint:
- (attributes["container"] == nil or attributes["container"] == "") and attributes["pod"] != nil and attributes["pod"] != "" and IsMatch(metric.name, "^container_network_") != true
filter/cw_k8s_ci_v0_neuron:
error_mode: ignore
metrics:
Expand All @@ -181,7 +186,7 @@ processors:
error_mode: ignore
metrics:
metric:
- IsMatch(name, "^(up|scrape_duration_seconds|scrape_samples_scraped|scrape_samples_post_metric_relabeling|scrape_series_added)$")
- IsMatch(name, "^(up|scrape_duration_seconds|scrape_samples_scraped|scrape_samples_post_metric_relabeling|scrape_series_added|node_scrape_collector_duration_seconds|node_scrape_collector_success)$")
groupbyattrs/cw_k8s_ci_v0_cadvisor:
keys:
- container
Expand Down Expand Up @@ -1636,6 +1641,7 @@ service:
- transform/cw_k8s_ci_v0_set_cluster_name
- filter/cw_k8s_ci_v0_cadvisor_empty
- filter/cw_k8s_ci_v0_cadvisor_pod
- filter/cw_k8s_ci_v0_cadvisor_rollup
- groupbyattrs/cw_k8s_ci_v0_cadvisor
- transform/cw_k8s_ci_v0_cadvisor_promote
- transform/cw_k8s_ci_v0_set_node_name
Expand Down
8 changes: 7 additions & 1 deletion translator/tocwconfig/sampleConfig/opentelemetry/default_otel_config_aks.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -242,6 +242,11 @@ processors:
metrics:
datapoint:
- attributes["container"] == "POD"
filter/cw_k8s_ci_v0_cadvisor_rollup:
error_mode: ignore
metrics:
datapoint:
- (attributes["container"] == nil or attributes["container"] == "") and attributes["pod"] != nil and attributes["pod"] != "" and IsMatch(metric.name, "^container_network_") != true
filter/cw_k8s_ci_v0_neuron:
error_mode: ignore
metrics:
Expand All @@ -251,7 +256,7 @@ processors:
error_mode: ignore
metrics:
metric:
- IsMatch(name, "^(up|scrape_duration_seconds|scrape_samples_scraped|scrape_samples_post_metric_relabeling|scrape_series_added)$")
- IsMatch(name, "^(up|scrape_duration_seconds|scrape_samples_scraped|scrape_samples_post_metric_relabeling|scrape_series_added|node_scrape_collector_duration_seconds|node_scrape_collector_success)$")
groupbyattrs/cw_k8s_ci_v0_cadvisor:
keys:
- container
Expand Down Expand Up @@ -1628,6 +1633,7 @@ service:
- transform/cw_k8s_ci_v0_set_cluster_name
- filter/cw_k8s_ci_v0_cadvisor_empty
- filter/cw_k8s_ci_v0_cadvisor_pod
- filter/cw_k8s_ci_v0_cadvisor_rollup
- groupbyattrs/cw_k8s_ci_v0_cadvisor
- transform/cw_k8s_ci_v0_cadvisor_promote
- transform/cw_k8s_ci_v0_set_node_name
Expand Down
8 changes: 7 additions & 1 deletion translator/tocwconfig/sampleConfig/opentelemetry/default_otel_config_gke.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -242,6 +242,11 @@ processors:
metrics:
datapoint:
- attributes["container"] == "POD"
filter/cw_k8s_ci_v0_cadvisor_rollup:
error_mode: ignore
metrics:
datapoint:
- (attributes["container"] == nil or attributes["container"] == "") and attributes["pod"] != nil and attributes["pod"] != "" and IsMatch(metric.name, "^container_network_") != true
filter/cw_k8s_ci_v0_neuron:
error_mode: ignore
metrics:
Expand All @@ -251,7 +256,7 @@ processors:
error_mode: ignore
metrics:
metric:
- IsMatch(name, "^(up|scrape_duration_seconds|scrape_samples_scraped|scrape_samples_post_metric_relabeling|scrape_series_added)$")
- IsMatch(name, "^(up|scrape_duration_seconds|scrape_samples_scraped|scrape_samples_post_metric_relabeling|scrape_series_added|node_scrape_collector_duration_seconds|node_scrape_collector_success)$")
groupbyattrs/cw_k8s_ci_v0_cadvisor:
keys:
- container
Expand Down Expand Up @@ -1627,6 +1632,7 @@ service:
- transform/cw_k8s_ci_v0_set_cluster_name
- filter/cw_k8s_ci_v0_cadvisor_empty
- filter/cw_k8s_ci_v0_cadvisor_pod
- filter/cw_k8s_ci_v0_cadvisor_rollup
- groupbyattrs/cw_k8s_ci_v0_cadvisor
- transform/cw_k8s_ci_v0_cadvisor_promote
- transform/cw_k8s_ci_v0_set_node_name
Expand Down
4 changes: 3 additions & 1 deletion translator/translate/otel/pipeline/opentelemetry/containerinsights/apiserver.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,11 +33,13 @@ receivers:
server_name: {{ .ApiserverTLSServerName }}
{{- end }}
processors:
# All CI pipelines declare this processor under one name and it is merged, so every
# copy must list the same names. node_scrape_collector_* come from node-exporter.
filter/cw_k8s_ci_v0_scrape_metadata:
error_mode: ignore
metrics:
metric:
- IsMatch(name, "^(up|scrape_duration_seconds|scrape_samples_scraped|scrape_samples_post_metric_relabeling|scrape_series_added)$")
- IsMatch(name, "^(up|scrape_duration_seconds|scrape_samples_scraped|scrape_samples_post_metric_relabeling|scrape_series_added|node_scrape_collector_duration_seconds|node_scrape_collector_success)$")
transform/cw_k8s_ci_v0_set_unit:
error_mode: ignore
metric_statements:
Expand Down
12 changes: 11 additions & 1 deletion translator/translate/otel/pipeline/opentelemetry/containerinsights/cadvisor.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,11 +17,13 @@ receivers:
tls_config:
insecure_skip_verify: true
processors:
# All CI pipelines declare this processor under one name and it is merged, so every
# copy must list the same names. node_scrape_collector_* come from node-exporter.
filter/cw_k8s_ci_v0_scrape_metadata:
error_mode: ignore
metrics:
metric:
- IsMatch(name, "^(up|scrape_duration_seconds|scrape_samples_scraped|scrape_samples_post_metric_relabeling|scrape_series_added)$")
- IsMatch(name, "^(up|scrape_duration_seconds|scrape_samples_scraped|scrape_samples_post_metric_relabeling|scrape_series_added|node_scrape_collector_duration_seconds|node_scrape_collector_success)$")
transform/cw_k8s_ci_v0_set_unit:
error_mode: ignore
metric_statements:
Expand Down Expand Up @@ -78,6 +80,13 @@ processors:
metrics:
datapoint:
- attributes["container"] == "POD"
# Drop the pod cgroup slice and pause/sandbox duplicates of each container_* family.
# container_network_* is sandbox-only so must stay; `container` arrives absent, not "".
filter/cw_k8s_ci_v0_cadvisor_rollup:
error_mode: ignore
metrics:
datapoint:
- (attributes["container"] == nil or attributes["container"] == "") and attributes["pod"] != nil and attributes["pod"] != "" and IsMatch(metric.name, "^container_network_") != true
groupbyattrs/cw_k8s_ci_v0_cadvisor:
keys:
- container
Expand Down Expand Up @@ -225,6 +234,7 @@ service:
- transform/cw_k8s_ci_v0_set_cluster_name
- filter/cw_k8s_ci_v0_cadvisor_empty
- filter/cw_k8s_ci_v0_cadvisor_pod
- filter/cw_k8s_ci_v0_cadvisor_rollup
- groupbyattrs/cw_k8s_ci_v0_cadvisor
- transform/cw_k8s_ci_v0_cadvisor_promote
- transform/cw_k8s_ci_v0_set_node_name
Expand Down
4 changes: 3 additions & 1 deletion translator/translate/otel/pipeline/opentelemetry/containerinsights/dcgm.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,11 +15,13 @@ receivers:
tls_config:
ca_file: /etc/amazon-cloudwatch-observability-agent-client-cert/tls-ca.crt
processors:
# All CI pipelines declare this processor under one name and it is merged, so every
# copy must list the same names. node_scrape_collector_* come from node-exporter.
filter/cw_k8s_ci_v0_scrape_metadata:
error_mode: ignore
metrics:
metric:
- IsMatch(name, "^(up|scrape_duration_seconds|scrape_samples_scraped|scrape_samples_post_metric_relabeling|scrape_series_added)$")
- IsMatch(name, "^(up|scrape_duration_seconds|scrape_samples_scraped|scrape_samples_post_metric_relabeling|scrape_series_added|node_scrape_collector_duration_seconds|node_scrape_collector_success)$")
transform/cw_k8s_ci_v0_set_unit:
error_mode: ignore
metric_statements:
Expand Down
4 changes: 3 additions & 1 deletion translator/translate/otel/pipeline/opentelemetry/containerinsights/ebs_csi.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,11 +27,13 @@ receivers:
scrape_interval: {{.CollectionInterval}}
scrape_timeout: {{.ScrapeTimeout}}
processors:
# All CI pipelines declare this processor under one name and it is merged, so every
# copy must list the same names. node_scrape_collector_* come from node-exporter.
filter/cw_k8s_ci_v0_scrape_metadata:
error_mode: ignore
metrics:
metric:
- IsMatch(name, "^(up|scrape_duration_seconds|scrape_samples_scraped|scrape_samples_post_metric_relabeling|scrape_series_added)$")
- IsMatch(name, "^(up|scrape_duration_seconds|scrape_samples_scraped|scrape_samples_post_metric_relabeling|scrape_series_added|node_scrape_collector_duration_seconds|node_scrape_collector_success)$")
transform/cw_k8s_ci_v0_set_unit:
error_mode: ignore
metric_statements:
Expand Down
4 changes: 3 additions & 1 deletion translator/translate/otel/pipeline/opentelemetry/containerinsights/karpenter.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,11 +28,13 @@ receivers:
- source_labels: [__meta_kubernetes_pod_node_name]
target_label: node
processors:
# All CI pipelines declare this processor under one name and it is merged, so every
# copy must list the same names. node_scrape_collector_* come from node-exporter.
filter/cw_k8s_ci_v0_scrape_metadata:
error_mode: ignore
metrics:
metric:
- IsMatch(name, "^(up|scrape_duration_seconds|scrape_samples_scraped|scrape_samples_post_metric_relabeling|scrape_series_added)$")
- IsMatch(name, "^(up|scrape_duration_seconds|scrape_samples_scraped|scrape_samples_post_metric_relabeling|scrape_series_added|node_scrape_collector_duration_seconds|node_scrape_collector_success)$")
transform/cw_k8s_ci_v0_set_unit:
error_mode: ignore
metric_statements:
Expand Down
4 changes: 3 additions & 1 deletion translator/translate/otel/pipeline/opentelemetry/containerinsights/keda.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,11 +28,13 @@ receivers:
- source_labels: [__meta_kubernetes_pod_node_name]
target_label: node
processors:
# All CI pipelines declare this processor under one name and it is merged, so every
# copy must list the same names. node_scrape_collector_* come from node-exporter.
filter/cw_k8s_ci_v0_scrape_metadata:
error_mode: ignore
metrics:
metric:
- IsMatch(name, "^(up|scrape_duration_seconds|scrape_samples_scraped|scrape_samples_post_metric_relabeling|scrape_series_added)$")
- IsMatch(name, "^(up|scrape_duration_seconds|scrape_samples_scraped|scrape_samples_post_metric_relabeling|scrape_series_added|node_scrape_collector_duration_seconds|node_scrape_collector_success)$")
# Drop generic client-go / control-plane / Go-runtime metrics KEDA exposes; they
# collide with the apiserver pipeline which owns them under a different scope.
filter/cw_k8s_ci_v0_keda_drop_non_keda:
Expand Down
4 changes: 3 additions & 1 deletion translator/translate/otel/pipeline/opentelemetry/containerinsights/kube_state_metrics.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,11 +15,13 @@ receivers:
tls_config:
ca_file: /etc/amazon-cloudwatch-observability-agent-cert/tls-ca.crt
processors:
# All CI pipelines declare this processor under one name and it is merged, so every
# copy must list the same names. node_scrape_collector_* come from node-exporter.
filter/cw_k8s_ci_v0_scrape_metadata:
error_mode: ignore
metrics:
metric:
- IsMatch(name, "^(up|scrape_duration_seconds|scrape_samples_scraped|scrape_samples_post_metric_relabeling|scrape_series_added)$")
- IsMatch(name, "^(up|scrape_duration_seconds|scrape_samples_scraped|scrape_samples_post_metric_relabeling|scrape_series_added|node_scrape_collector_duration_seconds|node_scrape_collector_success)$")
transform/cw_k8s_ci_v0_set_unit:
error_mode: ignore
metric_statements:
Expand Down
4 changes: 3 additions & 1 deletion translator/translate/otel/pipeline/opentelemetry/containerinsights/lis_csi.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,11 +27,13 @@ receivers:
scrape_interval: {{.CollectionInterval}}
scrape_timeout: {{.ScrapeTimeout}}
processors:
# All CI pipelines declare this processor under one name and it is merged, so every
# copy must list the same names. node_scrape_collector_* come from node-exporter.
filter/cw_k8s_ci_v0_scrape_metadata:
error_mode: ignore
metrics:
metric:
- IsMatch(name, "^(up|scrape_duration_seconds|scrape_samples_scraped|scrape_samples_post_metric_relabeling|scrape_series_added)$")
- IsMatch(name, "^(up|scrape_duration_seconds|scrape_samples_scraped|scrape_samples_post_metric_relabeling|scrape_series_added|node_scrape_collector_duration_seconds|node_scrape_collector_success)$")
transform/cw_k8s_ci_v0_set_unit:
error_mode: ignore
metric_statements:
Expand Down
4 changes: 3 additions & 1 deletion translator/translate/otel/pipeline/opentelemetry/containerinsights/neuron.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,11 +15,13 @@ receivers:
tls_config:
ca_file: /etc/amazon-cloudwatch-observability-agent-client-cert/tls-ca.crt
processors:
# All CI pipelines declare this processor under one name and it is merged, so every
# copy must list the same names. node_scrape_collector_* come from node-exporter.
filter/cw_k8s_ci_v0_scrape_metadata:
error_mode: ignore
metrics:
metric:
- IsMatch(name, "^(up|scrape_duration_seconds|scrape_samples_scraped|scrape_samples_post_metric_relabeling|scrape_series_added)$")
- IsMatch(name, "^(up|scrape_duration_seconds|scrape_samples_scraped|scrape_samples_post_metric_relabeling|scrape_series_added|node_scrape_collector_duration_seconds|node_scrape_collector_success)$")
metricstarttime/cw_k8s_ci_v0: {}
transform/cw_k8s_ci_v0_set_cluster_name:
error_mode: ignore
Expand Down
4 changes: 3 additions & 1 deletion translator/translate/otel/pipeline/opentelemetry/containerinsights/node_exporter.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,11 +15,13 @@ receivers:
tls_config:
ca_file: /etc/amazon-cloudwatch-observability-agent-client-cert/tls-ca.crt
processors:
# node-exporter's own per-collector scrape timing/status, same category as the five
# scrape_* names. node_textfile_scrape_error stays: kubernetes-mixin alerts on it.
filter/cw_k8s_ci_v0_scrape_metadata:
error_mode: ignore
metrics:
metric:
- IsMatch(name, "^(up|scrape_duration_seconds|scrape_samples_scraped|scrape_samples_post_metric_relabeling|scrape_series_added)$")
- IsMatch(name, "^(up|scrape_duration_seconds|scrape_samples_scraped|scrape_samples_post_metric_relabeling|scrape_series_added|node_scrape_collector_duration_seconds|node_scrape_collector_success)$")
transform/cw_k8s_ci_v0_set_unit:
error_mode: ignore
metric_statements:
Expand Down
Loading