Full Observability Stack

Complete three-pillar observability platform — metrics, logs, and traces — deployed to Kubernetes via GitOps (ArgoCD) with SLO tracking and automated alerting.

How the pieces fit together

Component	Deployed as	Role
Prometheus + Alertmanager	kube-prometheus-stack Helm chart, via ArgoCD	Scrapes cluster/pod/app metrics, evaluates alert rules, routes notifications
Grafana	Bundled with kube-prometheus-stack (grafana.enabled: true)	Dashboards, with Prometheus/Loki/Tempo wired up as datasources
Loki + Promtail	loki-stack Helm chart, via ArgoCD	Promtail tails pod logs on every node and ships them to Loki for aggregation/query
Tempo	tempo Helm chart, via ArgoCD	Stores distributed traces (OTLP/Jaeger/Zipkin ingest) and generates metrics from trace data back into Prometheus

ArgoCD is the deployment mechanism: each argocd/*.yaml file is an Application resource that points ArgoCD at an upstream Helm chart (Prometheus community / Grafana charts) with pinned targetRevisions, syncing it into the monitoring namespace with automated prune + self-heal.

Grafana is pre-wired with Loki and Tempo as datasources (prometheus/values.yaml), so metrics, logs, and traces are queryable from one place, and slo-alerts.yaml / platform-alerts.yaml turn the raw metrics into paging-worthy signals.

Folder structure

argocd/                        ArgoCD Application manifests (one per chart)
  loki.yaml                    Deploys the loki-stack chart (Loki + Promtail)
  prometheus.yaml              Deploys kube-prometheus-stack (Prometheus, Alertmanager, Grafana)
  tempo.yaml                   Deploys the tempo chart
prometheus/
  values.yaml                  Helm values for kube-prometheus-stack (retention, scrape configs, Grafana datasources/dashboards, Alertmanager routing)
  alertmanager-config.example.yaml   Documents the Alertmanager Secret layout (see Security section)
  alerts/
    platform-alerts.yaml       PrometheusRule: infra/pod/resource/HTTP alerts
    slo-alerts.yaml            PrometheusRule: SLO error-budget burn-rate alerts
loki/values.yaml                Helm values for loki-stack (Loki storage/limits, Promtail pipeline)
tempo/values.yaml                Helm values for the tempo chart (receivers, retention, metrics-generator)
grafana/dashboards/              Dashboard JSON exports (provisioned into the "Platform" folder)
scripts/
  slo_calculator.py            Queries Prometheus and prints SLO/error-budget status
  requirements.txt              Python dependencies for slo_calculator.py

SLO Tracking

Two SLOs tracked out of the box:

• Availability SLO — 99.9% HTTP success rate (30-day window)
• Latency SLO — P99 response time ≤ 1 second

Error budget burn rate alerts fire at 14× (critical) and 6× (warning) before the budget is exhausted — following Google SRE alert methodology. These rely on your services exposing http_requests_total and http_request_duration_seconds_bucket metrics (standard Prometheus client library naming) — adjust the PromQL in prometheus/alerts/slo-alerts.yaml if your app uses different metric names.

Prerequisites

• A Kubernetes cluster you can kubectl into, with a default StorageClass available for the PersistentVolumeClaims used by Prometheus/Loki/Tempo/Grafana.
• ArgoCD installed in the cluster (argocd namespace) — this repo only contains the Application manifests ArgoCD consumes, not ArgoCD itself.
• kubectl and (optionally) the argocd CLI configured against the cluster.
• Python 3.9+ if you want to run scripts/slo_calculator.py locally.

Security setup (do this before syncing)

Two components read credentials from Kubernetes Secrets rather than from files in this repo — create them first so the ArgoCD sync doesn't come up with default/empty credentials:

# Grafana admin login (kube-prometheus-stack defaults to a well-known
# password if this isn't set — always override it)
kubectl create secret generic grafana-admin-credentials \
  --namespace monitoring \
  --from-literal=admin-user=admin \
  --from-literal=admin-password='<strong-random-password>'

# Alertmanager routing (Slack webhook URL / PagerDuty routing key).
# See prometheus/alertmanager-config.example.yaml for the file layout.
kubectl create secret generic alertmanager-platform-config \
  --namespace monitoring \
  --from-file=alertmanager.yaml=./alertmanager-config.yaml

Never commit a filled-in alertmanager-config.yaml or real credentials — .gitignore already excludes common secret-shaped filenames (*.pem, *.key, kubeconfig, *-credentials.yaml, alertmanager-config.yaml, etc.).

Deploy

# Apply all ArgoCD applications
kubectl apply -f argocd/

# Check SLO status
pip install -r scripts/requirements.txt
python scripts/slo_calculator.py --prometheus http://localhost:9090

# Access Grafana
kubectl port-forward svc/kube-prometheus-stack-grafana 3000:80 -n monitoring
# Open http://localhost:3000 and log in with the credentials from
# the grafana-admin-credentials secret created above.

Note: argocd/prometheus.yaml references helm.valueFiles: [values.yaml] against the upstream kube-prometheus-stack chart repo. For ArgoCD to resolve prometheus/values.yaml from this git repo, point the Application at this repo as an additional Helm value source (ArgoCD multiple sources) or apply the chart with -f prometheus/values.yaml directly if managing it outside ArgoCD.

Alert Coverage

• Pod crash looping
• Deployment replica mismatch
• CPU/Memory > 90% of limits
• PersistentVolume > 85% full
• HTTP 5xx rate > 5%
• P99 latency > 2 seconds
• SLO error budget burn rate (fast + slow burn)