diff --git a/.gitattributes b/.gitattributes index 569be426630d..8b238566943d 100644 --- a/.gitattributes +++ b/.gitattributes @@ -9,3 +9,10 @@ versions.lock text eol=lf /.* export-ignore /dev-docs export-ignore AGENTS.md export-ignore + +# Monitoring mixin generated outputs — always LF to ensure make regeneration is idempotent. +solr/monitoring/grafana-solr-dashboard.json text eol=lf +solr/monitoring/prometheus-solr-alerts.yml text eol=lf + +# Contributor-only tooling, excluded from the source release like dev-docs. +solr/monitoring/dev export-ignore diff --git a/changelog/unreleased/SOLR-18533-monitoring-dev-tooling.yml b/changelog/unreleased/SOLR-18533-monitoring-dev-tooling.yml new file mode 100644 index 000000000000..33d0f76a65e2 --- /dev/null +++ b/changelog/unreleased/SOLR-18533-monitoring-dev-tooling.yml @@ -0,0 +1,14 @@ +# See https://github.com/apache/solr/blob/main/dev-docs/changelog.adoc + +title: > + Add tooling for building Solr's Grafana dashboard and Prometheus alert rules from jsonnet sources, + under `solr/monitoring`, together with a docker-compose stack for testing them locally. The + dashboard and alert rules shipped so far are placeholders: the real dashboard follows in + SOLR-18534 and the real alert rules in SOLR-18535. +type: other +authors: + - name: Jan Høydahl + url: https://home.apache.org/phonebook.html?uid=janhoy +links: + - name: SOLR-18533 + url: https://issues.apache.org/jira/browse/SOLR-18533 diff --git a/solr/monitoring/README.md b/solr/monitoring/README.md new file mode 100644 index 000000000000..b7131b0a9978 --- /dev/null +++ b/solr/monitoring/README.md @@ -0,0 +1,203 @@ + +# Solr Monitoring + +This directory provides two ready-to-use monitoring artifacts for Solr that you can drop into +your existing Prometheus + Grafana installation: + +| File | Description | +|---|---| +| **`grafana-solr-dashboard.json`** | Grafana dashboard — import directly into Grafana | +| **`prometheus-solr-alerts.yml`** | Prometheus alert rules — reference from your `prometheus.yml` | + +These are the **main artifacts**. Everything else in this directory supports their creation +or testing. + +| File / Directory | Description | +|---|---| +| `mixin/` | Jsonnet source (single source of truth used to regenerate the artifacts above) | +| `dev/` | **Developer convenience only** — docker-compose stack for testing changes to the artifacts | + +> **Both artifacts are placeholders right now** — a two-panel dashboard and a single alert, which +> exist to prove the build pipeline works end to end. The real dashboard arrives in +> [SOLR-18534](https://issues.apache.org/jira/browse/SOLR-18534), the alert rules in +> [SOLR-18535](https://issues.apache.org/jira/browse/SOLR-18535), and OTLP push support in +> [SOLR-18536](https://issues.apache.org/jira/browse/SOLR-18536). + +For the full integration guide see the Solr Reference Guide: +**[Monitoring Solr with Prometheus and Grafana](../solr-ref-guide/modules/deployment-guide/pages/monitoring-with-prometheus-and-grafana.adoc)** + +--- + +## Importing the Dashboard + +1. Open Grafana → **Dashboards → Import** +2. Upload `grafana-solr-dashboard.json` or paste its contents +3. Select your Prometheus datasource when prompted +4. Click **Import** + +The placeholder dashboard has two panels, Request Rate and Heap Used, and two dropdowns: +`datasource` and `interval`. + +--- + +## Loading Alert Rules into Prometheus + +Add the alert rules file to your Prometheus configuration: + +```yaml +# prometheus.yml +rule_files: + - /path/to/prometheus-solr-alerts.yml +``` + +Validate locally before deploying: + +```bash +promtool check rules prometheus-solr-alerts.yml +``` + +One placeholder rule is included: `SolrNodeDown`, which fires when a Solr target has failed to +scrape for 2 minutes. + +--- + +## Customizing + +All tunable settings live in `mixin/config.libsonnet`: + +```jsonnet +{ + _config:: { + solrSelector: 'job="solr"', // label selector identifying Solr targets + rateInterval: '5m', // window for rate() queries + }, +} +``` + +`solrSelector` must match your Prometheus scrape job, or no panel or rule will match a series. +Edit the file and regenerate, or patch `_config` from your own jsonnet without touching this tree: + +```jsonnet +(import 'mixin.libsonnet') + { _config+:: { solrSelector: 'job="my-solr"' } } +``` + +Every key in `config.libsonnet` is consumed by the dashboard or the alert rules; the file grows as +they do. + +--- + +## Mixin Build + +Only contributors who need to **regenerate** the artifacts need build tooling. +Operators can use the pre-generated files directly. + +There are two equivalent ways to build: + +### Option A — Docker (no local tool installation required) + +```bash +cd mixin +./make.sh install # fetch grafonnet dependency into vendor/ +./make.sh all # regenerate both artifacts +./make.sh check # validate alert rules +``` + +`make.sh` automatically builds the `solr-mixin-make:latest` Docker image on first run, with all +tool versions pinned in `mixin/Dockerfile`. Pass `--rebuild` to force a fresh image build +(e.g. after a Dockerfile update). + +### Option B — Local tools + +Install the required tools once: + +| Tool | Purpose | Install | +|---|---|---| +| `jsonnet` (go-jsonnet) | Evaluate jsonnet source | `brew install go-jsonnet` or `go install github.com/google/go-jsonnet/cmd/jsonnet@latest` | +| `jb` (jsonnet-bundler) | Manage dependencies | `brew install jsonnet-bundler` or `go install github.com/jsonnet-bundler/jsonnet-bundler/cmd/jb@latest` | +| `gojsontoyaml` | Convert JSON→YAML for alert rules | `go install github.com/brancz/gojsontoyaml@latest` | +| `promtool` *(optional)* | Validate alert rules | `brew install prometheus` | + +Then build: + +```bash +cd mixin +make install # fetch grafonnet dependency into vendor/ +make all # regenerate both artifacts +``` + +A target aborts before touching any output if a tool is missing, and writes through a temp file, +so a failed build cannot leave a half-written artifact behind. + +### Makefile targets + +Both `./make.sh ` and `make ` accept the same targets: + +| Target | Action | +|---|---| +| `install` | Download jsonnet dependencies (grafonnet) into `vendor/` | +| `dashboards` | Regenerate `grafana-solr-dashboard.json` | +| `alerts` | Regenerate `prometheus-solr-alerts.yml` | +| `all` | Regenerate both outputs | +| `check` | Run `promtool check rules` on alert rules | +| `fmt` | Auto-format all `.libsonnet` source files | + +`vendor/` is not checked in; `jsonnetfile.lock.json` pins the exact grafonnet revision, so +`make install` reproduces the same dependency tree. + +**Commit the regenerated artifacts together with the jsonnet change that produced them** — +nothing in the Gradle build regenerates or verifies them. + +--- + +## Developer Convenience Stack + +The `dev/` directory contains a docker-compose stack that starts a local Solr + Prometheus ++ Grafana + Alertmanager environment. **This is solely for contributors who want to +visually test changes to the two artifacts above.** It is not a reference deployment +or a recommended production setup, and is excluded from the source release. + +```bash +cd dev +./stack.sh # bundled Solr nodes + monitoring + traffic generator +./stack.sh --local-solr # monitoring only, scraping a Solr you already run on :8983 +./stack.sh --down # stop and remove everything +``` + +Use `stack.sh` rather than `docker compose up` directly: the bundled Solr nodes sit behind the +`bundled-solr` compose profile, so a plain `docker compose up` would start the monitoring services +with no Solr to scrape. + +Services: Solr (`:8983`, `:8984`), Prometheus (`:9090`), Grafana (`:3000`), Alertmanager +(`:9093`). Grafana opens straight on the provisioned dashboard with no login — anonymous viewing +is enabled because this stack is local and throwaway; sign in as `admin`/`admin` to edit. The +Prometheus datasource is provisioned too, and Prometheus loads the alert rules. Both artifacts are bind-mounted as single +files, which the containers resolve once at start, so after regenerating one, recreate them: + +```bash +docker compose up -d --force-recreate grafana prometheus +``` + +The monitoring images are pinned for reproducibility. Override `SOLR_IMAGE`, `PROMETHEUS_IMAGE`, +`GRAFANA_IMAGE` or `ALERTMANAGER_IMAGE` to test against other versions: + +```bash +GRAFANA_IMAGE=grafana/grafana:latest ./stack.sh +``` + +`trafficgen/` is a small Python container that indexes and queries continuously, so the panels +show movement. It exists for this stack only. diff --git a/solr/monitoring/dev/alertmanager/alertmanager.yml b/solr/monitoring/dev/alertmanager/alertmanager.yml new file mode 100644 index 000000000000..dd475ba305f5 --- /dev/null +++ b/solr/monitoring/dev/alertmanager/alertmanager.yml @@ -0,0 +1,86 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# alertmanager.yml — Example Alertmanager routing configuration for Solr alerts. +# +# Two-tier severity routing: +# severity: critical → critical-receiver (intended for: PagerDuty, OpsGenie, paging on-call) +# severity: warning → warning-receiver (intended for: Slack, email, non-urgent notification) +# +# Replace the stub "null" receivers with your real notification integrations. +# See https://prometheus.io/docs/alerting/latest/configuration/ for receiver examples. + +global: + resolve_timeout: 5m + +route: + # Default receiver for any alert not matched by child routes. + receiver: "default" + group_by: ["alertname", "instance"] + group_wait: 30s + group_interval: 5m + repeat_interval: 4h + + routes: + # Critical alerts — page the on-call engineer immediately. + # Solr critical alerts: SolrHighHeapUsage, SolrJvmGcThrashing, SolrLowDiskSpace + - matchers: + - severity = "critical" + receiver: "critical-receiver" + group_wait: 10s + repeat_interval: 1h + + # Warning alerts — notify the team but do not page. + # Solr warning alerts: SolrHighSearchLatency, SolrHighErrorRate, + # SolrOverseerQueueBuildup, SolrHighMmapRatio + - matchers: + - severity = "warning" + receiver: "warning-receiver" + repeat_interval: 6h + +receivers: + # Default catch-all receiver (no-op stub). + - name: "default" + + # Critical receiver — replace with PagerDuty, OpsGenie, VictorOps, or similar. + # PagerDuty example: + # - name: "critical-receiver" + # pagerduty_configs: + # - service_key: YOUR_PAGERDUTY_INTEGRATION_KEY + - name: "critical-receiver" + + # Warning receiver — replace with Slack, email, or similar. + # Slack example: + # - name: "warning-receiver" + # slack_configs: + # - api_url: https://hooks.slack.com/services/YOUR/SLACK/WEBHOOK + # channel: '#solr-alerts' + # title: '{{ .GroupLabels.alertname }}' + # text: '{{ range .Alerts }}{{ .Annotations.description }}{{ end }}' + # Email example: + # - name: "warning-receiver" + # email_configs: + # - to: 'solr-team@example.com' + # from: 'alertmanager@example.com' + # smarthost: 'smtp.example.com:587' + - name: "warning-receiver" + +inhibit_rules: + # Suppress warning alerts on an instance when a critical alert is already firing for it. + - source_matchers: + - severity = "critical" + target_matchers: + - severity = "warning" + equal: ["instance"] diff --git a/solr/monitoring/dev/docker-compose.yml b/solr/monitoring/dev/docker-compose.yml new file mode 100644 index 000000000000..4af98d4691b2 --- /dev/null +++ b/solr/monitoring/dev/docker-compose.yml @@ -0,0 +1,155 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# ============================================================================ +# DEVELOPER CONVENIENCE ONLY — NOT FOR PRODUCTION USE +# ============================================================================ +# +# This docker-compose stack is provided solely to make it easy for contributors +# to view and test changes to the two generated monitoring artifacts: +# +# ../grafana-solr-dashboard.json — Grafana dashboard +# ../prometheus-solr-alerts.yml — Prometheus alert rules +# +# It is NOT intended as a reference deployment or a recommended production setup. +# For integrating the artifacts into your own Prometheus + Grafana installation, +# see the Solr Reference Guide (monitoring-with-prometheus-and-grafana.adoc). +# +# Quick start: +# cd solr/monitoring/dev +# ./stack.sh # Full stack with bundled Solr nodes +# ./stack.sh --local-solr # Monitoring only — point at your locally-running Solr +# ./stack.sh --help # All options +# +# Ports are published on loopback only: the stack runs Solr and ZooKeeper without +# authentication, so it must not be reachable from other hosts on the network. +# +# Services (full stack): +# solr1 http://localhost:8983 (SolrCloud node 1, embedded ZooKeeper) +# solr2 http://localhost:8984 (SolrCloud node 2) +# Prometheus http://localhost:9090 +# Grafana http://localhost:3000 (admin / admin) +# Alertmanager http://localhost:9093 +# trafficgen (no port — continuously indexes/searches the 'test' collection) +# +# Environment variables (set by stack.sh, or override manually): +# PROMETHEUS_CONFIG Path to the prometheus config to mount (default: ./prometheus/prometheus.yml) +# SOLR_URL Solr base URL for trafficgen (default: http://solr1:8983/solr) +# SOLR_IMAGE Solr image for the bundled nodes (default: solr:10) +# PROMETHEUS_IMAGE Prometheus image (default: pinned, see below) +# GRAFANA_IMAGE Grafana image (default: pinned, see below) +# ALERTMANAGER_IMAGE Alertmanager image (default: pinned, see below) +# +# The monitoring images are pinned so a stack that worked yesterday still works +# today; override any of them to test against a newer release. +# ============================================================================ + +services: + solr1: + image: ${SOLR_IMAGE:-solr:10} + profiles: [bundled-solr] + ports: + - "127.0.0.1:8983:8983" + - "127.0.0.1:9983:9983" # embedded ZooKeeper (Solr 10 default: ZK_PORT = SOLR_PORT + 1000) + # No ZK_HOST set: Solr 10 defaults to SolrCloud with embedded ZooKeeper on port 9983. + healthcheck: + test: ["CMD-SHELL", "curl -sf http://localhost:8983/solr/admin/info/system || exit 1"] + interval: 10s + timeout: 5s + retries: 12 + start_period: 40s + + solr2: + image: ${SOLR_IMAGE:-solr:10} + profiles: [bundled-solr] + ports: + - "127.0.0.1:8984:8983" + environment: + ZK_HOST: "solr1:9983" + depends_on: + solr1: + condition: service_healthy + healthcheck: + test: ["CMD-SHELL", "curl -sf http://localhost:8983/solr/admin/info/system || exit 1"] + interval: 10s + timeout: 5s + retries: 12 + start_period: 30s + + prometheus: + image: ${PROMETHEUS_IMAGE:-prom/prometheus:v3.15.0} + ports: + - "127.0.0.1:9090:9090" + extra_hosts: + - "host.docker.internal:host-gateway" + volumes: + - ${PROMETHEUS_CONFIG:-./prometheus/prometheus.yml}:/etc/prometheus/prometheus.yml:ro + - ../prometheus-solr-alerts.yml:/etc/prometheus/rules/solr-alerts.yml:ro + command: + - "--config.file=/etc/prometheus/prometheus.yml" + - "--storage.tsdb.retention.time=7d" + - "--web.enable-lifecycle" + depends_on: + solr1: + condition: service_healthy + required: false + solr2: + condition: service_healthy + required: false + restart: unless-stopped + + grafana: + image: ${GRAFANA_IMAGE:-grafana/grafana:13.2.3} + ports: + - "127.0.0.1:3000:3000" + environment: + GF_SECURITY_ADMIN_USER: admin + GF_SECURITY_ADMIN_PASSWORD: admin + GF_USERS_ALLOW_SIGN_UP: "false" + # Local throwaway stack: let anyone view without logging in, so the dashboard + # is one click away. Log in as admin/admin to edit. + GF_AUTH_ANONYMOUS_ENABLED: "true" + GF_AUTH_ANONYMOUS_ORG_ROLE: Viewer + # Land on the provisioned dashboard instead of Grafana's welcome page. + GF_DASHBOARDS_DEFAULT_HOME_DASHBOARD_PATH: /var/lib/grafana/dashboards/grafana-solr-dashboard.json + volumes: + - ./grafana/provisioning:/etc/grafana/provisioning:ro + - ../grafana-solr-dashboard.json:/var/lib/grafana/dashboards/grafana-solr-dashboard.json:ro + depends_on: + - prometheus + restart: unless-stopped + + alertmanager: + image: ${ALERTMANAGER_IMAGE:-prom/alertmanager:v0.34.1} + ports: + - "127.0.0.1:9093:9093" + volumes: + - ./alertmanager/alertmanager.yml:/etc/alertmanager/alertmanager.yml:ro + restart: unless-stopped + + trafficgen: + build: ./trafficgen + extra_hosts: + - "host.docker.internal:host-gateway" + environment: + SOLR_URL: ${SOLR_URL:-http://solr1:8983/solr} + depends_on: + solr1: + condition: service_healthy + required: false + solr2: + condition: service_healthy + required: false + restart: unless-stopped diff --git a/solr/monitoring/dev/grafana/provisioning/dashboards/dashboards.yml b/solr/monitoring/dev/grafana/provisioning/dashboards/dashboards.yml new file mode 100644 index 000000000000..8fab25f256f2 --- /dev/null +++ b/solr/monitoring/dev/grafana/provisioning/dashboards/dashboards.yml @@ -0,0 +1,28 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# Grafana dashboard provisioning — tells Grafana to load dashboards from the +# /var/lib/grafana/dashboards directory (mounted by docker-compose.yml). + +apiVersion: 1 + +providers: + - name: Solr + folder: Solr + type: file + disableDeletion: false + editable: false + options: + path: /var/lib/grafana/dashboards diff --git a/solr/monitoring/dev/grafana/provisioning/datasources/prometheus.yml b/solr/monitoring/dev/grafana/provisioning/datasources/prometheus.yml new file mode 100644 index 000000000000..1e877ed476ec --- /dev/null +++ b/solr/monitoring/dev/grafana/provisioning/datasources/prometheus.yml @@ -0,0 +1,29 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# Grafana datasource provisioning — automatically configures the Prometheus +# datasource when Grafana starts (used by docker-compose.yml). + +apiVersion: 1 + +datasources: + - name: Prometheus + type: prometheus + access: proxy + url: http://prometheus:9090 + isDefault: true + editable: false + jsonData: + timeInterval: "15s" diff --git a/solr/monitoring/dev/prometheus/prometheus-local.yml b/solr/monitoring/dev/prometheus/prometheus-local.yml new file mode 100644 index 000000000000..b3935124c3f7 --- /dev/null +++ b/solr/monitoring/dev/prometheus/prometheus-local.yml @@ -0,0 +1,48 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +# prometheus-local.yml — scrape config for a Solr you run yourself. +# +# Used by: ./stack.sh --local-solr +# +# 'host.docker.internal' resolves to the host machine from inside Docker (Mac, +# Windows, and Linux with Docker 20.10+). docker-compose.yml adds the matching +# extra_hosts entry so this also works on Linux. +# +# A single node on :8983 is scraped by default. Add further ports to the targets +# list to match a multi-node local SolrCloud setup. + +global: + scrape_interval: 15s + evaluation_interval: 15s + +rule_files: + - /etc/prometheus/rules/solr-alerts.yml + +# Hand firing alerts to the Alertmanager container in this stack; without this +# the rules evaluate but nothing is ever delivered. +alerting: + alertmanagers: + - static_configs: + - targets: ['alertmanager:9093'] + +scrape_configs: + - job_name: solr + # The v1 endpoint, which every Solr 10.x build serves. Solr 10.1 and later + # also serve the shorter v2 alias /api/metrics. + metrics_path: /solr/admin/metrics + scrape_interval: 15s + static_configs: + - targets: ['host.docker.internal:8983'] diff --git a/solr/monitoring/dev/prometheus/prometheus.yml b/solr/monitoring/dev/prometheus/prometheus.yml new file mode 100644 index 000000000000..821ce876be71 --- /dev/null +++ b/solr/monitoring/dev/prometheus/prometheus.yml @@ -0,0 +1,43 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +# prometheus.yml — scrape config for the bundled Solr nodes in the dev stack. +# +# Used by docker-compose.yml in this directory. The job name must match the +# mixin's solrSelector (job="solr" by default), or the generated dashboard and +# alert rules will not match any series. + +global: + scrape_interval: 15s + evaluation_interval: 15s + +rule_files: + - /etc/prometheus/rules/solr-alerts.yml + +# Hand firing alerts to the Alertmanager container in this stack; without this +# the rules evaluate but nothing is ever delivered. +alerting: + alertmanagers: + - static_configs: + - targets: ['alertmanager:9093'] + +scrape_configs: + - job_name: solr + # The v1 endpoint, which every Solr 10.x build serves. Solr 10.1 and later + # also serve the shorter v2 alias /api/metrics. + metrics_path: /solr/admin/metrics + scrape_interval: 15s + static_configs: + - targets: ['solr1:8983', 'solr2:8983'] diff --git a/solr/monitoring/dev/stack.sh b/solr/monitoring/dev/stack.sh new file mode 100755 index 000000000000..3d6129faef0d --- /dev/null +++ b/solr/monitoring/dev/stack.sh @@ -0,0 +1,130 @@ +#!/usr/bin/env bash +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# stack.sh — convenience launcher for the Solr monitoring dev stack. +# +# Usage: +# ./stack.sh Full stack: bundled Solr + monitoring + trafficgen +# ./stack.sh --local-solr Monitoring only; Prometheus scrapes your local Solr +# at localhost:8983 (no bundled Solr started) +# ./stack.sh --local-solr URL Same, but trafficgen is also pointed at URL +# (implies --trafficgen) +# ./stack.sh --local-solr --trafficgen Local mode + trafficgen at default localhost:8983 +# ./stack.sh --down Stop and remove all containers +# ./stack.sh --help Show this help +# +# In --local-solr mode Prometheus uses prometheus/prometheus-local.yml which scrapes +# host.docker.internal:8983. Edit that file to add further nodes or change the port. + +set -euo pipefail + +cd "$(dirname "$0")" + +LOCAL_MODE=false +WITH_TRAFFICGEN=false +CUSTOM_SOLR_URL="" +DOWN_MODE=false + +usage() { + awk '/^# Usage:/,/^[^#]/' "$0" | grep '^#' | sed 's/^# \{0,1\}//' + exit 0 +} + +# Translate long options to short equivalents so getopts can process them uniformly. +_args=() +for _a in "$@"; do + case "$_a" in + --local-solr) _args+=(-l) ;; + --trafficgen) _args+=(-t) ;; + --down) _args+=(-d) ;; + --help) _args+=(-h) ;; + *) _args+=("$_a") ;; + esac +done +set -- "${_args[@]+"${_args[@]}"}" + +while getopts "ltdh" opt; do + case "$opt" in + l) LOCAL_MODE=true ;; + t) WITH_TRAFFICGEN=true ;; + d) DOWN_MODE=true ;; + h) usage ;; + ?) echo "Unknown option. Run ./stack.sh --help for usage." >&2; exit 1 ;; + esac +done +shift $((OPTIND - 1)) + +# Optional URL positional argument following -l / --local-solr +if $LOCAL_MODE && [[ $# -gt 0 ]]; then + CUSTOM_SOLR_URL="$1" + WITH_TRAFFICGEN=true + shift +fi + +if $DOWN_MODE; then + # Include the bundled-solr profile so solr1/solr2 are stopped even if they were started + # with that profile active. Passing --profile to a non-running service is harmless. + docker compose --profile bundled-solr down + echo "All containers stopped and removed." + exit 0 +fi + +if $LOCAL_MODE; then + export PROMETHEUS_CONFIG="./prometheus/prometheus-local.yml" + + if [[ -n "$CUSTOM_SOLR_URL" ]]; then + # Replace 'localhost' with host.docker.internal so trafficgen can reach the host + export SOLR_URL="${CUSTOM_SOLR_URL/localhost/host.docker.internal}" + else + export SOLR_URL="http://host.docker.internal:8983/solr" + fi + + SERVICES="prometheus grafana alertmanager" + if $WITH_TRAFFICGEN; then + SERVICES="$SERVICES trafficgen" + fi + + # shellcheck disable=SC2086 + docker compose up -d $SERVICES + + echo "" + echo "Monitoring stack started in local-Solr mode." + echo "Prometheus is scraping your local Solr via host.docker.internal" + if $WITH_TRAFFICGEN; then + echo "Traffic generator is running against: $SOLR_URL" + fi + echo "" + echo " Dashboard: http://localhost:3000/d/solr-mixin-example" + echo " Grafana: http://localhost:3000 (view without login; admin/admin to edit)" + echo " Prometheus: http://localhost:9090" + echo " Alertmanager: http://localhost:9093" + echo "" + echo "Tip: edit prometheus/prometheus-local.yml to add further nodes or change the port." +else + # Bundled mode: activate the bundled-solr profile so solr1 and solr2 start + COMPOSE_PROFILES=bundled-solr docker compose up -d + + echo "" + echo "Full monitoring stack started." + echo " Solr 1: http://localhost:8983" + echo " Solr 2: http://localhost:8984" + echo " Dashboard: http://localhost:3000/d/solr-mixin-example" + echo " Grafana: http://localhost:3000 (view without login; admin/admin to edit)" + echo " Prometheus: http://localhost:9090" + echo " Alertmanager: http://localhost:9093" + echo "" + echo "Tip: Prometheus may show scrape errors for the first 30-60 seconds while Solr starts." +fi diff --git a/solr/monitoring/dev/trafficgen/Dockerfile b/solr/monitoring/dev/trafficgen/Dockerfile new file mode 100644 index 000000000000..5fa4c2371708 --- /dev/null +++ b/solr/monitoring/dev/trafficgen/Dockerfile @@ -0,0 +1,29 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +FROM python:3.12-slim + +WORKDIR /app + +COPY requirements.txt . +RUN pip install --no-cache-dir -r requirements.txt + +COPY trafficgen.py . +COPY trafficgen.yml . + +RUN useradd -r -u 1000 -g root trafficgen +USER 1000 + +ENTRYPOINT ["python3", "trafficgen.py"] diff --git a/solr/monitoring/dev/trafficgen/requirements.txt b/solr/monitoring/dev/trafficgen/requirements.txt new file mode 100644 index 000000000000..f84ec71246fe --- /dev/null +++ b/solr/monitoring/dev/trafficgen/requirements.txt @@ -0,0 +1,3 @@ +pysolr>=3.10.0 +requests>=2.31.0 +PyYAML>=6.0 diff --git a/solr/monitoring/dev/trafficgen/trafficgen.py b/solr/monitoring/dev/trafficgen/trafficgen.py new file mode 100644 index 000000000000..292092a789d5 --- /dev/null +++ b/solr/monitoring/dev/trafficgen/trafficgen.py @@ -0,0 +1,298 @@ +#!/usr/bin/env python3 +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +"""trafficgen.py - Traffic generator for Apache Solr + +Generates continuous index and search traffic against a Solr endpoint. +Configuration is read from a YAML file (default: trafficgen.yml in same directory). +If the config has no 'words' key, a vocabulary is generated in RAM at startup. + +Environment variable overrides: + STOP_AFTER= Stop after this many seconds (default: run forever) + CONFIG_FILE= Path to config file (default: trafficgen.yml) + INDEX_SLEEP_SECS= Override indexing.sleepSecs + SEARCH_SLEEP_SECS= Override searching.sleepSecs + DELETIONS_SLEEP_SECS= Override deletions.sleepSecs + QUIET=true Suppress per-operation log lines + SOLR_URL= Override solrUrl +""" + +import argparse +import logging +import os +import random +import sys +import threading +import time +import uuid + +import pysolr +import requests +import yaml + + +def generate_words(count=300): + """Generate a pronounceable in-RAM vocabulary of `count` unique words.""" + consonants = "bcdfghjklmnprstvwz" + vowels = "aeiou" + words = set() + while len(words) < count: + length = random.randint(4, 9) + word = "" + for i in range(length): + word += random.choice(consonants if i % 2 == 0 else vowels) + words.add(word) + return sorted(words) + + +def load_config(config_file): + with open(config_file, "r") as f: + return yaml.safe_load(f) + + +def apply_env_overrides(config): + if os.environ.get("SOLR_URL"): + config["solrUrl"] = os.environ["SOLR_URL"] + if os.environ.get("INDEX_SLEEP_SECS"): + config["indexing"]["sleepSecs"] = float(os.environ["INDEX_SLEEP_SECS"]) + # Support both correct and typo variant from spec + search_secs = os.environ.get("SEARCH_SLEEP_SECS") or os.environ.get("SEARCH_SEEP_SECS") + if search_secs: + config["searching"]["sleepSecs"] = float(search_secs) + if os.environ.get("DELETIONS_SLEEP_SECS"): + config["deletions"]["sleepSecs"] = float(os.environ["DELETIONS_SLEEP_SECS"]) + return config + + +def ensure_collection_exists(solr_url, collection_config): + name = collection_config["name"] + num_shards = collection_config.get("numShards", 1) + replication_factor = collection_config.get("replicationFactor", 1) + config_set = collection_config.get("configSet", "_default") + + admin_url = f"{solr_url.rstrip('/')}/admin/collections" + + try: + resp = requests.get(admin_url, params={"action": "LIST", "wt": "json"}, timeout=10) + resp.raise_for_status() + collections = resp.json().get("collections", []) + except Exception as e: + logging.warning(f"CREATE: Could not list collections (non-cloud Solr?): {e}") + return + + if name in collections: + logging.info(f"CREATE: Collection '{name}' already exists") + return + + logging.info( + f"CREATE: Creating collection '{name}' with numShards={num_shards}, " + f"replicationFactor={replication_factor}, configSet='{config_set}'" + ) + try: + create_resp = requests.get( + admin_url, + params={ + "action": "CREATE", + "name": name, + "numShards": num_shards, + "replicationFactor": replication_factor, + "collection.configName": config_set, + "wt": "json", + }, + timeout=30, + ) + create_resp.raise_for_status() + result = create_resp.json() + status = result.get("responseHeader", {}).get("status", -1) + if status != 0: + raise RuntimeError(f"Unexpected response status {status}: {result}") + logging.info(f"CREATE: Collection '{name}' created successfully") + except Exception as e: + logging.error(f"CREATE: Failed to create collection '{name}': {e}") + sys.exit(1) + + +def build_document(words, freetext_field, facet_field, num_words): + return { + "id": str(uuid.uuid4()), + freetext_field: " ".join(random.choices(words, k=num_words)), + facet_field: random.choice(words), + } + + +def run_indexing(solr, config, words, stop_event, quiet): + batch_size = config["indexing"]["batchSize"] + sleep_secs = config["indexing"]["sleepSecs"] + freetext_field = config["indexing"]["freetextFieldName"] + facet_field = config["indexing"]["facetFieldName"] + num_words = config["indexing"]["numWords"] + + while not stop_event.is_set(): + docs = [build_document(words, freetext_field, facet_field, num_words) for _ in range(batch_size)] + try: + t0 = time.monotonic() + solr.add(docs) + ms = int((time.monotonic() - t0) * 1000) + if not quiet: + logging.info(f"INDEX: batch={batch_size} field={freetext_field} time={ms}ms") + except Exception as e: + logging.error(f"INDEX: {e}") + stop_event.wait(sleep_secs) + + +def run_deletions(solr, config, words, stop_event, quiet): + sleep_secs = config["deletions"]["sleepSecs"] + freetext_field = config["indexing"]["freetextFieldName"] + + while not stop_event.is_set(): + word = random.choice(words) + q = f"{freetext_field}:{word}" + try: + t0 = time.monotonic() + solr.delete(q=q) + ms = int((time.monotonic() - t0) * 1000) + if not quiet: + logging.info(f"DELETE: q='{q}' time={ms}ms") + except Exception as e: + logging.error(f"DELETE: {e}") + stop_event.wait(sleep_secs) + + +def run_searching(solr, config, words, stop_event, quiet): + sleep_secs = config["searching"]["sleepSecs"] + freetext_field = config["searching"]["freetextFieldName"] + facet_field = config["searching"]["facetFieldName"] + num_words = config["searching"]["numWords"] + + while not stop_event.is_set(): + query_words = random.choices(words, k=num_words) + if num_words == 1: + q = f"{freetext_field}:{query_words[0]}" + else: + q = f"{freetext_field}:({' '.join(query_words)})" + try: + t0 = time.monotonic() + results = solr.search( + q, + rows=10, + facet="true", + **{"facet.field": facet_field}, + ) + ms = int((time.monotonic() - t0) * 1000) + if not quiet: + logging.info(f"SEARCH: q='{q}' hits={results.hits} time={ms}ms") + except Exception as e: + logging.error(f"SEARCH: {e}") + stop_event.wait(sleep_secs) + + +def main(): + parser = argparse.ArgumentParser( + description="Traffic generator for Apache Solr - continuously indexes and searches documents.", + formatter_class=argparse.RawDescriptionHelpFormatter, + epilog=__doc__, + ) + parser.parse_args() + + logging.basicConfig( + level=logging.INFO, + format="%(asctime)s %(levelname)s %(message)s", + datefmt="%Y-%m-%dT%H:%M:%S", + stream=sys.stdout, + ) + logging.getLogger("pysolr").setLevel(logging.WARNING) + + config_file = os.environ.get("CONFIG_FILE") + if not config_file: + script_dir = os.path.dirname(os.path.abspath(__file__)) + config_file = os.path.join(script_dir, "trafficgen.yml") + + logging.info(f"Loading config from {config_file}") + config = load_config(config_file) + config = apply_env_overrides(config) + + if config.get("words"): + words = config["words"] + logging.info(f"Vocabulary: {len(words)} words loaded from config") + else: + words = generate_words(300) + logging.info(f"Vocabulary: {len(words)} words auto-generated in RAM") + + quiet = os.environ.get("QUIET", "").lower() == "true" + stop_after_secs = float(os.environ["STOP_AFTER"]) if os.environ.get("STOP_AFTER") else None + + solr_url = config["solrUrl"].rstrip("/") + collection_name = config["collection"]["name"] + + logging.info(f"Solr URL: {solr_url}") + logging.info(f"Collection: {collection_name}") + + ensure_collection_exists(solr_url, config["collection"]) + + solr = pysolr.Solr( + f"{solr_url}/{collection_name}", + always_commit=True, + timeout=10, + ) + + stop_event = threading.Event() + + index_thread = threading.Thread( + target=run_indexing, + args=(solr, config, words, stop_event, quiet), + daemon=True, + name="indexer", + ) + search_thread = threading.Thread( + target=run_searching, + args=(solr, config, words, stop_event, quiet), + daemon=True, + name="searcher", + ) + deletion_thread = threading.Thread( + target=run_deletions, + args=(solr, config, words, stop_event, quiet), + daemon=True, + name="deletions", + ) + + index_thread.start() + search_thread.start() + deletion_thread.start() + + if stop_after_secs: + logging.info(f"Traffic generation started, will stop after {stop_after_secs}s") + else: + logging.info("Traffic generation started, running forever (set STOP_AFTER to limit duration)") + + try: + if stop_after_secs: + time.sleep(stop_after_secs) + logging.info("STOP_AFTER reached, shutting down") + else: + while True: + time.sleep(1) + except KeyboardInterrupt: + logging.info("Interrupted, shutting down") + finally: + stop_event.set() + index_thread.join(timeout=10) + search_thread.join(timeout=10) + deletion_thread.join(timeout=10) + logging.info("Traffic generation stopped") + + +if __name__ == "__main__": + main() diff --git a/solr/monitoring/dev/trafficgen/trafficgen.yml b/solr/monitoring/dev/trafficgen/trafficgen.yml new file mode 100644 index 000000000000..58610b827c43 --- /dev/null +++ b/solr/monitoring/dev/trafficgen/trafficgen.yml @@ -0,0 +1,45 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +# trafficgen.yml — configuration for the dev docker-compose stack. +# +# Tuned for SolrCloud (numShards: 2) with faster indexing so the +# Grafana dashboard panels show activity immediately. +# Solr URL is overridden at runtime via SOLR_URL env var in docker-compose.yml. +# Word list is auto-generated in RAM by trafficgen.py (no 'words' key needed). + +solrUrl: http://localhost:8983/solr + +collection: + name: test + numShards: 2 + replicationFactor: 2 + configSet: _default + +indexing: + batchSize: 10 + sleepSecs: 5 + freetextFieldName: text_t + numWords: 20 + facetFieldName: keywords_s + +searching: + sleepSecs: 1 + freetextFieldName: text_t + numWords: 1 + facetFieldName: keywords_s + +deletions: + sleepSecs: 30 diff --git a/solr/monitoring/grafana-solr-dashboard.json b/solr/monitoring/grafana-solr-dashboard.json new file mode 100644 index 000000000000..e35648dc22ed --- /dev/null +++ b/solr/monitoring/grafana-solr-dashboard.json @@ -0,0 +1,149 @@ +{ + "__license": "Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to You under the Apache License, Version 2.0 (the \"License\"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 — Generated by mixin/dashboards/dashboards.libsonnet. To regenerate: cd mixin && make dashboards", + "description": "Placeholder dashboard that exercises the monitoring mixin build. It is not the Solr dashboard — that is still being designed, see SOLR-18534.", + "editable": true, + "panels": [ + { + "datasource": { + "type": "datasource", + "uid": "-- Mixed --" + }, + "description": "Requests per second per node, across all handlers and cores.", + "fieldConfig": { + "defaults": { + "unit": "reqps" + } + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 0, + "y": 0 + }, + "id": 1, + "options": { + "legend": { + "displayMode": "list" + }, + "tooltip": { + "mode": "multi" + } + }, + "pluginVersion": "v11.4.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "$datasource" + }, + "expr": "sum by (instance)(rate(solr_core_requests_total{job=\"solr\"}[$interval]))", + "legendFormat": "{{instance}}" + } + ], + "title": "Request Rate", + "type": "timeseries" + }, + { + "datasource": { + "type": "datasource", + "uid": "-- Mixed --" + }, + "description": "JVM heap in use per node, summed over the heap memory pools. The inner max collapses pools reported by more than one OTel instrumentation scope, which differ by Solr version.", + "fieldConfig": { + "defaults": { + "unit": "bytes" + } + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 12, + "y": 0 + }, + "id": 2, + "options": { + "legend": { + "displayMode": "list" + }, + "tooltip": { + "mode": "multi" + } + }, + "pluginVersion": "v11.4.0", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "$datasource" + }, + "expr": "sum by (instance)(max by (instance, jvm_memory_pool_name)(jvm_memory_used_bytes{job=\"solr\",jvm_memory_type=\"heap\"}))", + "legendFormat": "{{instance}}" + } + ], + "title": "Heap Used", + "type": "timeseries" + } + ], + "refresh": "auto", + "schemaVersion": 36, + "tags": [ + "solr", + "prometheus", + "example" + ], + "templating": { + "list": [ + { + "label": "Data Source", + "name": "datasource", + "query": "prometheus", + "type": "datasource" + }, + { + "current": { + "selected": false, + "text": "5m", + "value": "5m" + }, + "label": "Interval", + "name": "interval", + "options": [ + { + "selected": true, + "text": "1m", + "value": "1m" + }, + { + "selected": false, + "text": "5m", + "value": "5m" + }, + { + "selected": false, + "text": "10m", + "value": "10m" + }, + { + "selected": false, + "text": "30m", + "value": "30m" + }, + { + "selected": false, + "text": "1h", + "value": "1h" + } + ], + "query": "1m,5m,10m,30m,1h", + "type": "interval" + } + ] + }, + "time": { + "from": "now-15m", + "to": "now" + }, + "timezone": "browser", + "title": "Solr Mixin Example", + "uid": "solr-mixin-example" +} diff --git a/solr/monitoring/mixin/.gitignore b/solr/monitoring/mixin/.gitignore new file mode 100644 index 000000000000..8cd29773f013 --- /dev/null +++ b/solr/monitoring/mixin/.gitignore @@ -0,0 +1,2 @@ +vendor/ +out/ diff --git a/solr/monitoring/mixin/Dockerfile b/solr/monitoring/mixin/Dockerfile new file mode 100644 index 000000000000..6f01dd0a7bea --- /dev/null +++ b/solr/monitoring/mixin/Dockerfile @@ -0,0 +1,55 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# ============================================================================ +# Build image for the Solr monitoring mixin. +# Contains: jsonnet, jsonnetfmt, jb (jsonnet-bundler), gojsontoyaml, promtool. +# +# Not intended for production use — developer tooling only. +# See make.sh for usage. +# ============================================================================ + +# --- Stage 1: compile Go-based tools ----------------------------------------- +FROM golang:1-bookworm AS go-tools + +ENV CGO_ENABLED=0 + +RUN go install github.com/google/go-jsonnet/cmd/jsonnet@v0.20.0 \ + && go install github.com/google/go-jsonnet/cmd/jsonnetfmt@v0.20.0 \ + && go install github.com/jsonnet-bundler/jsonnet-bundler/cmd/jb@v0.5.1 \ + && go install github.com/brancz/gojsontoyaml@v0.1.0 + +# --- Stage 2: extract promtool from the official Prometheus image ------------- +FROM prom/prometheus:v2.52.0 AS prometheus + +# --- Stage 3: lean runtime image --------------------------------------------- +FROM ubuntu:24.04 + +RUN apt-get update \ + && apt-get install -y --no-install-recommends make ca-certificates \ + && rm -rf /var/lib/apt/lists/* + +COPY --from=go-tools /go/bin/jsonnet /usr/local/bin/ +COPY --from=go-tools /go/bin/jsonnetfmt /usr/local/bin/ +COPY --from=go-tools /go/bin/jb /usr/local/bin/ +COPY --from=go-tools /go/bin/gojsontoyaml /usr/local/bin/ +COPY --from=prometheus /bin/promtool /usr/local/bin/ + +# The mixin source tree is expected to be mounted at /work/mixin +# (i.e., the monitoring/ directory is mounted as /work). +WORKDIR /work/mixin + +ENTRYPOINT ["make"] +CMD ["all"] diff --git a/solr/monitoring/mixin/Makefile b/solr/monitoring/mixin/Makefile new file mode 100644 index 000000000000..a764b38eaa7a --- /dev/null +++ b/solr/monitoring/mixin/Makefile @@ -0,0 +1,86 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +JSONNET = jsonnet +JSONNETFMT = jsonnetfmt +JB = jb +GOJSONTOYAML = gojsontoyaml +PROMTOOL = promtool + +JSONNET_FLAGS = -J vendor +FMT_FLAGS = -n 2 --max-blank-lines 2 --string-style s --comment-style s + +DASHBOARD_OUT = ../grafana-solr-dashboard.json +ALERTS_OUT = ../prometheus-solr-alerts.yml + +# Generated artifacts are written to a temp file and moved into place only on +# success, so a missing tool or a jsonnet error can never leave a committed +# artifact truncated. pipefail makes a failure anywhere in a pipe fatal. +SHELL := /bin/bash +.SHELLFLAGS := -eu -o pipefail -c + +# Abort before touching any output if a required tool is missing. +require_tool = @command -v $(1) >/dev/null 2>&1 || { \ + echo "ERROR: required tool '$(1)' not found in PATH." >&2; \ + echo " Install the local toolchain (see ../README.md, 'Mixin Build')," >&2; \ + echo " or run ./make.sh $@ to build inside Docker instead." >&2; \ + exit 1; } + +require_vendor = @test -d vendor || { \ + echo "ERROR: vendor/ is missing — run 'make install' (or ./make.sh install) first." >&2; \ + exit 1; } + +.PHONY: all dashboards alerts install check fmt clean + +all: dashboards alerts + +install: + $(call require_tool,$(JB)) + $(JB) install + @echo "→ Installed jsonnet dependencies into vendor/" + +dashboards: + $(call require_tool,$(JSONNET)) + $(require_vendor) + $(JSONNET) $(JSONNET_FLAGS) \ + -e '(import "mixin.libsonnet").grafanaDashboards["grafana-solr-dashboard.json"]' \ + > $(DASHBOARD_OUT).tmp || { rm -f $(DASHBOARD_OUT).tmp; exit 1; } + mv $(DASHBOARD_OUT).tmp $(DASHBOARD_OUT) + @echo "→ Generated $(DASHBOARD_OUT)" + +alerts: + $(call require_tool,$(JSONNET)) + $(call require_tool,$(GOJSONTOYAML)) + $(require_vendor) + cat license-header.yml > $(ALERTS_OUT).tmp + $(JSONNET) $(JSONNET_FLAGS) \ + -e '(import "mixin.libsonnet").prometheusAlerts' \ + | $(GOJSONTOYAML) >> $(ALERTS_OUT).tmp || { rm -f $(ALERTS_OUT).tmp; exit 1; } + mv $(ALERTS_OUT).tmp $(ALERTS_OUT) + @echo "→ Generated $(ALERTS_OUT)" + +check: + $(call require_tool,$(PROMTOOL)) + $(PROMTOOL) check rules $(ALERTS_OUT) + @echo "→ Alert rules validated OK" + +fmt: + $(call require_tool,$(JSONNETFMT)) + find . -path ./vendor -prune -o -name '*.libsonnet' -exec $(JSONNETFMT) $(FMT_FLAGS) -i {} \; + @echo "→ Formatted all .libsonnet files (vendor/ excluded)" + +clean: + rm -f $(DASHBOARD_OUT) $(ALERTS_OUT) $(DASHBOARD_OUT).tmp $(ALERTS_OUT).tmp + @echo "→ Removed generated artifacts" diff --git a/solr/monitoring/mixin/alerts/alerts.libsonnet b/solr/monitoring/mixin/alerts/alerts.libsonnet new file mode 100644 index 000000000000..93dad7f5fa20 --- /dev/null +++ b/solr/monitoring/mixin/alerts/alerts.libsonnet @@ -0,0 +1,51 @@ +// Licensed to the Apache Software Foundation (ASF) under one or more +// contributor license agreements. See the NOTICE file distributed with +// this work for additional information regarding copyright ownership. +// The ASF licenses this file to You under the Apache License, Version 2.0 +// (the "License"); you may not use this file except in compliance with +// the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +// alerts.libsonnet — placeholder alert rule. +// +// One rule, enough to show that the mixin build produces a file promtool +// accepts and Prometheus loads. The real rule set replaces this wholesale; +// see SOLR-18535. +// +// The rules are built from the configuration of the merged mixin root +// ($._config), not from a direct import, so that callers can override settings +// without editing this tree. + +local alertsFor(cfg) = { + groups: [ + { + name: 'SolrExampleAlerts', + rules: [ + { + alert: 'SolrNodeDown', + expr: 'up{%s} == 0' % cfg.solrSelector, + 'for': '2m', + labels: { severity: 'warning' }, + annotations: { + summary: 'Solr node {{ $labels.instance }} is not responding to scrapes', + description: ||| + Prometheus has failed to scrape {{ $labels.instance }} for 2 minutes. + Either the node is down or its metrics endpoint is unreachable. + |||, + }, + }, + ], + }, + ], +}; + +{ + prometheusAlerts+:: alertsFor($._config), +} diff --git a/solr/monitoring/mixin/config.libsonnet b/solr/monitoring/mixin/config.libsonnet new file mode 100644 index 000000000000..342948d14ddd --- /dev/null +++ b/solr/monitoring/mixin/config.libsonnet @@ -0,0 +1,35 @@ +// Licensed to the Apache Software Foundation (ASF) under one or more +// contributor license agreements. See the NOTICE file distributed with +// this work for additional information regarding copyright ownership. +// The ASF licenses this file to You under the Apache License, Version 2.0 +// (the "License"); you may not use this file except in compliance with +// the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +// config.libsonnet — settings you can override without editing the dashboard +// or alert sources. +// +// Every key here is consumed by dashboards/ or alerts/; nothing is declared +// ahead of its use. To override, import the mixin and patch _config: +// +// (import 'mixin.libsonnet') + { _config+:: { solrSelector: 'job="my-solr"' } } + +{ + _config:: { + // Label selector that identifies Solr targets in your Prometheus. Applied to + // every dashboard query and alert expression, so JVM metrics scraped from + // other Java services are not mistaken for Solr's. + solrSelector: 'job="solr"', + + // Window for rate() queries, and the preselected value of the dashboard's + // Interval dropdown. Raise it for sparse scrape intervals. + rateInterval: '5m', + }, +} diff --git a/solr/monitoring/mixin/dashboards/dashboards.libsonnet b/solr/monitoring/mixin/dashboards/dashboards.libsonnet new file mode 100644 index 000000000000..d214c72dbd77 --- /dev/null +++ b/solr/monitoring/mixin/dashboards/dashboards.libsonnet @@ -0,0 +1,102 @@ +// Licensed to the Apache Software Foundation (ASF) under one or more +// contributor license agreements. See the NOTICE file distributed with +// this work for additional information regarding copyright ownership. +// The ASF licenses this file to You under the Apache License, Version 2.0 +// (the "License"); you may not use this file except in compliance with +// the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +// dashboards.libsonnet — placeholder dashboard. +// +// Two panels, enough to show that the mixin build produces a dashboard Grafana +// can load and that config.libsonnet overrides reach the generated queries. +// The real dashboard replaces this wholesale; see SOLR-18534. +// +// The dashboard is built from the configuration of the merged mixin root +// ($._config), not from a direct import, so that callers can override settings +// without editing this tree. + +local g = import 'github.com/grafana/grafonnet/gen/grafonnet-latest/main.libsonnet'; + +local d = g.dashboard; +local p = g.panel; +local q = g.query.prometheus; +local v = g.dashboard.variable; + +local licenseComment = + 'Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. ' + + 'See the NOTICE file distributed with this work for additional information regarding copyright ownership. ' + + 'The ASF licenses this file to You under the Apache License, Version 2.0 (the "License"); ' + + 'you may not use this file except in compliance with the License. ' + + 'You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 ' + + '— Generated by mixin/dashboards/dashboards.libsonnet. To regenerate: cd mixin && make dashboards'; + +local prom(expr, legend='{{instance}}') = + q.new('$datasource', expr) + + q.withLegendFormat(legend); + +local ts(title, exprs, unit='short', desc='') = + p.timeSeries.new(title) + + p.timeSeries.queryOptions.withTargets(exprs) + + p.timeSeries.standardOptions.withUnit(unit) + + p.timeSeries.panelOptions.withDescription(desc) + + p.timeSeries.options.legend.withDisplayMode('list') + + p.timeSeries.options.tooltip.withMode('multi'); + +local dashboardFor(cfg) = + local datasourceVar = + v.datasource.new('datasource', 'prometheus') + + v.datasource.generalOptions.withLabel('Data Source'); + + local intervalVar = + v.interval.new('interval', ['1m', '5m', '10m', '30m', '1h']) + + v.interval.generalOptions.withCurrent(cfg.rateInterval) + + v.interval.generalOptions.withLabel('Interval'); + + local panels = [ + ts( + 'Request Rate', + [prom('sum by (instance)(rate(solr_core_requests_total{%s}[$interval]))' % cfg.solrSelector)], + unit='reqps', + desc='Requests per second per node, across all handlers and cores.' + ) + { gridPos: { x: 0, y: 0, w: 12, h: 8 } }, + + ts( + 'Heap Used', + [prom( + 'sum by (instance)(max by (instance, jvm_memory_pool_name)(jvm_memory_used_bytes{%s,jvm_memory_type="heap"}))' % cfg.solrSelector + )], + unit='bytes', + desc='JVM heap in use per node, summed over the heap memory pools. The inner max collapses pools reported by more than one OTel instrumentation scope, which differ by Solr version.' + ) + { gridPos: { x: 12, y: 0, w: 12, h: 8 } }, + ]; + + { __license: licenseComment } + + d.new('Solr Mixin Example') + + d.withUid('solr-mixin-example') + + d.withDescription( + 'Placeholder dashboard that exercises the monitoring mixin build. ' + + 'It is not the Solr dashboard — that is still being designed, see SOLR-18534.' + ) + + d.withTags(['solr', 'prometheus', 'example']) + + d.withRefresh('auto') + + d.withTimezone('browser') + + d.withEditable(true) + + d.withSchemaVersion(36) + + d.time.withFrom('now-15m') + + d.time.withTo('now') + + d.withVariables([datasourceVar, intervalVar]) + + d.withPanels(panels); + +{ + grafanaDashboards+:: { + 'grafana-solr-dashboard.json': dashboardFor($._config), + }, +} diff --git a/solr/monitoring/mixin/jsonnetfile.json b/solr/monitoring/mixin/jsonnetfile.json new file mode 100644 index 000000000000..2d56d912452c --- /dev/null +++ b/solr/monitoring/mixin/jsonnetfile.json @@ -0,0 +1,15 @@ +{ + "version": 1, + "dependencies": [ + { + "source": { + "git": { + "remote": "https://github.com/grafana/grafonnet.git", + "subdir": "gen/grafonnet-latest" + } + }, + "version": "main" + } + ], + "legacyImports": false +} diff --git a/solr/monitoring/mixin/jsonnetfile.lock.json b/solr/monitoring/mixin/jsonnetfile.lock.json new file mode 100644 index 000000000000..190410bcefdf --- /dev/null +++ b/solr/monitoring/mixin/jsonnetfile.lock.json @@ -0,0 +1,46 @@ +{ + "version": 1, + "dependencies": [ + { + "source": { + "git": { + "remote": "https://github.com/grafana/grafonnet.git", + "subdir": "gen/grafonnet-latest" + } + }, + "version": "7380c9c64fb973f34c3ec46265621a2b0dee0058", + "sum": "V9vAj21qJOc2DlMPDgB1eEjSQU4A+sAA4AXuJ6bd4xc=" + }, + { + "source": { + "git": { + "remote": "https://github.com/grafana/grafonnet.git", + "subdir": "gen/grafonnet-v11.4.0" + } + }, + "version": "7380c9c64fb973f34c3ec46265621a2b0dee0058", + "sum": "aVAX09paQYNOoCSKVpuk1exVIyBoMt/C50QJI+Q/3nA=" + }, + { + "source": { + "git": { + "remote": "https://github.com/jsonnet-libs/docsonnet.git", + "subdir": "doc-util" + } + }, + "version": "6ac6c69685b8c29c54515448eaca583da2d88150", + "sum": "BrAL/k23jq+xy9oA7TWIhUx07dsA/QLm3g7ktCwe//U=" + }, + { + "source": { + "git": { + "remote": "https://github.com/jsonnet-libs/xtd.git", + "subdir": "" + } + }, + "version": "4d7f8cb24d613430799f9d56809cc6964f35cea9", + "sum": "hOrwkOx34tOXqoDVnwuI/Uf/dr9HFFSPWpDPOvnEGrk=" + } + ], + "legacyImports": false +} diff --git a/solr/monitoring/mixin/license-header.yml b/solr/monitoring/mixin/license-header.yml new file mode 100644 index 000000000000..c8d2335f2825 --- /dev/null +++ b/solr/monitoring/mixin/license-header.yml @@ -0,0 +1,17 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +# This file is generated by mixin/alerts/alerts.libsonnet — do not edit directly. +# To regenerate: cd mixin && make alerts diff --git a/solr/monitoring/mixin/make.sh b/solr/monitoring/mixin/make.sh new file mode 100755 index 000000000000..7392e564da74 --- /dev/null +++ b/solr/monitoring/mixin/make.sh @@ -0,0 +1,67 @@ +#!/usr/bin/env bash +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# make.sh — Docker-based wrapper around the mixin Makefile. +# +# Usage: +# ./make.sh # equivalent to: make all +# ./make.sh install all # equivalent to: make install all +# ./make.sh check # equivalent to: make check +# ./make.sh --rebuild # force-rebuild the Docker image, then make all +# +# The Docker image (solr-mixin-make:latest) is built automatically on first +# use and reused on subsequent runs. Pass --rebuild to force a fresh build +# (e.g. after updating the Dockerfile or tool versions). + +set -euo pipefail + +IMAGE="solr-mixin-make:latest" +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +MONITORING_DIR="$(dirname "$SCRIPT_DIR")" + +# --- parse flags ------------------------------------------------------------- +REBUILD=false +MAKE_ARGS=() +for arg in "$@"; do + case "$arg" in + --rebuild) REBUILD=true ;; + *) MAKE_ARGS+=("$arg") ;; + esac +done + +# --- build image if missing or explicitly requested -------------------------- +if ! docker image inspect "$IMAGE" >/dev/null 2>&1 || [ "$REBUILD" = true ]; then + echo "[make.sh] Building Docker image $IMAGE ..." + docker build -t "$IMAGE" "$SCRIPT_DIR" +fi + +# --- run make inside the container ------------------------------------------- +# Mount the monitoring/ directory as /work so that the Makefile's relative +# paths (../grafana-solr-dashboard.json, ../prometheus-solr-alerts.yml) resolve +# correctly to the host filesystem. +TTY_FLAG="" +[ -t 0 ] && TTY_FLAG="-t" + +# Run as the invoking user, so that the regenerated artifacts and vendor/ end up +# owned by you rather than by root. Without this, a build on native Linux leaves +# files you cannot edit or clean without sudo. The tools are static binaries and +# do not need the user to exist in the image's passwd database, but they do need +# a writable HOME. +docker run --rm -i $TTY_FLAG \ + --user "$(id -u):$(id -g)" \ + -e HOME=/tmp \ + -v "$MONITORING_DIR:/work" \ + "$IMAGE" "${MAKE_ARGS[@]:-all}" diff --git a/solr/monitoring/mixin/mixin.libsonnet b/solr/monitoring/mixin/mixin.libsonnet new file mode 100644 index 000000000000..13902679e7de --- /dev/null +++ b/solr/monitoring/mixin/mixin.libsonnet @@ -0,0 +1,36 @@ +// Licensed to the Apache Software Foundation (ASF) under one or more +// contributor license agreements. See the NOTICE file distributed with +// this work for additional information regarding copyright ownership. +// The ASF licenses this file to You under the Apache License, Version 2.0 +// (the "License"); you may not use this file except in compliance with +// the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +// mixin.libsonnet — root entry point for the Solr monitoring mixin. +// +// Composes the configuration with the dashboards and alerts, so that both read +// their settings from the merged root. That late binding is what lets a caller +// override settings without editing this tree: +// +// (import 'mixin.libsonnet') + { _config+:: { solrSelector: 'job="my-solr"' } } +// +// Exposes: +// grafanaDashboards:: — map of filename -> Grafana dashboard JSON object +// prometheusAlerts:: — Prometheus alert groups object +// +// Usage: +// jsonnet -J vendor -e '(import "mixin.libsonnet").grafanaDashboards["grafana-solr-dashboard.json"]' \ +// > ../grafana-solr-dashboard.json +// jsonnet -J vendor -e '(import "mixin.libsonnet").prometheusAlerts' \ +// | gojsontoyaml > ../prometheus-solr-alerts.yml + +(import 'config.libsonnet') ++ (import 'dashboards/dashboards.libsonnet') ++ (import 'alerts/alerts.libsonnet') diff --git a/solr/monitoring/prometheus-solr-alerts.yml b/solr/monitoring/prometheus-solr-alerts.yml new file mode 100644 index 000000000000..d1cefdcba1ad --- /dev/null +++ b/solr/monitoring/prometheus-solr-alerts.yml @@ -0,0 +1,30 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +# This file is generated by mixin/alerts/alerts.libsonnet — do not edit directly. +# To regenerate: cd mixin && make alerts +groups: +- name: SolrExampleAlerts + rules: + - alert: SolrNodeDown + annotations: + description: | + Prometheus has failed to scrape {{ $labels.instance }} for 2 minutes. + Either the node is down or its metrics endpoint is unreachable. + summary: Solr node {{ $labels.instance }} is not responding to scrapes + expr: up{job="solr"} == 0 + for: 2m + labels: + severity: warning diff --git a/solr/solr-ref-guide/modules/deployment-guide/deployment-nav.adoc b/solr/solr-ref-guide/modules/deployment-guide/deployment-nav.adoc index b001d404ba33..8fd754306070 100644 --- a/solr/solr-ref-guide/modules/deployment-guide/deployment-nav.adoc +++ b/solr/solr-ref-guide/modules/deployment-guide/deployment-nav.adoc @@ -105,6 +105,7 @@ ** xref:configuring-logging.adoc[] ** xref:opentelemetry.adoc[] ** xref:metrics-reporting.adoc[] +** xref:monitoring-with-prometheus-and-grafana.adoc[] ** xref:distributed-tracing.adoc[] ** xref:performance-statistics-reference.adoc[] ** xref:jmx-with-solr.adoc[] diff --git a/solr/solr-ref-guide/modules/deployment-guide/images/monitoring-with-prometheus-and-grafana/grafana-solr-dashboard.png b/solr/solr-ref-guide/modules/deployment-guide/images/monitoring-with-prometheus-and-grafana/grafana-solr-dashboard.png index 69f21a4debbc..bc7b6ef2841a 100644 Binary files a/solr/solr-ref-guide/modules/deployment-guide/images/monitoring-with-prometheus-and-grafana/grafana-solr-dashboard.png and b/solr/solr-ref-guide/modules/deployment-guide/images/monitoring-with-prometheus-and-grafana/grafana-solr-dashboard.png differ diff --git a/solr/solr-ref-guide/modules/deployment-guide/images/monitoring-with-prometheus-and-grafana/prometheus-solr-ping.png b/solr/solr-ref-guide/modules/deployment-guide/images/monitoring-with-prometheus-and-grafana/prometheus-solr-ping.png deleted file mode 100644 index e4bd7f77cfea..000000000000 Binary files a/solr/solr-ref-guide/modules/deployment-guide/images/monitoring-with-prometheus-and-grafana/prometheus-solr-ping.png and /dev/null differ diff --git a/solr/solr-ref-guide/modules/deployment-guide/images/monitoring-with-prometheus-and-grafana/solr-exporter-diagram.png b/solr/solr-ref-guide/modules/deployment-guide/images/monitoring-with-prometheus-and-grafana/solr-exporter-diagram.png deleted file mode 100644 index 96efece464df..000000000000 Binary files a/solr/solr-ref-guide/modules/deployment-guide/images/monitoring-with-prometheus-and-grafana/solr-exporter-diagram.png and /dev/null differ diff --git a/solr/solr-ref-guide/modules/deployment-guide/images/monitoring-with-prometheus-and-grafana/solr-monitoring-diagram.png b/solr/solr-ref-guide/modules/deployment-guide/images/monitoring-with-prometheus-and-grafana/solr-monitoring-diagram.png new file mode 100644 index 000000000000..dce7b383b46c Binary files /dev/null and b/solr/solr-ref-guide/modules/deployment-guide/images/monitoring-with-prometheus-and-grafana/solr-monitoring-diagram.png differ diff --git a/solr/solr-ref-guide/modules/deployment-guide/images/monitoring-with-prometheus-and-grafana/solr-monitoring-diagram.svg b/solr/solr-ref-guide/modules/deployment-guide/images/monitoring-with-prometheus-and-grafana/solr-monitoring-diagram.svg new file mode 100644 index 000000000000..bd01251f8e9b --- /dev/null +++ b/solr/solr-ref-guide/modules/deployment-guide/images/monitoring-with-prometheus-and-grafana/solr-monitoring-diagram.svg @@ -0,0 +1,178 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + Solr 10.x Monitoring Architecture + + + + SolrCloud Cluster + + + + + S + Node 1 + :8983/api/metrics + OTLP http :4318 + + + + + S + Node 2 + :8983/api/metrics + OTLP http :4318 + + + + + OTel + OTel Collector + OTLP receiver + Prometheus exporter + OTLP exporter + + + + + P + Prometheus + scrape + store + PromQL alerts + + + + + G + Grafana + Dashboards + Explore / Alerts + + + + + A + Alertmanager + route + dedupe + silence rules + + + + Other Backends + OTLP-compatible backends + + + + email, chat, paging, webhooks + + + + + User / Ops Team + + + + + + + + + + + scrape /api/metrics + + + + + + + + + OTLP push + + + + Prom. exporter + + + + + + + + + + + + + + + + + + + Legend + + + + + Prometheus pull (scrape) + + + + + OTLP push + + diff --git a/solr/solr-ref-guide/modules/deployment-guide/pages/metrics-reporting.adoc b/solr/solr-ref-guide/modules/deployment-guide/pages/metrics-reporting.adoc index 880359491fd0..d47dd9ce6308 100644 --- a/solr/solr-ref-guide/modules/deployment-guide/pages/metrics-reporting.adoc +++ b/solr/solr-ref-guide/modules/deployment-guide/pages/metrics-reporting.adoc @@ -180,6 +180,8 @@ See https://prometheus.io/docs/concepts/data_model/[Prometheus Data Model] docum This endpoint can be used to pull/scrape metrics to a Prometheus server or any Prometheus-compatible backend directly from Solr. +TIP: For a worked Prometheus and Grafana setup, including the dashboard and alert rules Solr ships, see xref:monitoring-with-prometheus-and-grafana.adoc[]. + ==== Prometheus Setup The `prometheus-config.yml` file needs to be configured for a Prometheus server to scrape and collect metrics. A basic configuration for SolrCloud mode is as follows: diff --git a/solr/solr-ref-guide/modules/deployment-guide/pages/monitoring-with-prometheus-and-grafana.adoc b/solr/solr-ref-guide/modules/deployment-guide/pages/monitoring-with-prometheus-and-grafana.adoc new file mode 100644 index 000000000000..327d649e721a --- /dev/null +++ b/solr/solr-ref-guide/modules/deployment-guide/pages/monitoring-with-prometheus-and-grafana.adoc @@ -0,0 +1,164 @@ += Monitoring Solr with Prometheus and Grafana +// Licensed to the Apache Software Foundation (ASF) under one or more +// contributor license agreements. See the NOTICE file distributed with +// this work for additional information regarding copyright ownership. +// The ASF licenses this file to You under the Apache License, Version 2.0 +// (the "License"); you may not use this file except in compliance with +// the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +Solr ships two monitoring artifacts in `solr/monitoring/`: + +* `grafana-solr-dashboard.json` — a Grafana dashboard +* `prometheus-solr-alerts.yml` — a set of Prometheus alert rules + +Import these into your own Prometheus + Grafana installation. +Solr exposes all metrics natively, so no exporter process is required. +For the full list of available metrics see xref:metrics-reporting.adoc[]. + +[WARNING] +==== +Both artifacts are currently placeholders — a two-panel dashboard and a single alert — that exist to prove the build pipeline works. +The dashboard worth deploying arrives in https://issues.apache.org/jira/browse/SOLR-18534[SOLR-18534] and the alert rules in https://issues.apache.org/jira/browse/SOLR-18535[SOLR-18535]. +==== + +.Solr monitoring architecture — Prometheus pull path and OTLP push path +image::monitoring-with-prometheus-and-grafana/solr-monitoring-diagram.png[Solr monitoring architecture diagram,800] + +== The Grafana Dashboard + +Import the dashboard by going to *Dashboards → Import* in Grafana and uploading `grafana-solr-dashboard.json`. + +.The placeholder dashboard, running against two Solr nodes +image::monitoring-with-prometheus-and-grafana/grafana-solr-dashboard.png[Grafana dashboard screenshot,800] + +Dropdowns at the top of the dashboard scope what the panels show. +The placeholder has two: `datasource`, so one import can serve several Prometheus instances, and `interval`, which sets the window for rate calculations. + +[[metrics-collection]] +== Getting Metrics into Prometheus + +There are two ways to get Solr metrics into Prometheus. + +[[prometheus-scrape]] +=== Option 1: Prometheus Pull (recommended) + +Prometheus scrapes each Solr node directly — no extra process needed. +Add a scrape job to your `prometheus.yml`: + +[source,yaml] +---- +scrape_configs: + - job_name: solr # <1> + metrics_path: /solr/admin/metrics # <2> + scrape_interval: 15s + static_configs: + - targets: ['solr-node1:8983', 'solr-node2:8983'] +---- +<1> The dashboard and alert rules select on this job name. + If you use another, set `solrSelector` to match — see <>. +<2> Solr 10.1 and later also serve this at the shorter v2 alias `/api/metrics`. + +Scrape every node rather than one: each reports its own JVM and the cores it hosts. +If metrics collection is switched off for a node, both endpoints answer `HTTP 510` and Prometheus marks the target down. + +[[otlp-push]] +=== Option 2: OTLP Push via OpenTelemetry Collector + +When Solr's xref:opentelemetry.adoc[OpenTelemetry module] is enabled, Solr can push metrics (and traces) via OTLP to an https://opentelemetry.io/docs/collector/[OpenTelemetry Collector], which can forward them to Prometheus or to any other OTLP-compatible backend. + +NOTE: An example Collector configuration, and the labelling needed to make both paths produce identical series, will be documented here as part of https://issues.apache.org/jira/browse/SOLR-18536[SOLR-18536]. + +[[alert-rules]] +== Alert Rules + +Reference `prometheus-solr-alerts.yml` from your `prometheus.yml`: + +[source,yaml] +---- +rule_files: + - /path/to/prometheus-solr-alerts.yml +---- + +Validate the file before deploying it, and after any edit: + +[source,bash] +---- +promtool check rules prometheus-solr-alerts.yml +---- + +Rules carry a `severity` label for routing in Alertmanager. +One placeholder rule is included today: + +[cols="2,1,3",options="header"] +|=== +| Alert | Severity | Fires when +| `SolrNodeDown` | warning | A Solr target has failed to scrape for 2 minutes +|=== + +NOTE: The rules covering heap, GC, disk, latency, error rate and SolrCloud health arrive in https://issues.apache.org/jira/browse/SOLR-18535[SOLR-18535]. + +[[customizing]] +== Customizing the Artifacts + +Both artifacts are generated from a https://jsonnet.org/[jsonnet] mixin source in `solr/monitoring/mixin/`, following the conventions of the https://github.com/monitoring-mixins/docs[monitoring-mixins] community. +Edit the jsonnet, not the generated output. + +Tunable settings are centralized in `mixin/config.libsonnet`: + +[source,jsonnet] +---- +{ + _config:: { + solrSelector: 'job="solr"', // label selector identifying Solr targets + rateInterval: '5m', // window for rate() queries + }, +} +---- + +`solrSelector` is applied to every dashboard query and alert expression, so JVM metrics scraped from other Java services are not read as Solr's. +Either edit the file and regenerate, or leave this tree untouched and patch the configuration from your own jsonnet: + +[source,jsonnet] +---- +(import 'mixin.libsonnet') + { _config+:: { solrSelector: 'job="my-solr"' } } +---- + +Regenerating requires `jsonnet`, `jsonnet-bundler` and `gojsontoyaml`, or the included Docker wrapper that supplies all three: + +[source,bash] +---- +cd solr/monitoring/mixin +./make.sh install # fetch jsonnet dependencies +./make.sh all # regenerate both artifacts +---- + +See `solr/monitoring/README.md` for the remaining build targets and the local toolchain alternative. + +[[dev-stack]] +== Trying It Locally + +The Solr source tree carries a docker-compose stack that runs Solr, Prometheus, Grafana and Alertmanager together with the generated artifacts already wired in. +It exists to make changes to those artifacts easy to see, and is contributor tooling rather than a deployment to model yours on — it is not part of the Solr release. + +[source,bash] +---- +cd solr/monitoring/dev +./stack.sh +---- + +Grafana is then on `http://localhost:3000`, opening directly on the provisioned dashboard, with Prometheus on `:9090` and Alertmanager on `:9093`. +`./stack.sh --local-solr` skips the bundled Solr nodes and scrapes a Solr you are already running. + +[[distributed-tracing]] +== Metrics and Tracing Together + +The same xref:opentelemetry.adoc[OpenTelemetry] instrumentation that produces these metrics also emits xref:distributed-tracing.adoc[distributed traces], so a latency spike on the dashboard can be followed into the traces of the requests that caused it. +Both signals are exported via OTLP and can flow through the same Collector pipeline to a unified observability backend, or independently to specialized ones.