🌱 Add bounded self-metrics for GPU utilization worker

[kubestellar-hive]

Fixes #23242

What

Adds bounded Prometheus metrics for the GPU utilization background worker (pkg/api/gpu_utilization_worker.go), which polls active GPU reservations, queries pods/GPU nodes per cluster, optionally scrapes DCGM, and sends threshold alerts — entirely outside the HTTP request path, so it had no signal on /metrics beyond slog error logs.

New metrics (pkg/api/metrics/metrics.go)

• console_gpu_util_scrape_cycles_total (counter) — completed poll cycles
• console_gpu_util_scrape_duration_seconds (histogram) — poll cycle duration
• console_gpu_util_reservation_collect_total{outcome} (counter) — fixed outcome set: success/pods_error/nodes_error/snapshot_error
• console_gpu_util_dcgm_scrape_errors_total (counter)
• console_gpu_util_alert_send_errors_total (counter)

No label is derived from reservation ID, cluster name, or any other unbounded value — the outcome label only ever takes one of the four fixed constants.

Other changes

• Two new panels added to the existing Grafana self-metrics dashboard (deploy/grafana/console-self-metrics-dashboard.json): poll-cycle rate/p95 duration, and collection outcomes/DCGM/alert error rates.
• New unit test (pkg/api/metrics/metrics_test.go) asserting all five series surface on /metrics with expected label values.

Backend status

No observability backend is confirmed for this repo (per prior audit #23055). This PR only adds in-process Prometheus client metrics exposed on the existing /metrics endpoint — no exporter or external data flow is introduced.

Testing

• go build ./... — passes
• go test ./pkg/api/... ./pkg/api/metrics/... — passes (including new TestGPUUtilRecordFunctions)
• gofmt -l — clean
• Dashboard JSON validated with json.load

— hive: agent=telemetry backend=copilot model=claude-sonnet-4-6

[kubestellar-hive]

Important

Held for human sign-off on the direction, not on the code.

This PR's only tracked rationale is #23242, which the hive filed itself — issue #23242 was filed by kubestellar-hive[bot] and no human has acknowledged it. An agent-filed issue does not, on its own, establish that anyone agreed to the direction (hivecommons/hive#5117).

The change may well be right; nothing here is a review of it. To release the hold, acknowledge the direction on that issue — comment on it, assign yourself, or add the approved-direction label — and remove the hold label here.

[kubestellar-prow]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign eeshaansa for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[netlify]

✅ Deploy Preview for kubestellarconsole canceled.

Name	Link
🔨 Latest commit	e49e6bd
🔍 Latest deploy log	https://app.netlify.com/projects/kubestellarconsole/deploys/6a9fc18a80bb1e0009f8243c

[github-actions]

👋 Hey @kubestellar-hive[bot] — thanks for opening this PR!

🤖 This project is developed exclusively using AI coding assistants.

Please do not attempt to code anything for this project manually.
All contributions should be authored using an AI coding tool such as:

• Claude Code (Opus 4.5 / 4.6) — recommended
• GitHub Copilot
• Cursor
• Other AI coding assistants

This ensures consistency in code style, architecture patterns, test coverage,
and commit quality across the entire codebase.

---

This is an automated message.

[kubestellar-prow]

PR needs rebase.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.