[Observability][P2] $pipeline selector returns different value-sets across dashboards (15/19/20)

[SatoryKono]

Finding (P2-3 — Grafana Dashboard Audit 2026-07-17)

The $pipeline template selector resolves to different value-sets depending on which universe metric each dashboard uses.

Evidence (matrix-selector.csv, live-resolved)

• bioetl-control-plane-v1: 15 pipelines via label_values(bioetl_control_plane_run_type_universe, pipeline).
• bioetl-silver-reject-explorer: 19 via label_values(bioetl_records_processed_total, pipeline).
• others (overview/runtime/dq/provider/workflow/alerts): 20 via bioetl_overview/runtime_pipeline_run_type_universe.

Affected

• All dashboards exposing a $pipeline selector.

Impact

The same operator control offers different option lists across the navigation bus, causing scope confusion on dashboard-to-dashboard handoff.

Recommendation

Standardize the $pipeline selector on one universe recording rule (or explicitly document why a given surface intentionally scopes differently).

Report: dashboard-audit-report.md (§2, P2-3).

[SatoryKono]

Current-state audit and refreshed plan (2026-07-19)

Verdict: the observation remains relevant, but this should be treated as a selector-contract/handoff issue, not an automatic requirement for one universal value set. Audited against main@400ed42.

Static current state:

• Alerts, DQ, Overview, Provider, and Workflow use bioetl_overview_pipeline_run_type_universe; Runtime uses bioetl_runtime_pipeline_run_type_universe.
• The recording rule defines bioetl_overview_pipeline_run_type_universe as an exact alias of the runtime universe. Those primary dashboards therefore have one common source contract even though two metric names appear in JSON.
• Control Plane intentionally uses bioetl_control_plane_run_type_universe; Explorer intentionally uses bioetl_records_processed_total. The canonical selector docs already describe these role-specific universes.
• The historical live counts (15/19/20) cannot be rechecked on the current host because Prometheus is unavailable. Counts alone are insufficient to establish drift: the exact extra/missing values and their provenance are required.

Refreshed implementation plan

1. Capture exact sets in Run bounded ChEMBL dashboard reconciliation campaign #6324. For all eight dashboards, persist sorted (pipeline, run_type) values, query source, sample timestamp, and datasource response. Produce pairwise set differences rather than counts only.
2. Classify every difference. Mark each extra/missing value as runtime-observed, control-plane-only historical/planned, Explorer row-backed, stale Pushgateway residue, alias/config drift, or unexplained. Unexplained values block closure.
3. Enforce the intended relations. Primary dashboard universes must be equal. Explorer must remain evidence-scoped to pipelines with record data and should be subset-safe for handoff. Control Plane may expose control-plane-only values, but every extra must have a documented artifact/lifecycle reason. Encode these relations in selector fixtures/tests instead of forcing flat equality.
4. Fix navigation behavior. Cross-dashboard links must preserve a selected value when the target universe supports it; otherwise show an explicit unsupported/unknown state and a recovery choice. Never silently substitute another pipeline.
5. Standardize only unexplained drift. If the live evidence shows accidental differences, move the affected dashboard to the canonical recording rule. If differences are intentional, update docs/03-guides/dashboards/contracts/selector-contracts.yaml, variable-reference.md, and panel descriptions with the exact rationale.
6. Verify. Extend tests/integration/test_grafana_selector_contract.py, run selector/dashboard integration checks, and complete a live navigation audit over the captured value matrix.

Suggested issue framing: “Validate role-specific pipeline selector set relations and cross-dashboard handoffs.”

[SatoryKono]

Closeout — live selector relation and navigation audit (2026-07-20)

Verdict: PASS; unexplained drift = 0.

Audited live Grafana 12.0.0 through its provisioned Prometheus datasource at main@0b65839d20f818b15132abc32e4d4f32062cd745.

Exact resolved sets

• Six shared dashboards (Overview, Runtime, Provider, DQ, Workflow, Alerts): 27 pipelines / 54 (pipeline, run_type) pairs. Overview and Runtime were exactly equal at both 12h and 24h; all 27 pipelines had backfill and incremental.
• Control Plane: 15 pipelines / 16 pairs.
• Silver Reject Explorer: 15 pipelines / 16 pairs.

Shared pipelines:
chembl_activity, chembl_assay, chembl_assay_parameters, chembl_cell_line, chembl_compound_record, chembl_molecule, chembl_protein_class, chembl_publication, chembl_publication_similarity, chembl_publication_term, chembl_subcellular_fraction, chembl_target, chembl_target_component, chembl_target_protein_classification, chembl_tissue, composite_activity, composite_assay, composite_molecule, composite_publication, composite_target, crossref_publication, openalex_publication, pubchem_compound, pubmed_publication, semanticscholar_publication, uniprot_idmapping, uniprot_protein.

Control Plane / Explorer pipelines:
chembl_activity, chembl_assay, chembl_assay_parameters, chembl_cell_line, chembl_compound_record, chembl_molecule, chembl_protein_class, chembl_publication, chembl_publication_similarity, chembl_publication_term, chembl_subcellular_fraction, chembl_target, chembl_target_component, chembl_target_protein_classification, chembl_tissue.

Their 16 pairs are chembl_activity:{backfill,incremental} plus incremental for the other 14 pipelines.

Provenance classification

• All 38 canonical-only pairs are present in bioetl_pipeline_runs_total.
• Every observed Control Plane pair is backed by bioetl_control_plane_manifest_writes_total; Control Plane-only unexplained values: 0.
• Every Explorer pair is backed by bioetl_records_processed_total; Explorer values outside canonical: 0.
• bioetl_workflow_pipeline_expected had no samples in the captured windows, but runtime/manifest/row sources fully explained every observed value.

Handoff audit

• All 8 dashboards were live and provisioned.
• 56/56 dashboard-to-dashboard links targeted every other dashboard.
• Every handoff preserved ${__url_time_range} and a variable-backed var-pipeline; silent substitutions: 0.
• Control Plane and Explorer role-local misses retain explicit fail-closed recovery guidance.

Repo-side closure evidence and regression guard were prepared at:

• reports/observability/grafana/selector-audit-2026-07-20/selector-evidence.json
• reports/observability/grafana/selector-audit-2026-07-20/selector-audit-report.md
• tests/integration/test_grafana_selector_contract.py

Validation:

• 219 focused Grafana/Prometheus integration tests passed.
• 25 architecture/debt/docs guard tests passed.
• Ruff and JSON/YAML parsing passed.
• Docs link check reported only the two pre-existing unrelated broken links in SKILLS-CATALOG.md and the hierarchical-evidence skill mirror.

The shipped role-specific selector contract is therefore validated rather than flattened. Closing as completed.