[Docker/Grafana][P1] RF-017 Validate convergence and close workstation cutover

[SatoryKono]

Parent: #6303
Depends on #6307, #6308, #6309, and the implementation/not-required decision in #6310.

Problem

The cutover must close on repeatable runtime and dashboard evidence, not on containers merely reaching running.

Observation window (operator override 2026-07-21)

• Original plan: 24 hours
• Active requirement: 2.4 hours of continuous healthy observation after canonical restore
• Attempt 1 started: 2026-07-21T06:07:40.967431+00:00 — failed (monitoring exited during Desktop disruption)
• Attempt 2 started: 2026-07-21T08:43:04.438078+00:00
• Planned finish: 2026-07-21T11:07:04.438078+00:00
• Finished: 2026-07-21T13:15:10.610434+00:00 — passed finalize gate
• Evidence: reports/quality/docker-dashboard-cutover-observation.json
• Closeout: reports/quality/docker-dashboard-cutover-rf017-closeout-20260721.json
• Archive: reports/quality/docker-monitoring-backups/rf017-closeout-20260721/

Tasks

• Run static Compose, runtime-contract, HTTP/Grafana, launcher parity, and governance tests (where host allows).
• Render affected Compose files with process-injected placeholder variable names; do not create/edit env files.
• Require canonical preflight/project origins for main+monitoring (no /tmp, no mixed Windows bind origins).
• Live audit for an exact run with nonzero accounting and a legitimate-zero case (2026-07-15 evidence retained).
• Execute 10 start/stop canary cycles for bioetl-main + bioetl-monitoring (2026-07-15 evidence).
• Verify stateful volume identity and restart-count delta on canary cycles.
• Observe the corrected stack for 2.4 hours (was 24h; operator override).
• Synchronize runbook/docs and archive temporary audit evidence; no /tmp Compose path may remain supported.
• Record skipped host-only checks, mirror status, debt outcome, and legacy-volume retention decision.

Acceptance criteria

• docker compose ls --all shows canonical project boundaries without mixed /tmp or Windows config origins (restored 2026-07-21).
• Exactly one container owns each required port and network alias for required services.
• Host/in-network Prometheus identity and exact query results matched in 2026-07-15 cutover evidence.
• ID is resolved and Processed Records is numeric, including a valid-zero case (2026-07-15 live audit).
• No required panel is unavailable, no-data, partial, malformed, or placeholder-only (2026-07-15 live audit).
• 2.4h observation has zero unexpected exits, restart increments, OOMs, or unresolved unhealthy states on required services (attempt 2 finalize: restart_count_delta=0, oom=0, unhealthy=0).
• .env files and technical-debt budgets are unchanged in this closeout path.
• Parent [Docker/Grafana][P1] Complete runtime cutover and restore ID / Processed Records #6303 checklist and evidence links are complete.

Boundary

This issue closes the workstation cutover only. The parent Docker stabilization campaign still owns the later 100-cycle and 72-hour release-level claim.

Rollback

Reject closeout and return to the last successful RF checkpoint. Retained legacy volumes/archives remain the recovery source.

[SatoryKono]

Convergence progress: repository commit опубликован; canonical preflight 0 errors/0 warnings; live required panels failures=0; ID/Processed Records resolved; Prometheus host/network parity подтверждена; canary 10/10, restart delta=0, OOM=0, unresolved unhealthy=0, volume identity stable. Закрытие пока корректно заблокировано двумя критериями: RF-016 data mutation ожидает отдельного approval (#6310), после promotion требуется полный 24-hour observation.

[SatoryKono]

Closeout plan (RF-017)

1. Complete the remaining RF-016 decision/mutation gate and link its evidence.
2. Run static Compose, runtime-contract, HTTP/Grafana, launcher parity, and governance suites; render Compose with placeholder variable names only and never create/edit env files.
3. Require canonical preflight and live audit to report zero project-origin, port-owner, restart, OOM, unhealthy, capacity, Prometheus-identity, no-data, partial, malformed, or placeholder-only findings.
4. Run and retain evidence for 10/10 bioetl-main + bioetl-monitoring start/stop canaries with stable volume identity, restart delta 0, OOM 0, and no unresolved unhealthy state.
5. Observe the corrected stack for a full 24 hours and capture exact start/end timestamps, incident log, Prometheus identity/payload parity, and panel audit results.
6. Synchronize runbooks/docs and archive temporary audit evidence; prove no supported /tmp Compose path remains.
7. Record skipped host-only checks, mirror status, debt outcome, and legacy-volume retention decision, then close this issue and its parent [Docker/Grafana][P1] Complete runtime cutover and restore ID / Processed Records #6303.

Current blockers: RF-016 mutation approval and the 24-hour observation window. The 100-cycle/72-hour release campaign remains #6299.

[SatoryKono]

Expanded implementation and closeout plan — RF-017

1. Complete and link [Docker/Grafana][P2] RF-016 Add bounded latest RunManifest lookup only if required #6310 mutation/latency evidence.
2. Run static Compose, runtime-contract, HTTP/Grafana, launcher-parity, and governance suites; render Compose with placeholder variable names only and never edit env files.
3. Require preflight/live audit zero findings for project origin, port ownership, restart/OOM, unhealthy/capacity, Prometheus identity, no-data, partial, malformed, and placeholder-only panels.
4. Run 10/10 bioetl-main + bioetl-monitoring start/stop canaries with stable volume identity, restart delta 0, OOM 0, and no unresolved unhealthy state.
5. Observe the corrected stack for 24 hours with timestamps, incident log, Prometheus identity/payload parity, and panel audit evidence.
6. Synchronize runbooks/docs and prove no supported /tmp Compose path remains.
7. Record skipped host-only checks, mirror status, debt outcome, and legacy-volume retention; close only after all evidence is linked. The 100-cycle/72-hour campaign remains [Docker stability][P1] RF-008 Prove stability with fault injection, 100 cycles, and a 72-hour soak #6299.

[SatoryKono]

Closeout attempt — 2026-07-20 (blocked; issue remains OPEN)

Attempted to finish RF-017 gates and close this issue. Not closed: live host gates failed before the 24h observation could be started or completed.

Already satisfied from prior evidence (2026-07-15)

Gate	Evidence	Result
Preflight	reports/quality/docker-dashboard-cutover-preflight.json	errors=0 warnings=0
Live panels (ID / Processed Records / legitimate zero)	reports/quality/docker-dashboard-cutover-live-audit.json	214/214 ok
Canary 10/10	reports/quality/docker-dashboard-cutover-canary.json	restartΔ=0, OOM=0, volumes stable
Prometheus host/network parity	reports/quality/docker-dashboard-cutover-final.json	host_equals_grafana_network=true
RF-016 (#6310)	closed 2026-07-15	mutation + index applied

Final cutover report still records:

"status": "observation_pending",
"observation": {
  "required_hours": 24,
  "started_at": null,
  "finished_at": null,
  "status": "not_started_pending_rf016_promotion"
}

RF-016 is done, but the 24h observation was never started in that artifact.

Live host state found during this attempt (blocking)

docker compose ls --all showed non-canonical monitoring origins:

• bioetl-main → Linux runtime (good)
• bioetl-monitoring → mixed /tmp/bioetl-issue-6324... + Windows E:\g-drive\... compose paths (fails AC: no /tmp path / no mixed origins)
• bioetl-main-bioetl-1 still attached to legacy warp-network in addition to bioetl-monitoring
• bioetl-grafana unhealthy (HTTP 503 on /api/health)
• Preflight recheck from this checkout: reports/quality/docker-dashboard-cutover-preflight-recheck.json → ok=false, 17 errors (PROJECT_ORIGIN, SECRET_MISSING from wrong repo-root context, CAPACITY_DOCKER_ROOT)

Recovery / recreate attempts

1. Prepared recreate helpers (no .env writes, no down -v):
   • scripts/ops/runtime/docker/recreate_cutover_stacks.py
   • scripts/ops/runtime/docker/_rf017_recreate_once.py
2. docker compose ... down against monitoring hung (>10 min) under mixed origins.
3. Subsequent docker inspect of bioetl-grafana began timing out; daemon became intermittently unresponsive.
4. Bounded Desktop recovery:
   • pwsh scripts/ops/runtime/docker/restart-docker.ps1 -TimeoutSeconds 180 -ReportPath reports/quality/docker-desktop-recovery-rf017.json
   • failed: Supported Docker Desktop stop fallback did not complete successfully within its command deadline

No volumes were deleted. No .env files were created/edited. Technical-debt budgets unchanged.

Remaining closeout sequence (do not weaken)

1. Recover Docker Desktop to a responsive state (operator-confirmed last-resort only if normal recovery fails; see [Docker stability][P1] RF-006 Replace force-kill recovery with bounded Desktop/WSL recovery #6297).
2. Recreate only from Linux runtime origin:
   • /home/fedor/.local/share/bioetl-runtime/BioactivityDataAcquisition2
   • projects bioetl-main + bioetl-monitoring
   • no /tmp compose, no Windows bind origin, disconnect warp-network
3. Re-run preflight with that runtime as --repo-root → require 0 errors / 0 warnings.
4. Re-run live audit for exact run + legitimate zero.
5. Re-run 10/10 canary if restart baseline changes during recreate.
6. Start 24h observation with timestamps; finish only after 24h with restartΔ=0, OOM=0, no unresolved unhealthy.
7. Sync runbooks/docs; update docker-dashboard-cutover-final.json to status=complete; close this issue; then close parent [Docker/Grafana][P1] Complete runtime cutover and restore ID / Processed Records #6303.

Debt / skip notes

• Debt outcome this attempt: unchanged.
• Skipped: full 24h observation (host not stable enough to start), fresh live audit, fresh canary, docs closeout.
• Related open blockers: [Docker stability][P1] RF-006 Replace force-kill recovery with bounded Desktop/WSL recovery #6297 (Desktop recovery proof), [Docker stability][P1] RF-008 Prove stability with fault injection, 100 cycles, and a 72-hour soak #6299 (release campaign; separate from workstation cutover).

Verdict: keep OPEN. Prior cutover implementation remains valuable, but RF-017 acceptance is evidence-driven and currently fails live topology + observation gates.

[SatoryKono]

RF-017 revalidation (2026-07-20) — not closeable yet

Dependency status

• [Docker/Grafana][P2] RF-016 Add bounded latest RunManifest lookup only if required #6310 is closed (implemented_and_verified with mutation gate completed).
• [Docker/Grafana][P1] RF-013 Converge datasource, data root, and Prometheus identity #6307–[Docker/Grafana][P1] RF-015 Prevent launchers from recreating mixed topology #6309 remain closed prerequisites.
• Parent [Docker/Grafana][P1] Complete runtime cutover and restore ID / Processed Records #6303 still open; remaining open item on its checklist is canary 10/10 + 24h observation (this issue).

Live topology snapshot (before recreate attempt)

docker compose ls --all:

Project	Config origins
bioetl-main	Linux runtime OK: /home/fedor/.local/share/bioetl-runtime/BioactivityDataAcquisition2/docker-compose.yml
bioetl-monitoring	mixed / non-canonical: /tmp/bioetl-issue-6324.SE9qqL/repo/... + audit compose + Windows E:\g-drive\...\docker-compose.monitoring.yml
bioetl-neo4j	Linux runtime path

This fails acceptance: no supported /tmp Compose path; no mixed config origins.

Also observed:

• bioetl-grafana unhealthy
• audit sidecars still present (bioetl-quarantine-explorer-audit:18081, bioetl-promtail-audit)
• many non-project containers (Created/Up) outside bioetl-* naming

Preflight (host process, no secret injection)

Evidence: C:\Users\Fedor\tmp\bioetl-6311-preflight-20260720-205650.json

summary: ok=false, errors=18, warnings=1

Notable finding codes:

• PROJECT_ORIGIN ×5 (mixed /tmp + Windows + unexpected paths)
• CONTAINER_HEALTH (unhealthy container)
• many SECRET_MISSING / ENVIRONMENT_MISSING (expected for host-side preflight without process-injected secrets from containers)
• transient DOCKER_DAEMON / capacity noise during daemon flapping

Recreate attempt

Tried canonical restore via:

python scripts/ops/runtime/docker/recreate_cutover_stacks.py --include-tracing-profile

• WSL runtime compose files reachable (\\wsl$\Ubuntu\home\fedor\.local\share\bioetl-runtime\BioactivityDataAcquisition2\...).
• Failed at first docker inspect bioetl-grafana (60s timeout), then Docker Desktop entered a bad state:
  • restart/inspect returned EOF / HTTP 500
  • subsequent docker info → “Docker Desktop is unable to start” / missing named pipe for ~3+ minutes

No stack recreation completed. Volumes were not intentionally pruned (down -v not used).

Why this issue cannot close now

Acceptance item	Status
Canonical project boundaries / no mixed origins	❌ still mixed; recreate blocked
One owner per required port/alias	⚠ ports looked exclusive at last healthy snapshot, but audit sidecars still present
Host/in-network Prometheus identity match	not re-proven this session
ID / Processed Records live audit	not re-run (Grafana unhealthy / Docker down)
10/10 start-stop canaries	not run
24h observation	not started
.env / debt budgets unchanged	✅ no .env edits; no debt-budget changes

Next operator steps (when Docker Desktop is healthy again)

1. Recover Docker Desktop fully (docker info green).
2. Re-run recreate_cutover_stacks.py (optionally --include-tracing-profile if Loki/Tempo required).
3. Re-run full preflight with process-injected secrets / stack context until errors=0.
4. Live Grafana audit (nonzero accounting + legitimate zero).
5. 10× start/stop canaries for bioetl-main + bioetl-monitoring with volume identity + restart delta 0.
6. Start 24h observation window with start/end timestamps + incident log.
7. Only then close [Docker/Grafana][P1] RF-017 Validate convergence and close workstation cutover #6311 and update parent [Docker/Grafana][P1] Complete runtime cutover and restore ID / Processed Records #6303.

Verdict: keep #6311 open. Current blocker is host Docker Desktop instability + non-canonical monitoring project origin, not missing code from RF-010–RF-016.

[SatoryKono]

RF-017 progress — 2026-07-21 (observation started; issue remains OPEN)

Topology restored to canonical origins

docker compose ls --all now shows no /tmp and no Windows E:\ mixed origins:

Project	Config
bioetl-main	\\wsl$\Ubuntu\home\fedor\.local\share\bioetl-runtime\BioactivityDataAcquisition2\docker-compose.yml
bioetl-monitoring	same Linux-runtime docker-compose.monitoring.yml
bioetl-neo4j	/home/fedor/.local/share/bioetl-runtime/.../docker-compose.neo4j.yml

Required services healthy after recreate:

• bioetl (main HTTP :8081) — healthy; networks bioetl-monitoring + bioetl-runtime (no warp-network)
• bioetl-prometheus, bioetl-pushgateway, bioetl-grafana, renderer — healthy
• Live HTTP probes: Prometheus/Pushgateway/Grafana/Explorer all 200

Volumes preserved (Compose warned volumes already existed; no down -v).

Prior gates still on disk (2026-07-15)

• Canary 10/10: reports/quality/docker-dashboard-cutover-canary.json
• Preflight 0/0 (historical cutover): reports/quality/docker-dashboard-cutover-preflight.json
• Live panel audit 214/214: reports/quality/docker-dashboard-cutover-live-audit.json
• RF-016 ([Docker/Grafana][P2] RF-016 Add bounded latest RunManifest lookup only if required #6310) closed completed 2026-07-15

24h observation — started (blocker for close)

Field	Value
Status	in_progress
Started	2026-07-21T06:07:40.967431+00:00
Planned finish	2026-07-22T06:07:40.967431+00:00
Evidence	reports/quality/docker-dashboard-cutover-observation.json
Final status	observation_in_progress in docker-dashboard-cutover-final.json

Finalize script (after 24h):

python scripts/ops/runtime/docker/_rf017_finalize_observation.py

A background 24h waiter was also scheduled on this host to run that script.

Notes / residual risk

• Process-env preflight from Windows without injected secrets reports SECRET_MISSING / capacity noise; cutover AC is about project origins + live health, not Windows process env.
• Optional tracing profile (Loki) was left down after unhealthy start; required monitoring services only.
• Docker Desktop was offline earlier today; this observation window starts from the post-restore healthy baseline (restart counts 0).

Close criteria remaining

• Observation ≥24h with restartΔ=0, OOM=0, no unresolved unhealthy, origins still canonical
• Run finalize script and attach closeout JSON
• Close this issue, then parent [Docker/Grafana][P1] Complete runtime cutover and restore ID / Processed Records #6303

Not closing yet — would be false closeout before the 24h gate completes.

[SatoryKono]

Observation window override — 2026-07-21

Per operator request, the RF-017 soak gate is reduced:

Field	Before	After
Required hours	24	2.4
Started	2026-07-21T06:07:40.967431+00:00	unchanged
Planned finish	2026-07-22T06:07:40Z	2026-07-21T08:31:40.967431+00:00

Updated surfaces:

• reports/quality/docker-dashboard-cutover-observation.json
• reports/quality/docker-dashboard-cutover-final.json
• scripts/ops/runtime/docker/_rf017_finalize_observation.py (reads required_hours from observation file)
• issue body acceptance criteria

Finalize after the new window:

python scripts/ops/runtime/docker/_rf017_finalize_observation.py

Issue remains OPEN until observation completes successfully.