From 5be08fcca7c7ef36d7023cbd29ed4f8dda6f10fd Mon Sep 17 00:00:00 2001 From: Fabio Leitao Date: Wed, 12 Aug 2026 10:00:04 -0300 Subject: [PATCH] docs(plans): refresh lab observability stack status after ingest validation (#1542) Phases A+C live, F partial, D not adopted; document native OTel receivers and Redpanda/OpenSearch as validation targets only. --- .../OPERATOR_TODAY_MODE_2026-08-12.md | 4 +- .../OPERATOR_TODAY_MODE_2026-08-12.pt_BR.md | 4 +- docs/plans/PLANS_HUB.md | 2 +- docs/plans/PLANS_TODO.md | 4 +- docs/plans/PLAN_LAB_OP_OBSERVABILITY_STACK.md | 42 ++++++++++++---- .../PLAN_LAB_OP_OBSERVABILITY_STACK.pt_BR.md | 50 +++++++++++++------ 6 files changed, 72 insertions(+), 34 deletions(-) diff --git a/docs/ops/today-mode/OPERATOR_TODAY_MODE_2026-08-12.md b/docs/ops/today-mode/OPERATOR_TODAY_MODE_2026-08-12.md index 2fc6e77e6..9a4a26a14 100644 --- a/docs/ops/today-mode/OPERATOR_TODAY_MODE_2026-08-12.md +++ b/docs/ops/today-mode/OPERATOR_TODAY_MODE_2026-08-12.md @@ -30,10 +30,10 @@ | Band | Item | State | Next thin slice | | ---- | ---- | ----- | --------------- | -| **Docs almost ready** | [#1542](https://github.com/DataBoar/data-boar/issues/1542) — refresh `PLAN_LAB_OP_OBSERVABILITY_STACK` Status + native receivers | OPEN — lab facts done, **plan header stale** | One docs PR: Status line + phase note (Phase A+C live; D **not** adopted; Redpanda/OpenSearch = receiver targets only) | +| **Docs almost ready** | [#1542](https://github.com/DataBoar/data-boar/issues/1542) — refresh `PLAN_LAB_OP_OBSERVABILITY_STACK` Status + native receivers | In PR (docs branch) — Status + §1.1 receivers | Merge docs PR; closes lab↔plan gap | | **Docs drift** | [#1538](https://github.com/DataBoar/data-boar/issues/1538) — product tiers / open-core plan stale | OPEN · `no-code-yet` | Thin docs when band allows | | **Docs drift** | [#1541](https://github.com/DataBoar/data-boar/issues/1541) — `PLAN_NATIVE_PACKAGES` missing MSI/Homebrew/Windows-CI links | OPEN | Linkage-only PR | -| **Product OTel gaps** | [#1529](https://github.com/DataBoar/data-boar/issues/1529) LoggerProvider → Loki | OPEN · P3 | After lab log path stable; enables log proof of licensed filter | +| **Product OTel gaps** | [#1529](https://github.com/DataBoar/data-boar/issues/1529) LoggerProvider → Loki | ✅ merged **#1544** | Evidence under `docs/ops/evidence/otel_1529_*` | | **Product OTel gaps** | [#1535](https://github.com/DataBoar/data-boar/issues/1535) OTel only on `--web`/`--demo` | OPEN | Design slice: oneshot CLI / exports visibility | | **Maestro / lab trust** | [#1540](https://github.com/DataBoar/data-boar/issues/1540) preflight that gate trusts wired OTel | OPEN | Verify, do not assume | | **Packaging P1** | [#1427](https://github.com/DataBoar/data-boar/issues/1427) Windows CI zero jobs | OPEN · P1 | Blocks MSI/winget story with [#1467](https://github.com/DataBoar/data-boar/issues/1467) | diff --git a/docs/ops/today-mode/OPERATOR_TODAY_MODE_2026-08-12.pt_BR.md b/docs/ops/today-mode/OPERATOR_TODAY_MODE_2026-08-12.pt_BR.md index e37c1ee38..3f09076da 100644 --- a/docs/ops/today-mode/OPERATOR_TODAY_MODE_2026-08-12.pt_BR.md +++ b/docs/ops/today-mode/OPERATOR_TODAY_MODE_2026-08-12.pt_BR.md @@ -30,10 +30,10 @@ | Faixa | Item | Estado | Próxima fatia | | ----- | ---- | ------ | ------------- | -| **Docs quase prontos** | [#1542](https://github.com/DataBoar/data-boar/issues/1542) — Status do `PLAN_LAB_OP_OBSERVABILITY_STACK` + receivers nativos | OPEN — lab feito, **header do plano velho** | Um PR só docs | +| **Docs quase prontos** | [#1542](https://github.com/DataBoar/data-boar/issues/1542) — Status do `PLAN_LAB_OP_OBSERVABILITY_STACK` + receivers nativos | Em PR (branch docs) — Status + §1.1 | Merge do PR de docs; fecha gap lab↔plano | | **Drift docs** | [#1538](https://github.com/DataBoar/data-boar/issues/1538) tiers / open-core | OPEN | Docs fino | | **Drift docs** | [#1541](https://github.com/DataBoar/data-boar/issues/1541) links MSI/Homebrew/Windows-CI no plano de packaging | OPEN | PR só links | -| **Gaps OTel produto** | [#1529](https://github.com/DataBoar/data-boar/issues/1529) LoggerProvider → Loki | OPEN · P3 | Depois do path de log do lab estável | +| **Gaps OTel produto** | [#1529](https://github.com/DataBoar/data-boar/issues/1529) LoggerProvider → Loki | ✅ merge **#1544** | Evidência em `docs/ops/evidence/otel_1529_*` | | **Gaps OTel produto** | [#1535](https://github.com/DataBoar/data-boar/issues/1535) OTel só em `--web`/`--demo` | OPEN | Design: oneshot CLI | | **Maestro / lab** | [#1540](https://github.com/DataBoar/data-boar/issues/1540) preflight OTel no gate | OPEN | Verificar, não assumir | | **Packaging P1** | [#1427](https://github.com/DataBoar/data-boar/issues/1427) CI Windows zero jobs | OPEN · P1 | Bloqueia narrativa MSI com [#1467](https://github.com/DataBoar/data-boar/issues/1467) | diff --git a/docs/plans/PLANS_HUB.md b/docs/plans/PLANS_HUB.md index 18107b1a4..9e97811ee 100644 --- a/docs/plans/PLANS_HUB.md +++ b/docs/plans/PLANS_HUB.md @@ -90,7 +90,7 @@ Do **not** edit the table manually; refresh with `python scripts/plans_hub_sync. | **Open** | [PLAN_LAB_OP_CAPEX_OPEX_PROCUREMENT.md](PLAN_LAB_OP_CAPEX_OPEX_PROCUREMENT.md) | LAB-OP — CAPEX / OPEX & Procurement (tracked plan) | **Status:** Active **Date:** 2026-04-02 **Authors:** Fabio Leitao **Priority:** H2 | — | | **Open** | [PLAN_LAB_OP_CAPEX_OPEX_PROCUREMENT.pt_BR.md](PLAN_LAB_OP_CAPEX_OPEX_PROCUREMENT.pt_BR.md) | LAB-OP — CAPEX / OPEX & Procurement (plano rastreado) | **Status:** Pending **Date:** 2026-04-02 **Authors:** Fabio Leitao **Priority:** H2 | — | | **Open** | [PLAN_LAB_OP_OBSERVABILITY_STACK.md](PLAN_LAB_OP_OBSERVABILITY_STACK.md) | Lab-op observability stack — metrics, logs, dashboards (plan only) | Sequence **optional** homelab instrumentation—**Grafana**, time-series DBs, **centralized logs**—without blocking Data Boar development or **–1L** validation. **No** implementation in this repo; operator deploys via Comp | — | -| **Open** | [PLAN_LAB_OP_OBSERVABILITY_STACK.pt_BR.md](PLAN_LAB_OP_OBSERVABILITY_STACK.pt_BR.md) | Stack de observabilidade no lab-op — métricas, logs, dashboards (só plano) | **Status:** Active (Fase A + parte da F no ar no lab; emit do produto = [#1500](https://github.com/DataBoar/data-boar/issues/1500)) **Date:** 2026-08-09 **Authors:** Fabio Leitao **Priority:** H2 | — | +| **Open** | [PLAN_LAB_OP_OBSERVABILITY_STACK.pt_BR.md](PLAN_LAB_OP_OBSERVABILITY_STACK.pt_BR.md) | Stack de observabilidade no lab-op — métricas, logs, dashboards (só plano) | **Status:** Active (Fases **A** + **C** no ar no lab; Fase **F** parcial; Fase **D** não adotada; emit do produto = [#1500](https://github.com/DataBoar/data-boar/issues/1500) + [#1529](https://github.com/DataBoar/data-bo | — | | **Open** | [PLAN_LATO_SENSU_THESIS.md](PLAN_LATO_SENSU_THESIS.md) | Plan: Lato sensu thesis – Data Boar as applied artifact | **Status:** Deferred **Date:** 2026-03-17 **Authors:** Fabio Leitao **Priority:** H3 | — | | **Open** | [PLAN_LOCUST_LOAD_TEST_INTEGRATION.md](PLAN_LOCUST_LOAD_TEST_INTEGRATION.md) | Plan: Locust load test integration with Maestro | Locust HTTP load testing wired into Maestro's web persona pipeline; tests API scan, health, dashboard endpoints against synthetic lab targets. | [PLAN_MAESTRO_BENCHMARK_METRICS_AND_FIX.md](PLAN_MAESTRO_BENCHMARK_METRICS_AND_FIX.md) [PLAN_SELENIUM_QA_TEST_SUITE.md](PLAN_SELENIUM_QA_TEST_SUITE.md) | | **Open** | [PLAN_MAESTRO_BENCHMARK_METRICS_AND_FIX.md](PLAN_MAESTRO_BENCHMARK_METRICS_AND_FIX.md) | Plan: Maestro benchmark metrics, bug fixes, and full test matrix | Fix 3 confirmed Maestro bugs causing 9h wait-states; add per-run metrics collection; build comprehensive test matrix across all lab personas, OSes, protocols, and DB targets for v1.7.3 vs v1.7.4-rc A/B. | [PLAN_LOCUST_LOAD_TEST_INTEGRATION.md](PLAN_LOCUST_LOAD_TEST_INTEGRATION.md) [PLAN_SQL_SAMPLING_SRE_AND_AUDIT_EVIDENCE.md](PLAN_SQL_SAMPLING_SRE_AND_AUDIT_EVIDENCE.md) | diff --git a/docs/plans/PLANS_TODO.md b/docs/plans/PLANS_TODO.md index d27041598..07d044fc3 100644 --- a/docs/plans/PLANS_TODO.md +++ b/docs/plans/PLANS_TODO.md @@ -55,9 +55,9 @@ When revising **license terms** for IP, commerciality, and profitability, explic When **partners** or **buyers** anchor on a vertical (MSP, insurance, RPO, real estate, NGO, EdTech, port, law, accounting, pharma), route demand **without scope creep**: (1) use the narrative index **[../use-cases/README.md](../use-cases/README.md)** ([pt-BR](../use-cases/README.pt_BR.md)); (2) map gaps to **shipped** operator surfaces (`USAGE`, connectors, reports) before inventing new flags; (3) promote **one** backlog row here or in the relevant `PLAN_*.md` when **repeatable revenue evidence** exists; keep token batching per **[TOKEN_AWARE_USAGE.md](TOKEN_AWARE_USAGE.md)**. -### Integration / active threads — last refreshed **2026-08-09** +### Integration / active threads — last refreshed **2026-08-12** -- **OpenTelemetry product emit ([#1500](https://github.com/DataBoar/data-boar/issues/1500)):** ✅ shipped **#1503** — [PLAN_DATABOAR_OTEL_INSTRUMENTATION.md](completed/PLAN_DATABOAR_OTEL_INSTRUMENTATION.md) (opt-in `DATA_BOAR_OTEL_ENABLED` + `[otel]`). Lab receive stack remains [PLAN_LAB_OP_OBSERVABILITY_STACK.md](PLAN_LAB_OP_OBSERVABILITY_STACK.md). GenAI/MCP OTel still [#1457](https://github.com/DataBoar/data-boar/issues/1457). +- **OpenTelemetry product emit ([#1500](https://github.com/DataBoar/data-boar/issues/1500)):** ✅ shipped **#1503** — [PLAN_DATABOAR_OTEL_INSTRUMENTATION.md](completed/PLAN_DATABOAR_OTEL_INSTRUMENTATION.md) (opt-in `DATA_BOAR_OTEL_ENABLED` + `[otel]`). **Logs bridge ([#1529](https://github.com/DataBoar/data-boar/issues/1529)):** ✅ **#1544** (`LoggerProvider` → OTLP → Loki). Lab receive stack: [PLAN_LAB_OP_OBSERVABILITY_STACK.md](PLAN_LAB_OP_OBSERVABILITY_STACK.md) — Phases **A**+**C** live, **F** partial, **D** not adopted (doc refresh [#1542](https://github.com/DataBoar/data-boar/issues/1542)). GenAI/MCP OTel still [#1457](https://github.com/DataBoar/data-boar/issues/1457). - **Rust regex stage (open-domain YAML / [#1414](https://github.com/DataBoar/data-boar/issues/1414)):** [PLAN_RUST_REGEX_STAGE.md](PLAN_RUST_REGEX_STAGE.md) — **form B Accepted** ([ADR 0083](../adr/ADR-0083-rust-regex-stage-superset-accept-form-b.md)); **RegexSet rejected** (probe 2026-07-31: 5.7× slower than cached `Regex` loop; [ADR 0078](../adr/ADR-0078-multi-pattern-regex-benchmark-gate-regexset-before-vectorscan.md) amended). Path: Python translator (port `translate.py`) + `Vec` cache; 162 accel / 112 translated / 1 fallback measured. **#1412 is merge prerequisite** (engine+version, accelerated/translated/fallback counts+reasons, rust-only patterns). Not the old ML-gating prefilter/latch. Zero hardcoded patterns in `lib.rs`. Open-core: Community refuses unsafe patterns + warns (#829); B removes a paid-path limitation only. Bus [#1413](https://github.com/DataBoar/data-boar/issues/1413). - **Completão A/B benchmark (`v1.7.3` lab ref vs `origin/main`):** `scripts/benchmark-ab.ps1` + `docs/plans/BENCHMARK_EVOLUTION.md` — **2026-04-27** hardened **`lab-op-git-ensure-ref`** (SSH stderr / `Start-Process`), **`completaoImageRefs`** parsing, and image-preflight coercion in **`lab-completao-orchestrate.ps1`**; benchmark default is **lab-ref-only** Round A (avoid local checkout of legacy tag unless `-LocalLegacyCheckout`). **Wall-clock A/B not finished** in the agent tool window (Round A can exceed **~60 min** on the full manifest) — operator: run the script locally to produce **`benchmark_runs/times.txt`** + **`telemetry.json`**; private checklist **`docs/private/homelab/BENCHMARK_AB_OPERATOR_NOTES_2026-04-27.pt_BR.md`**. **Safe recall axis (#1338):** blocking gate in `tests/benchmarks/benchmark_gate.py` + `run_official_bench.py` — exact finding parity on synthetic corpora; **#1337** sampling dedup waits for #1338 merge — [PLAN_BENCHMARK_SAFE_AXIS.md](PLAN_BENCHMARK_SAFE_AXIS.md). diff --git a/docs/plans/PLAN_LAB_OP_OBSERVABILITY_STACK.md b/docs/plans/PLAN_LAB_OP_OBSERVABILITY_STACK.md index 31e697f73..6f11e20a4 100644 --- a/docs/plans/PLAN_LAB_OP_OBSERVABILITY_STACK.md +++ b/docs/plans/PLAN_LAB_OP_OBSERVABILITY_STACK.md @@ -1,13 +1,13 @@ # Lab-op observability stack — metrics, logs, dashboards (plan only) -**Status:** Active (Phase A + part of F live in lab; product emit = [#1500](https://github.com/DataBoar/data-boar/issues/1500)) -**Date:** 2026-08-09 +**Status:** Active (Phases **A** + **C** live in lab; Phase **F** partial; Phase **D** not adopted; product emit = [#1500](https://github.com/DataBoar/data-boar/issues/1500) + [#1529](https://github.com/DataBoar/data-boar/issues/1529)) +**Date:** 2026-08-12 **Authors:** Fabio Leitao **Priority:** H2 **Português (Brasil):** [PLAN_LAB_OP_OBSERVABILITY_STACK.pt_BR.md](PLAN_LAB_OP_OBSERVABILITY_STACK.pt_BR.md) -**Product emit (code in this repo):** [PLAN_DATABOAR_OTEL_INSTRUMENTATION.md](completed/PLAN_DATABOAR_OTEL_INSTRUMENTATION.md) — FastAPI/SQLAlchemy OTLP opt-in; does not replace operator-side collector/Grafana deploy. +**Product emit (code in this repo):** [PLAN_DATABOAR_OTEL_INSTRUMENTATION.md](completed/PLAN_DATABOAR_OTEL_INSTRUMENTATION.md) — FastAPI/SQLAlchemy OTLP opt-in (**traces + metrics + logs** via `LoggerProvider`, [#1529](https://github.com/DataBoar/data-boar/issues/1529)); does not replace operator-side collector/Grafana deploy. **Purpose:** Sequence **optional** homelab instrumentation—**Grafana**, time-series DBs, **centralized logs**—without blocking Data Boar development or **–1L** validation. **No** implementation in this repo; operator deploys via Compose, k3s Helm, or vendor appliances on **lab-op** hosts (ThinkPad LAB-NODE-01, LAB-NODE-02, Proxmox guests). @@ -25,17 +25,34 @@ ## 1. Recommended sequence (light → heavy) -| Phase | Stack | Role | Notes | +| Phase | Stack | Role | Lab status (2026-08-12) / notes | | ----- | ----- | ---- | ----- | -| **A** | **Grafana** + **Prometheus** (+ `node_exporter` / `snmp_exporter` on targets) | Metrics, dashboards, alerts (PromQL) | **Default** recommendation for homelab. Aligns with [SNMP_LAB_TARGETS.md](../private.example/homelab/SNMP_LAB_TARGETS.md) “production monitoring” narrative. | -| **B** | **Grafana** + **InfluxDB** (+ **Telegraf** collectors) | Metrics if you prefer InfluxQL/Flux | Valid alternative to Prometheus TSDB; slightly different ops model. Pick **one** TSDB pillar (Prometheus **or** Influx), not both, unless you have a clear split (e.g. SNMP in Prometheus, app metrics in Influx). | -| **C** | **Promtail** + **Loki** + **Grafana** | Log aggregation, lower footprint than ELK/Graylog | “LGTM” family; good for **JSON/syslog** shipping and Grafana Explore. | -| **D** | **Graylog** + **OpenSearch** | Full-text log search, streams, pipelines | **Graylog 5+** uses **OpenSearch** (not Elasticsearch) as default backend—check current Graylog docs. **Heavy:** plan **≥8 GB** for OpenSearch alone on small labs; prefer a **dedicated VM**. | -| **E** | **Wazuh** | Security posture, vulns, hardening | [LAB_OP_MINIMAL_CONTAINER_STACK.md](../ops/LAB_OP_MINIMAL_CONTAINER_STACK.md) §6 — complements metrics/logs; does not replace them. **NIST/CIS buyer vocabulary:** [WAZUH_NIST_CIS_LABOP_ALIGNMENT.md](../ops/inspirations/WAZUH_NIST_CIS_LABOP_ALIGNMENT.md). | -| **F** | **Traces / APM-class** (pick **one** initially) | Request flows, latency, service dependencies | **Dynatrace**-class depth without default SaaS spend: **SigNoz**, **Grafana Tempo** + **OpenTelemetry**, **Jaeger**, or light **Netdata**; **Dynatrace** itself = trial/enterprise SaaS — see **§3** links. Run on a **VM** after **A** + **C** are stable. | +| **A** | **Grafana** + **Prometheus** (+ `node_exporter` / `snmp_exporter` on targets) | Metrics, dashboards, alerts (PromQL) | **Live** in lab (local Grafana + Grafana Cloud via PDC for Prom/Loki/Tempo/Pyroscope). Default homelab metrics pillar. Aligns with [SNMP_LAB_TARGETS.md](../private.example/homelab/SNMP_LAB_TARGETS.md). | +| **B** | **Grafana** + **InfluxDB** (+ **Telegraf** collectors) | Metrics if you prefer InfluxQL/Flux | **Not** the active lab pillar. Valid alternative to Prometheus TSDB; pick **one** TSDB (Prometheus **or** Influx) unless you have a clear split. | +| **C** | **Loki** + **Grafana** (OTLP / collectors; Promtail optional) | Log aggregation, lower footprint than ELK/Graylog | **Live** — primary log pillar (LGTM / Grafana Cloud Loki). Syslog + filelog + OTLP app logs exercised end-to-end. See **§1.1**. | +| **D** | **Graylog** + **OpenSearch** | Full-text log search, streams, pipelines | **Not adopted.** No Graylog deploy. OpenSearch may appear in lab **only** as an OTel `elasticsearch` **receiver validation target** (cluster **metrics**), not as a log-search backend — see **§1.1**. | +| **E** | **Wazuh** | Security posture, vulns, hardening | Still optional / sequenced later. [LAB_OP_MINIMAL_CONTAINER_STACK.md](../ops/LAB_OP_MINIMAL_CONTAINER_STACK.md) §6. **NIST/CIS:** [WAZUH_NIST_CIS_LABOP_ALIGNMENT.md](../ops/inspirations/WAZUH_NIST_CIS_LABOP_ALIGNMENT.md). | +| **F** | **Traces / APM-class** (pick **one** initially) | Request flows, latency, service dependencies | **Partial:** Tempo + OTel collector path live; product emit shipped ([#1500](https://github.com/DataBoar/data-boar/issues/1500), [#1529](https://github.com/DataBoar/data-boar/issues/1529)). Broader APM still backlog. | **Not recommended on the same LAB-NODE-01 simultaneously:** Graylog + OpenSearch + full Prometheus + Loki + Wazuh + k3s + trace backend. Choose **A or B**, **C or D**, **E** when resources exist, **F** when traces justify the RAM (often a **separate** VM). +### 1.1 Lab ingest validation snapshot (2026-08-11→12) + +Operator lab session validated **native OpenTelemetry Collector Contrib receivers** with real traffic. Facts for this plan (not a production architecture change): + +| Receiver / path | Result | Interpretation | +| ---- | ------ | -------------- | +| `syslog`, `tcp_log`, `udp_log` | Exercised end-to-end | Useful for SIEM-style / journald→rsyslog→remote and generic line protocols | +| `splunk_hec`, `webhook_event` | Exercised end-to-end | HTTP ingest options; webhook needs an explicit `path` (root `/` → 404) | +| `kafka` (Redpanda in Compose) | Exercised end-to-end | **Redpanda is a receiver validation target only** — not a production Kafka adoption for lab-op | +| `elasticsearch` (OpenSearch in Compose) | Cluster scrape only | Receiver exposes **cluster health metrics** (`elasticsearch_cluster_health`); **does not** ingest log documents. **Not** Phase **D** / Graylog | +| `gelf` | **Unsupported** | No native `gelf` receiver in `otelcol-contrib` v0.158.0 — document as a limitation, do not invent a bridge here | +| Product OTLP logs → Loki | Proven | Evidence: `docs/ops/evidence/otel_1529_loggerprovider_loki_2026-08-12.json` | + +**Do not read Compose presence of Redpanda or OpenSearch as Phase D.** Phase **C** (Loki) remains the log pillar; Phase **D** stays optional and **not** selected. + +Doc refresh tracking: [#1542](https://github.com/DataBoar/data-boar/issues/1542). + --- ## 2. Product notes (operator choice) @@ -64,5 +81,8 @@ URLs, retention, LDAP, and LAN firewall rules belong in **`docs/private/homelab/ - **Sequencing spine (firewall → access → logs → Wazuh):** [PLAN_LAB_FIREWALL_ACCESS_AND_OBSERVABILITY.md](PLAN_LAB_FIREWALL_ACCESS_AND_OBSERVABILITY.md) ([pt-BR](PLAN_LAB_FIREWALL_ACCESS_AND_OBSERVABILITY.pt_BR.md)) — use when UniFi/L3 work is active **before** or **alongside** phases A–F below. - **Learning links (Grafana / Elastic stack / traces):** [LAB_OP_OBSERVABILITY_LEARNING_LINKS.md](../ops/inspirations/LAB_OP_OBSERVABILITY_LEARNING_LINKS.md) ([pt-BR](../ops/inspirations/LAB_OP_OBSERVABILITY_LEARNING_LINKS.pt_BR.md)). - **LAB_OP_MINIMAL_CONTAINER_STACK.md** §7 — short pointer here. -- **Reminder (when hardware allows):** deploy **syslog/logs** first (**Promtail + Loki + Grafana**, phase **C**); add **Wazuh** (phase **E**) on a **VM/tower** with enough RAM — minimal operator checklist: [OBSERVABILITY_SYSLOG_DETECTION_CHECKLIST.md](../private.example/homelab/OBSERVABILITY_SYSLOG_DETECTION_CHECKLIST.md) ([pt-BR](../private.example/homelab/OBSERVABILITY_SYSLOG_DETECTION_CHECKLIST.pt_BR.md)). +- **Reminder (when hardware allows):** keep **syslog/logs** on the Loki path (phase **C**) before adding **Wazuh** (phase **E**) on a **VM/tower** with enough RAM — minimal operator checklist: [OBSERVABILITY_SYSLOG_DETECTION_CHECKLIST.md](../private.example/homelab/OBSERVABILITY_SYSLOG_DETECTION_CHECKLIST.md) ([pt-BR](../private.example/homelab/OBSERVABILITY_SYSLOG_DETECTION_CHECKLIST.pt_BR.md)). +- **Status refresh (2026-08-12):** [#1542](https://github.com/DataBoar/data-boar/issues/1542) — Phases **A**+**C** live; **F** partial; **D** not adopted; native receiver inventory in **§1.1**. + +**State:** Active plan (operator-side stack). Product OTLP emit lives under [PLAN_DATABOAR_OTEL_INSTRUMENTATION.md](completed/PLAN_DATABOAR_OTEL_INSTRUMENTATION.md). Private runbooks stay in `docs/private/homelab/`. diff --git a/docs/plans/PLAN_LAB_OP_OBSERVABILITY_STACK.pt_BR.md b/docs/plans/PLAN_LAB_OP_OBSERVABILITY_STACK.pt_BR.md index 2789d9a32..947e20d3e 100644 --- a/docs/plans/PLAN_LAB_OP_OBSERVABILITY_STACK.pt_BR.md +++ b/docs/plans/PLAN_LAB_OP_OBSERVABILITY_STACK.pt_BR.md @@ -1,13 +1,13 @@ # Stack de observabilidade no lab-op — métricas, logs, dashboards (só plano) -**Status:** Active (Fase A + parte da F no ar no lab; emit do produto = [#1500](https://github.com/DataBoar/data-boar/issues/1500)) -**Date:** 2026-08-09 +**Status:** Active (Fases **A** + **C** no ar no lab; Fase **F** parcial; Fase **D** não adotada; emit do produto = [#1500](https://github.com/DataBoar/data-boar/issues/1500) + [#1529](https://github.com/DataBoar/data-boar/issues/1529)) +**Date:** 2026-08-12 **Authors:** Fabio Leitao **Priority:** H2 **English:** [PLAN_LAB_OP_OBSERVABILITY_STACK.md](PLAN_LAB_OP_OBSERVABILITY_STACK.md) -**Emit do produto (código neste repo):** [PLAN_DATABOAR_OTEL_INSTRUMENTATION.md](completed/PLAN_DATABOAR_OTEL_INSTRUMENTATION.md) — OTLP opt-in FastAPI/SQLAlchemy; não substitui o deploy do collector/Grafana no lado do operador. +**Emit do produto (código neste repo):** [PLAN_DATABOAR_OTEL_INSTRUMENTATION.md](completed/PLAN_DATABOAR_OTEL_INSTRUMENTATION.md) — OTLP opt-in FastAPI/SQLAlchemy (**traces + metrics + logs** via `LoggerProvider`, [#1529](https://github.com/DataBoar/data-boar/issues/1529)); não substitui o deploy do collector/Grafana no lado do operador. **Objetivo:** Ordenar instrumentação **opcional** do homelab — **Grafana**, bases de séries temporais, **centralização de logs** — sem bloquear desenvolvimento do Data Boar nem a validação **–1L**. **Sem** implementação neste repositório; o operador instala via Compose, Helm no k3s ou appliance na **lab-op** (LAB-NODE-01, LAB-NODE-02, VMs Proxmox). @@ -19,29 +19,46 @@ | Stack mínima de contentores (Podman + k3s opcional) | [LAB_OP_MINIMAL_CONTAINER_STACK.pt_BR.md](../ops/LAB_OP_MINIMAL_CONTAINER_STACK.pt_BR.md) §1–§3 | | Probes SNMP / firewall (opcional) | [SNMP_LAB_TARGETS.pt_BR.md](../private.example/homelab/SNMP_LAB_TARGETS.pt_BR.md) | -**Realidade de hardware:** LAB-NODE-01 com **≤16 GB RAM** não deve correr Prometheus + Loki + Graylog + OpenSearch + Wazuh + k3s **em simultâneo**. Escolha **um** caminho de métricas e **um** de logs; stacks pesados no **torre/VM Proxmox** quando existir. +**Realidade de hardware:** LAB-NODE-01 com **≤16 GB RAM** não deve rodar Prometheus + Loki + Graylog + OpenSearch + Wazuh + k3s **ao mesmo tempo**. Escolha **um** caminho de métricas e **um** de logs; stacks pesados no **torre/VM Proxmox** quando existir. --- ## 1. Sequência recomendada (leve → pesada) -| Fase | Stack | Função | Notas | -| ---- | ----- | ------ | ----- | -| **A** | **Grafana** + **Prometheus** (+ `node_exporter` / `snmp_exporter`) | Métricas, dashboards, alertas (PromQL) | **Recomendação predefinida** para homelab. Alinha com a narrativa de monitorização em [SNMP_LAB_TARGETS.pt_BR.md](../private.example/homelab/SNMP_LAB_TARGETS.pt_BR.md). | -| **B** | **Grafana** + **InfluxDB** (+ **Telegraf**) | Métricas se preferir InfluxQL/Flux | Alternativa válida ao TSDB do Prometheus; escolha **um** pilar TSDB (Prometheus **ou** Influx), salvo divisão explícita (ex.: SNMP no Prometheus, métricas de app no Influx). | -| **C** | **Promtail** + **Loki** + **Grafana** | Agregação de logs, pegada menor que ELK/Graylog | Família “LGTM”; bom para envio de **JSON/syslog** e Explore no Grafana. | -| **D** | **Graylog** + **OpenSearch** | Busca full-text, streams, pipelines | **Graylog 5+** usa **OpenSearch** por padrão — ver documentação atual; **Elasticsearch** já não é o default universal. **Pesado:** reserva **≥8 GB** só para OpenSearch em labs pequenos; preferir **VM dedicada**. | -| **E** | **Wazuh** | Postura de segurança, vulns, hardening | [LAB_OP_MINIMAL_CONTAINER_STACK.pt_BR.md](../ops/LAB_OP_MINIMAL_CONTAINER_STACK.pt_BR.md) §6 — complementa métricas/logs; não os substitui. **NIST/CIS:** [WAZUH_NIST_CIS_LABOP_ALIGNMENT.pt_BR.md](../ops/inspirations/WAZUH_NIST_CIS_LABOP_ALIGNMENT.pt_BR.md). | -| **F** | **Traces / estilo APM** (escolher **um** de início) | Fluxos de pedido, latência, dependências | Profundidade estilo **Dynatrace** sem SaaS por padrão: **SigNoz**, **Grafana Tempo** + **OpenTelemetry**, **Jaeger** ou **Netdata** leve; **Dynatrace** = trial/enterprise — ver **§3**. VM depois de **A** + **C** estáveis. | +| Fase | Stack | Função | Estado no lab (2026-08-12) / notas | +| ---- | ----- | ------ | ----- | +| **A** | **Grafana** + **Prometheus** (+ `node_exporter` / `snmp_exporter`) | Métricas, dashboards, alertas (PromQL) | **No ar** no lab (Grafana local + Grafana Cloud via PDC para Prom/Loki/Tempo/Pyroscope). Pilar de métricas padrão. Alinha com [SNMP_LAB_TARGETS.pt_BR.md](../private.example/homelab/SNMP_LAB_TARGETS.pt_BR.md). | +| **B** | **Grafana** + **InfluxDB** (+ **Telegraf**) | Métricas se preferir InfluxQL/Flux | **Não** é o pilar ativo do lab. Alternativa válida ao TSDB do Prometheus; escolha **um** pilar TSDB (Prometheus **ou** Influx), salvo divisão explícita. | +| **C** | **Loki** + **Grafana** (OTLP / collectors; Promtail opcional) | Agregação de logs, pegada menor que ELK/Graylog | **No ar** — pilar principal de logs (LGTM / Grafana Cloud Loki). Syslog + filelog + logs OTLP do app exercitados ponta a ponta. Ver **§1.1**. | +| **D** | **Graylog** + **OpenSearch** | Busca full-text, streams, pipelines | **Não adotada.** Sem deploy de Graylog. OpenSearch no lab só como **alvo de validação** do receiver OTel `elasticsearch` (métrica de cluster), não como backend de busca de log — ver **§1.1**. | +| **E** | **Wazuh** | Postura de segurança, vulns, hardening | Ainda opcional / sequenciado depois. [LAB_OP_MINIMAL_CONTAINER_STACK.pt_BR.md](../ops/LAB_OP_MINIMAL_CONTAINER_STACK.pt_BR.md) §6. **NIST/CIS:** [WAZUH_NIST_CIS_LABOP_ALIGNMENT.pt_BR.md](../ops/inspirations/WAZUH_NIST_CIS_LABOP_ALIGNMENT.pt_BR.md). | +| **F** | **Traces / estilo APM** (escolher **um** de início) | Fluxos de pedido, latência, dependências | **Parcial:** caminho Tempo + collector OTel no ar; emit do produto shipped ([#1500](https://github.com/DataBoar/data-boar/issues/1500), [#1529](https://github.com/DataBoar/data-boar/issues/1529)). APM mais amplo ainda em backlog. | -**Evitar no mesmo LAB-NODE-01 em simultâneo:** Graylog + OpenSearch + Prometheus completo + Loki + Wazuh + k3s + backend de traces. Escolha **A ou B**, **C ou D**, **E** quando houver recursos, **F** quando traces justificarem a RAM (muitas vezes **VM** à parte). +**Evitar no mesmo LAB-NODE-01 ao mesmo tempo:** Graylog + OpenSearch + Prometheus completo + Loki + Wazuh + k3s + backend de traces. Escolha **A ou B**, **C ou D**, **E** quando houver recursos, **F** quando traces justificarem a RAM (muitas vezes **VM** à parte). + +### 1.1 Snapshot de validação de ingestão no lab (2026-08-11→12) + +Sessão de lab do operador validou **receivers nativos do OpenTelemetry Collector Contrib** com tráfego real. Fatos para este plano (não é mudança de arquitetura de produção): + +| Receiver / caminho | Resultado | Interpretação | +| ---- | --------- | ------------- | +| `syslog`, `tcp_log`, `udp_log` | Exercitados ponta a ponta | Úteis para estilo SIEM / journald→rsyslog→remoto e protocolos de linha genéricos | +| `splunk_hec`, `webhook_event` | Exercitados ponta a ponta | Opções HTTP; webhook precisa de `path` explícito (`/` na raiz → 404) | +| `kafka` (Redpanda no Compose) | Exercitado ponta a ponta | **Redpanda é só alvo de validação do receiver** — não é adoção de Kafka de produção no lab-op | +| `elasticsearch` (OpenSearch no Compose) | Só scrape de cluster | O receiver expõe **métricas de saúde do cluster** (`elasticsearch_cluster_health`); **não** ingere documentos de log. **Não** é Fase **D** / Graylog | +| `gelf` | **Sem suporte** | Não há receiver nativo `gelf` no `otelcol-contrib` v0.158.0 — documentar como limitação; não inventar bridge aqui | +| Logs OTLP do produto → Loki | Provado | Evidência: `docs/ops/evidence/otel_1529_loggerprovider_loki_2026-08-12.json` | + +**Não leia a presença de Redpanda ou OpenSearch no Compose como Fase D.** A Fase **C** (Loki) continua sendo o pilar de log; a Fase **D** segue opcional e **não** selecionada. + +Acompanhamento da atualização do doc: [#1542](https://github.com/DataBoar/data-boar/issues/1542). --- ## 2. Notas de escolha de produto - **Grafana** costuma ser o hub de **visualização**; liga a Prometheus, InfluxDB, Loki e muitos outros. -- **Elasticsearch** vs **OpenSearch:** para **Graylog**, siga o backend suportado pela versão; não assuma tutoriais “ELK” genéricos sem verificar versão. +- **Elasticsearch** vs **OpenSearch:** para **Graylog**, siga o backend suportado pela versão; não assuma tutoriais “ELK” genéricos sem verificar versão. No lab atual, OpenSearch **não** implica Graylog. - **InfluxDB** 3.x vs 2.x: confirma imagem/docs ao copiar Compose — há mudanças entre majors. --- @@ -64,6 +81,7 @@ URLs, retenção, LDAP e regras de firewall na LAN ficam em **`docs/private/home - **Sequenciamento (firewall → acesso → logs → Wazuh):** [PLAN_LAB_FIREWALL_ACCESS_AND_OBSERVABILITY.pt_BR.md](PLAN_LAB_FIREWALL_ACCESS_AND_OBSERVABILITY.pt_BR.md) ([EN](PLAN_LAB_FIREWALL_ACCESS_AND_OBSERVABILITY.md)) — usar quando o trabalho UniFi/L3 estiver ativo **antes** ou **em paralelo** com as fases A–F abaixo. - **Links de aprendizado (Grafana / stack Elastic / traces):** [LAB_OP_OBSERVABILITY_LEARNING_LINKS.pt_BR.md](../ops/inspirations/LAB_OP_OBSERVABILITY_LEARNING_LINKS.pt_BR.md) ([EN](../ops/inspirations/LAB_OP_OBSERVABILITY_LEARNING_LINKS.md)). - **LAB_OP_MINIMAL_CONTAINER_STACK.pt_BR.md** §7 — ponteiro para este plano. -- **Lembrete (quando houver hardware):** primeiro **syslog/logs** (**Promtail + Loki + Grafana**, fase **C**); depois **Wazuh** (fase **E**) numa **VM/torre** com RAM suficiente — checklist mínimo: [OBSERVABILITY_SYSLOG_DETECTION_CHECKLIST.pt_BR.md](../private.example/homelab/OBSERVABILITY_SYSLOG_DETECTION_CHECKLIST.pt_BR.md) ([EN](../private.example/homelab/OBSERVABILITY_SYSLOG_DETECTION_CHECKLIST.md)). +- **Lembrete (quando houver hardware):** manter **syslog/logs** no caminho Loki (fase **C**) antes de adicionar **Wazuh** (fase **E**) numa **VM/torre** com RAM suficiente — checklist mínimo: [OBSERVABILITY_SYSLOG_DETECTION_CHECKLIST.pt_BR.md](../private.example/homelab/OBSERVABILITY_SYSLOG_DETECTION_CHECKLIST.pt_BR.md) ([EN](../private.example/homelab/OBSERVABILITY_SYSLOG_DETECTION_CHECKLIST.md)). +- **Atualização de status (2026-08-12):** [#1542](https://github.com/DataBoar/data-boar/issues/1542) — Fases **A**+**C** no ar; **F** parcial; **D** não adotada; inventário de receivers em **§1.1**. -**Estado:** ⬜ Só plano — sem código no repo até um runbook do operador justificar um PR pequeno (ex.: Compose de exemplo **sem segredos** em `docs/private.example/`). +**Estado:** Plano ativo (stack no lado do operador). Emit OTLP do produto em [PLAN_DATABOAR_OTEL_INSTRUMENTATION.md](completed/PLAN_DATABOAR_OTEL_INSTRUMENTATION.md). Runbooks privados em `docs/private/homelab/`.