Skip to content

docs: incident ingest webhook for external alerts - #491

Open
prabhatsharma wants to merge 3 commits into
mainfrom
feat/incident-webhook-ingest
Open

docs: incident ingest webhook for external alerts#491
prabhatsharma wants to merge 3 commits into
mainfrom
feat/incident-webhook-ingest

Conversation

@prabhatsharma

Copy link
Copy Markdown
Contributor

Documents the incident ingest webhook shipped in openobserve/openobserve#13501 (with openobserve/o2-enterprise#2287). Design rationale: openobserve/o2-enterprise#2286.

What's here

New pageuser-guide/analytics/incidents/incident-webhook.md

POST /api/v2/{org_id}/alerts/incidents/ingest accepts alerts from Alertmanager, Datadog, Grafana, or anything that can POST JSON, and correlates them into incidents alongside the alerts OpenObserve evaluates itself.

Sections: endpoint and auth, a working curl, full payload reference, how correlation decides, alert identity and when a delivery notifies, severity mapping, resolve, idempotency, Alertmanager forwarding, viewing external alerts, UI entry point, errors.

Overview link — adds an "Alerts from external systems" subsection to the Incident Management page under Automatic alert correlation, since that's where someone looking for this starts.

Nav — new .pages in incidents/ so the webhook page sits under Incident Management rather than falling to alphabetical order.

Three things the page is deliberate about

  • Only labels drives correlation. Stated plainly and repeated, because it's the one thing that determines whether the feature works for a reader. It also explains that semantic field groups normalize instance / host / k8s.node.name, so nobody goes looking for a per-vendor adapter that doesn't exist.
  • The Alertmanager example is a transform, not webhook_configs. Alertmanager's native payload is rejected by this endpoint. A webhook_configs snippet would look authoritative and fail with a 400 — so the page shows the jq reshape and says why.
  • Resolves carry a warning. Skipping them isn't a silent no-op: incidents stay open for O2_INCIDENTS_AUTO_RESOLVE_AFTER_MINUTES (3 days by default). Called out in an admonition rather than buried.

Verification

mkdocs build --strict could not run here — the repo venv is missing git-revision-date-localized. Validated manually instead:

  • Both relative links resolve to real files (index.md, ../alerts/index.md).
  • Both in-page anchors (#payload-reference, #severity-mapping) match real headings.
  • Both .pages files parse as YAML and reference only files that exist.

Worth a strict build in CI to confirm.

Reviewer notes

The payload reference, severity table, and action values were taken from the implementation rather than the design doc — ExternalAlertPayload::validate, map_external_severity, and ExternalIngestAction in src/config/src/meta/alerts/incidents.rs. Size limits (64 label entries, 1024 chars per key/value, 128-char source, 512-char alert_name, 2048-char external_url) and the 30-minute dedup window are the real enforced values.

Documents POST /api/v2/{org_id}/alerts/incidents/ingest, which accepts alerts
from systems outside OpenObserve and correlates them into incidents alongside
native alerts.

Covers the payload reference, how correlation actually decides (only `labels`
matters, and semantic field groups normalize vendor spellings so no per-vendor
adapter is needed), the alert-identity rules that govern whether a delivery
notifies, severity mapping, resolve semantics, and idempotency.

The Alertmanager section shows a payload transform rather than a bare
webhook_configs block — Alertmanager's native shape is rejected by this
endpoint, so pointing a webhook_config straight at it looks correct and 400s.

Also links the new page from the Incident Management overview, since that is
where someone looking for this will start.
@prabhatsharma
prabhatsharma force-pushed the feat/incident-webhook-ingest branch from 327f7e2 to 087f715 Compare July 28, 2026 18:11
Follows the second review pass on openobserve#13501:

- timestamp is microseconds and is now validated; document the rejection and
  why (a seconds value used to make the incident auto-resolve away silently).
- external_url must be http(s).
- An incident that also contains natively-evaluated alerts does not auto-close
  when its external alerts resolve. The page implied it would.
Without a dedup_key nothing is deduplicated. The page previously claimed
deliveries fell back to deduplicating on (source, alert_name), which the
implementation no longer does — and which would have collapsed two genuine
firings of the same rule into one.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant