docs: incident ingest webhook for external alerts - #491
Open
prabhatsharma wants to merge 3 commits into
Open
Conversation
Documents POST /api/v2/{org_id}/alerts/incidents/ingest, which accepts alerts
from systems outside OpenObserve and correlates them into incidents alongside
native alerts.
Covers the payload reference, how correlation actually decides (only `labels`
matters, and semantic field groups normalize vendor spellings so no per-vendor
adapter is needed), the alert-identity rules that govern whether a delivery
notifies, severity mapping, resolve semantics, and idempotency.
The Alertmanager section shows a payload transform rather than a bare
webhook_configs block — Alertmanager's native shape is rejected by this
endpoint, so pointing a webhook_config straight at it looks correct and 400s.
Also links the new page from the Incident Management overview, since that is
where someone looking for this will start.
prabhatsharma
force-pushed
the
feat/incident-webhook-ingest
branch
from
July 28, 2026 18:11
327f7e2 to
087f715
Compare
Follows the second review pass on openobserve#13501: - timestamp is microseconds and is now validated; document the rejection and why (a seconds value used to make the incident auto-resolve away silently). - external_url must be http(s). - An incident that also contains natively-evaluated alerts does not auto-close when its external alerts resolve. The page implied it would.
Without a dedup_key nothing is deduplicated. The page previously claimed deliveries fell back to deduplicating on (source, alert_name), which the implementation no longer does — and which would have collapsed two genuine firings of the same rule into one.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Documents the incident ingest webhook shipped in openobserve/openobserve#13501 (with openobserve/o2-enterprise#2287). Design rationale: openobserve/o2-enterprise#2286.
What's here
New page —
user-guide/analytics/incidents/incident-webhook.mdPOST /api/v2/{org_id}/alerts/incidents/ingestaccepts alerts from Alertmanager, Datadog, Grafana, or anything that can POST JSON, and correlates them into incidents alongside the alerts OpenObserve evaluates itself.Sections: endpoint and auth, a working curl, full payload reference, how correlation decides, alert identity and when a delivery notifies, severity mapping, resolve, idempotency, Alertmanager forwarding, viewing external alerts, UI entry point, errors.
Overview link — adds an "Alerts from external systems" subsection to the Incident Management page under Automatic alert correlation, since that's where someone looking for this starts.
Nav — new
.pagesinincidents/so the webhook page sits under Incident Management rather than falling to alphabetical order.Three things the page is deliberate about
labelsdrives correlation. Stated plainly and repeated, because it's the one thing that determines whether the feature works for a reader. It also explains that semantic field groups normalizeinstance/host/k8s.node.name, so nobody goes looking for a per-vendor adapter that doesn't exist.webhook_configs. Alertmanager's native payload is rejected by this endpoint. Awebhook_configssnippet would look authoritative and fail with a400— so the page shows thejqreshape and says why.O2_INCIDENTS_AUTO_RESOLVE_AFTER_MINUTES(3 days by default). Called out in an admonition rather than buried.Verification
mkdocs build --strictcould not run here — the repo venv is missinggit-revision-date-localized. Validated manually instead:index.md,../alerts/index.md).#payload-reference,#severity-mapping) match real headings..pagesfiles parse as YAML and reference only files that exist.Worth a strict build in CI to confirm.
Reviewer notes
The payload reference, severity table, and
actionvalues were taken from the implementation rather than the design doc —ExternalAlertPayload::validate,map_external_severity, andExternalIngestActioninsrc/config/src/meta/alerts/incidents.rs. Size limits (64 label entries, 1024 chars per key/value, 128-charsource, 512-charalert_name, 2048-charexternal_url) and the 30-minute dedup window are the real enforced values.