Skip to content

feat: add data collection endpoint and rule modules AB#9312 - #463

Merged
kfaessen merged 1 commit into
developfrom
feature/AB9312-data-collection-rule-modules
Aug 19, 2026
Merged

kfaessen merged 1 commit into
developfrom
feature/AB9312-data-collection-rule-modules

Conversation

@kfaessen

Copy link
Copy Markdown
Contributor

Adds two modules under modules/azure/: data_collection_endpoint and data_collection_rule.

Azure Monitor retires the HTTP Data Collector API on 14 September 2026. Its replacement, the Logs Ingestion API, needs a data collection endpoint and a data collection rule, and neither exists in this library yet.

The rule module takes a list of custom streams. Each stream gets a stream declaration describing the incoming payload plus a data flow with a transform into an existing custom table. That way a caller can move a table off the Data Collector API while keeping its column names, so the workbooks and alert rules that query those columns keep working.

Both modules pass terraform fmt -check and validate/validate.sh against azurerm 3.117.1, and they are already running on the VWT Telecom dev environment with 20 streams.

Both modules are needed to send custom logs through the Logs Ingestion API, which replaces the HTTP Data Collector API that Azure Monitor retires on 14 September 2026.

The rule module takes a list of custom streams. Each stream gets a stream declaration describing the incoming payload and a data flow that transforms it into an existing custom table, so a caller can keep the column names of a table that was previously fed by the Data Collector API.

@nbroers-hanab nbroers-hanab left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good. Both modules follow the repo layout (main/variables/outputs, required_version ~> 1.12, azurerm ~> 3.117, backend + provider block so validate.sh can override them) and the resource labels match the naming used elsewhere in modules/azure. Lint and Validate are green, and I checked the deployed pair in the Telecom dev environment: dce-hti-int-platform-dev is Succeeded with both endpoints populated, and dcr-hti-int-platform-dev has 20 stream declarations plus 20 matching data flows into the logAnalytics destination, with no kind set, which is right for direct ingestion.

A few non-blocking notes for a follow-up:

  • The constraints in the streams description (name must start with Custom-, output stream must point at an existing table) and the allowed column types are not enforced by validation blocks, so a typo surfaces as an Azure error at apply instead of at plan. An empty streams list runs into the provider's minimum of one data_flow for the same reason.
  • data_flow is a list in the provider, and iterating a map gives lexicographic order, so adding a stream that sorts early reshuffles the indices in the plan output. Cosmetic only, the result is the same.
  • transform_kql and output_stream could be optional(string) (with transform_kql defaulting to "source") now that the modules require Terraform 1.12, which saves callers from passing explicit nulls.
  • Consumers still need a Monitoring Metrics Publisher assignment on the rule before they can send anything. Worth calling out in the consuming repo, since it is outside the scope of these modules.

@kfaessen
kfaessen merged commit 4ae36cd into develop Aug 19, 2026
2 checks passed
@kfaessen
kfaessen deleted the feature/AB9312-data-collection-rule-modules branch August 19, 2026 12:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants