feat(connectors): add Apache Airflow trigger connector

[avirajkhare00]

Description

Add an Apache Airflow connector so Iggy can drive event-driven DAGs via the Airflow REST API.

This is tracked on the connector ecosystem roadmap (#2753) under Workflow & Orchestration:

Target	Type	Priority	Comp	Suggested stack
Apache Airflow	Trigger	P4	1/4	reqwest (REST API)

Roadmap intent: Iggy sensor/trigger for event-driven DAGs. There is no Airflow sink/source (or trigger plugin) in-tree today, no dedicated implementation issue before this one, and no open PR claiming the work.

Motivation

• Airflow is the industry-standard workflow orchestrator; pairing it with Iggy enables stream-driven DAG triggers instead of polling-only sensors.
• Existing connectors cover DBs, search, lakehouse, and HTTP egress, but nothing in the workflow/orchestration category is implemented yet (Airflow, Temporal, Prefect, Dagster are all still roadmap-only).
• A first cut can likely reuse patterns from http_sink (auth, retries, batching) while specializing for Airflow's trigger/DAG-run APIs.

Suggested v1 scope (trigger)

• Consume messages from configured Iggy streams/topics
• Trigger DAG runs via Airflow REST API (e.g. create DAG run / trigger endpoint)
• Config: Airflow base URL, auth (token/basic), dag_id, optional conf/payload mapping from message body, retry policy
• Map transient HTTP errors (5xx, timeouts) to retry; map permanent client errors (auth, missing DAG) to non-retry
• open() connectivity check against Airflow health/version endpoint; structured logging of trigger outcomes
• Unit tests + example config under core/connectors/runtime/example_config/connectors/

Out of scope for v1 (unless someone wants to expand): full Airflow source (task/DAG state → Iggy), Prefect/Dagster parity, Airflow provider package on the Airflow side.

Affected area / component

Connectors

Proposed solution

New sink-style (or dedicated trigger) plugin, e.g. airflow_sink / airflow_trigger, under core/connectors/sinks/, built on iggy_connector_sdk and reqwest, following existing sink lifecycle (open / consume / close) and config conventions.

Alternatives considered

• http_sink only: works for a one-off webhook but does not encode Airflow auth, DAG-run semantics, conf mapping, or operator-friendly defaults.
• Airflow-side provider/sensor only: useful later, but does not give Iggy-native plugin packaging, runtime metrics, or the same ops model as other connectors.
• Wait for a generic "workflow trigger" abstraction: delays a concrete Airflow integration that the roadmap already calls out.

Contribution

• I'm willing to submit a pull request to implement this feature

Good first issue

• I think this could be a good first issue for a new contributor

[hfl0506]

Hi @avirajkhare00 Can I work on this issue if it still applicable ? Thank you

[avirajkhare00]

hey @hfl0506
there are a lot of connectors you can pick up from. please see this: #2753

I will be working on this one