These processes currently run as manual stages, it would now be useful to have each module run as discrete stages for testing and configuration, as well as a unified scheduled pipeline. Proposal here is to add airflow.sh to this repo, which pulls all the required code and resources from the open web and organises into a single airflow DAG.
These processes currently run as manual stages, it would now be useful to have each module run as discrete stages for testing and configuration, as well as a unified scheduled pipeline. Proposal here is to add
airflow.shto this repo, which pulls all the required code and resources from the open web and organises into a single airflow DAG.