Hotfix: Replace logical_date with dag_run.start_date

[jayckaiser]

Hotfix: Replace logical_date with dag_run.start_date

Description & motivation

In current EdFiResourceDAG behavior, there is unexpected behavior that affects timestamps recorded in Stadium between scheduled and manual runs. Timestamps placed into the pull_date and pull_timestamp columns in raw.edfi3.{resource} and raw.edfi3._meta_change_version tables are derived from the logical_date attribute of a DAG Run. This variable reflects different things, depending on the type of run being completed:

• Scheduled Runs: Schedule-interval start-time (i.e., exactly 24 hours before the scheduled Execution Date of the DAG)
• Manual Runs: The actual timestamp of when the DAG was triggered

In practice, this means that if one runs the DAG manually, and then the DAG runs on its schedule later that night, the scheduled run timestamp will be recorded as occurring before the manual run. This means that if a given record is ingested to Stadium and then updated by that evening, the first version of the record will take precedence over the updated version by Stadium's staging dedupe rules.

This fix simply changes what variable is used as the timestamp record of the DAG run: the dag_run.start_time variable reflects the actual runtime of the DAG in all cases, never the start of the schedule interval. This change will prevent the misordering issue in almost all cases*.

PR Merge Priority:

• Low
• Medium
• High

This is a longstanding bug that was somehow never discovered previously (but it may be responsible for some of our historic sync-drift in SC). It is uncommon to need to manually rerun the EdFiResourceDAG, but we need to make sure that records are recorded in the correct timestamp-order in Snowflake. This issue was likely obscured through regular full-refreshes that have been completed in South Carolina and other Stadium implementations.

Changes to existing files:

• edu_edfi_airflow/callables/change_version.py: Retrieve dag_run.start_date instead of ds and ts, and convert to date and timestamp isoformat respectively.
• edu_edfi_airflow/callables/total_counts.py: Retrieve dag_run.start_date instead of ds and ts, and convert to date and timestamp isoformat respectively.
• edu_edfi_airflow/dags/edfi_resource_dag.py: Retrieve dag_run.start_date instead of ds_nodash and ts_nodash, and convert to ds_nodash and ts_nodash format respectively.

Tests and QC done:

This change has been tested successfully in SC dev, specifically for the copy_all_endpoints_to_snowflake and update_change_versions_in_snowflake tasks. I have not tested this change in the total-counts taskgroup (as these are not enabled in dev), but the logic is identical to the update-change-versions task.

Future ToDos & Questions:

*There is one case where misordering could still occur. When a task is cleared and rerun, the dag_run.start_date variable updates to that new task runtime. This is unexpected behavior, but I couldn't find a DagRun variable that represents actual DAG runtime that remains consistent across task reruns. If someone were to run the DAG, let either the copy_all_endpoints_into_snowflake or update_change_versions_in_snowflake tasks fail, then rerun the failed task after a subsequent run completes successfully, then we will still record timestamps in Stadium that do not reflect the original runtime of the run. This is an anti-pattern of this DAG, however (i.e., why rerun a task from the previous night when the subsequent run succeeded). I'd argue this is a reasonable concession, but if we can find a more consistent DagRun variable to use in place of start_date, that would remove any uncertainty.

Additionally, I did not make changes to any other helpers or DAGs that those used by EdFiResourceDAG. The EarthbeamDAG and its helpers will be deprecated in the near future.