Skip to content

Run the dashboard's per-environment tests as a matrix job (26 min -> ~7) #122

Description

@gaurav

The dashboard workflow runs the validation suite against each environment in sequence, and that loop is essentially the entire runtime. From the most recent run (26 minutes end to end):

25m 48s   Run the test suite against every target
    30s   everything else — checkout, report generation, website build, deploy

Six targets at roughly four minutes each, one after another, on one runner. Splitting them into a matrix job — one target per runner, each uploading its raw/$t.jsonl as an artifact, with a single dependent job that downloads them all, generates the report and deploys — would bring the wall clock down to about the slowest single target plus overhead: roughly 6-8 minutes instead of 26.

dashboard.yaml already carries a ponytail: note anticipating this ("sequential loop; split into a matrix job with artifact merging if total runtime ever approaches the 6h job limit"). The 6h limit is not the reason to do it — the feedback loop is. A four-hour-old dashboard is fine; waiting 26 minutes to see whether a change to the report generator worked is not.

Points to get right:

  • The matrix must be built from targets.ini, not hardcoded: read_targets() is the single source of truth as of Replace the website with a daily cross-environment test dashboard built around promotion drift #120. A fromJSON matrix fed by a small setup job that prints the target list would keep that property.
  • continue-on-error per target, since a nonzero pytest exit is normal here — the report is the artifact.
  • Only the final job should touch gh-pages; it force-pushes as a single commit.
  • Keep timeout 45m per target, which caps a hung environment.

Deliberately not done in #120: it restructures the job graph and the deploy step, and it cannot be validated without a full run, so it wants its own review.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions