Release/07 21 2026 by edandylytics · Pull Request #104 · edanalytics/runway

edandylytics · 2026-07-20T18:57:44Z

This is the PR for the 07/21/2026 release. It contains:

Runway now syncs tenants from UM
Runway will attempt cross-year ID matching if there were insufficient matches against the ODS roster:
- Fix/cross year second pass #87
- Fix/cross year second pass issues #100
Fix to an issue where the app sometimes did not receive error information from the Executor
Some security hardening:
- Redact sensitive headers from WAF logs #103
- Add safe-chain to App CI and create Executor CI #90
Various updates to dependencies:
Improvements to our deployment pipeline:
- Fix non-fatal build error during nx project graph resolution #91
- Dockefile/buildspec changes for faster buildtime; Some CF changes #80

Note that #79 and #82 are already merged to main, with a squash commit.

* run executor on large task when input files exceed size threshold Large (400+ MB) Star input files have caused the executor to run out of memory even on the medium task (EDFIAL-467). At run start, the app now sums the job's input file sizes from S3 and launches the large Fargate task when the total is at or above a threshold, defaulting to 100MB and configurable via LARGE_TASK_FILE_SIZE_THRESHOLD_MB. The threshold keys off file size rather than assessment type so any oversized job is routed to the bigger task, not just known-large templates. If a size lookup fails the run start fails loudly rather than falling back to a task size that might OOM. The corresponding bump to the large task definition's memory happens separately in CloudFormation. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * document unit testing alongside integration tests The repo just gained its first unit test, so the testing docs only covered integration tests. Record the convention: integration tests are the default, unit tests are for logic they can't practically reach (mocked AWS clients), specs live next to the code they test, and both suites run under api:test. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refine task size threshold config and drop selection logging Rename LARGE_TASK_FILE_SIZE_THRESHOLD_MB to ECS_FILE_SIZE_THRESHOLD_MB: shorter, and ECS anchors the context better than "large task" does. The config getter now returns null when the var is unset instead of baking in a default — AppConfigService should just retrieve config; calling code owns defaulting, so the 100MB fallback lives in ExecutorAwsService. Drop the task-size selection log line since file sizes and the chosen task are visible in AWS. Add a test covering a configured threshold overriding the default. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * use early return in threshold getter Separate "not configured" from "configured but unparseable" instead of funneling both through a NaN sentinel, which made the empty-string guard hard to follow. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * warn when ECS file size threshold is set but unparseable Per review, a misconfigured ECS_FILE_SIZE_THRESHOLD_MB shouldn't be silently ignored — log a warning so the typo is discoverable, but stay lenient and fall back to the default rather than failing run starts. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

* live large * noop * noop * noop * noop * comment to trigger build * again * run executor on large task when input files exceed size threshold Large (400+ MB) Star input files have caused the executor to run out of memory even on the medium task (EDFIAL-467). At run start, the app now sums the job's input file sizes from S3 and launches the large Fargate task when the total is at or above a threshold, defaulting to 100MB and configurable via LARGE_TASK_FILE_SIZE_THRESHOLD_MB. The threshold keys off file size rather than assessment type so any oversized job is routed to the bigger task, not just known-large templates. If a size lookup fails the run start fails loudly rather than falling back to a task size that might OOM. The corresponding bump to the large task definition's memory happens separately in CloudFormation. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * document unit testing alongside integration tests The repo just gained its first unit test, so the testing docs only covered integration tests. Record the convention: integration tests are the default, unit tests are for logic they can't practically reach (mocked AWS clients), specs live next to the code they test, and both suites run under api:test. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * refine task size threshold config and drop selection logging Rename LARGE_TASK_FILE_SIZE_THRESHOLD_MB to ECS_FILE_SIZE_THRESHOLD_MB: shorter, and ECS anchors the context better than "large task" does. The config getter now returns null when the var is unset instead of baking in a default — AppConfigService should just retrieve config; calling code owns defaulting, so the 100MB fallback lives in ExecutorAwsService. Drop the task-size selection log line since file sizes and the chosen task are visible in AWS. Add a test covering a configured threshold overriding the default. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Increased ram on large ecs size from 16384 to 30720 * bump version * update large memory size value --------- Co-authored-by: johncmerfeld <John.Merfeld@gmail.com> Co-authored-by: Andy Kitson <akitson@edanalytics.org> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

* migration * tests * add fields to created-modified * revisit * remove children * tests * add enum * update withoutaudit * um value * test * remove deletedon from withoutaudit * fixture values

Bumps [actions/checkout](https://github.com/actions/checkout) from 6.0.2 to 7.0.0. - [Release notes](https://github.com/actions/checkout/releases) - [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md) - [Commits](actions/checkout@de0fac2...9c091bb) --- updated-dependencies: - dependency-name: actions/checkout dependency-version: 7.0.0 dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Bumps [actions/setup-python](https://github.com/actions/setup-python) from 6.2.0 to 6.3.0. - [Release notes](https://github.com/actions/setup-python/releases) - [Commits](actions/setup-python@a309ff8...ece7cb0) --- updated-dependencies: - dependency-name: actions/setup-python dependency-version: 6.3.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Precreate Executor log groups and modify CloudWatch widget

* pinned * experimenting with snyk settings * move snyk file * move snyk file * remove snyk

* add safe-chain to app ci and create executor ci * test safe-chain CI

* Define earthmover_cmd method * Refactor em call in earthmover_ddeps * refactor em calls in earthmover_run * Call check_returncode from earthmover_cmd

* Constrain x_year pass IDs based on match rate * Enforce match rate only if x_year pass cannot run * Only use unmatched students in second pass if we hit match threshold * Enforce match threshold on second pass * clean up first run vars for readability

* migration * tests * add fields to created-modified * sync module * envs * alconfig * remove erroring line * test file * spec * add top level error boundary * revisit * wip * remove children * tests * add enum * global * fix test * config * tx * fix test * rename manager * remove duplicate const * remove managedby from tenant * remove managedby * dont delete non sync managed partners absent from UM * iterate over handlers * change al to um * simplify handlers * check sourcekey is sync manager * remove tx reference * remove newline * test * use aws secret * remove coordinator * move pg boss setvice * fetchaws * default cron * add tyr catch * remove cron from umconfig * default cron * missed al->um * flip condition for readability * make boss private * remove tenantsbypartner * missed rename in test * filter and check managedby * remove getpartners and gettenants * partner array * move apipartnercodes up * promise all for fetch only * add managedby to where clause * remove bypartner map * remove runsync * refactor tenantsToDelete for readability * pr comments * add test * remove catch in sync method * await getter * adjust pg boss service * add managedby on tenant updates

* remove check_returncode() from earthmover_cmd * add back explicit calls of check_returncode()

* Dockerfile changes to intoduce USER in the beginning and buildspec buildx changes for caching * new parameter CreateExecutorS3Bucket and cloudwatch dashboard change * hardcode cloudwatch widget to job executor medium * update version

* add Auth0ConfigSecret parameter * bump vesion

* removing parameter * rename secret name

* Run second pass if num_unmatched_students in None * conditionally enforce cross year match rate threshold based on first run match rate * remove try/finally from cross_year_pass

* logging tests * trying this out * testing limit * add byte limits to to_json * rm limit from format_exc * fix truncation * fix order of operations * rm test logging * rm logging * leave no trace --------- Co-authored-by: johncmerfeld <John.Merfeld@gmail.com>

* Redact sensitive headers from WAF logs * Bump CF template versions

snyk-io-us · 2026-07-20T18:58:06Z

✅ Snyk checks have passed. No issues have been found so far.

Status	Scan Engine	Critical	High	Medium	Low	Total (0)
✅	Open Source Security	0	0	0	0	0 issues
✅	Licenses	0	0	0	0	0 issues

💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse.

edandylytics · 2026-07-20T21:40:30Z

+partd==1.4.2
+propcache==0.5.2
+pyarrow==24.0.0
+pyasn1==0.6.3


@johncmerfeld @theokaufman Snyk is flagging vulnerabilities in pyasn1@0.6.3 that are fixed in 0.6.4. Could you take a look at whether we can upgrade? If it's straightforward, let's just take the version bump. Otherwise, let's figure out whether these impact Runway and should block the deploy.

Here are the vulnerabilities: https://security.snyk.io/package/pip/pyasn1/0.6.3

Definitely no problem to upgrade; I'll open a PR right now to unblock the release

zabdykadyrova

Looks good to me.

edandylytics and others added 21 commits June 12, 2026 16:55

precreate log groups and modify cloudwatch widget

cab3d84

version bump

560dfa5

EDFIAL-457 add migration (#73)

526b4bf

* migration * tests * add fields to created-modified * revisit * remove children * tests * add enum * update withoutaudit * um value * test * remove deletedon from withoutaudit * fixture values

Merge pull request #85 from edanalytics/feature/cloudwatch-widget-update

e9e44fa

Precreate Executor log groups and modify CloudWatch widget

Executor: pin package dependencies (#89)

3a78458

* pinned * experimenting with snyk settings * move snyk file * move snyk file * remove snyk

use absolute path to prevent build error (#91)

6894c71

Add safe-chain to App CI and create Executor CI (#90)

2f28b86

* add safe-chain to app ci and create executor ci * test safe-chain CI

earthmover_cmd refactor [EDFIAL-466] (#92)

62273bf

* Define earthmover_cmd method * Refactor em call in earthmover_ddeps * refactor em calls in earthmover_run * Call check_returncode from earthmover_cmd

Fix/cross year second pass (#87)

0012cdc

* Constrain x_year pass IDs based on match rate * Enforce match rate only if x_year pass cannot run * Only use unmatched students in second pass if we hit match threshold * Enforce match threshold on second pass * clean up first run vars for readability

Fix/return code (#96)

1d20cd8

* remove check_returncode() from earthmover_cmd * add back explicit calls of check_returncode()

Add Auth0ConfigSecret parameter (#97)

f5f9683

* add Auth0ConfigSecret parameter * bump vesion

Remove Auth0ConfigSecret parameter (#98)

9485276

* removing parameter * rename secret name

Fix/cross year second pass issues (#100)

640e164

* Run second pass if num_unmatched_students in None * conditionally enforce cross year match rate threshold based on first run match rate * remove try/finally from cross_year_pass

Redact sensitive headers from WAF logs (#103)

7b4bdeb

* Redact sensitive headers from WAF logs * Bump CF template versions

edandylytics commented Jul 20, 2026

View reviewed changes

johncmerfeld added 2 commits July 20, 2026 16:48

Executor: bump pyasn1 version (#105)

81baccd

Executor: bump setuptools version (#106)

4cb82a6

johncmerfeld approved these changes Jul 20, 2026

View reviewed changes

Merge branch 'main' into release/07-21-2026

d96fae2

edandylytics requested review from rtavernaea and zabdykadyrova July 21, 2026 15:36

edandylytics marked this pull request as ready for review July 21, 2026 15:58

zabdykadyrova approved these changes Jul 21, 2026

View reviewed changes

rtavernaea approved these changes Jul 21, 2026

View reviewed changes

edandylytics merged commit 28f5523 into main Jul 21, 2026
11 checks passed

edandylytics deleted the release/07-21-2026 branch July 21, 2026 21:00

Navigation Menu

Uh oh!

Release/07 21 2026 - #104

Release/07 21 2026#104
edandylytics merged 24 commits into
mainfrom
release/07-21-2026

edandylytics commented Jul 20, 2026 •

edited

Loading

Uh oh!

snyk-io-us Bot commented Jul 20, 2026 •

edited

Loading

Uh oh!

edandylytics Jul 20, 2026

Uh oh!

johncmerfeld Jul 20, 2026

Uh oh!

zabdykadyrova left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

Uh oh!

Conversation

edandylytics commented Jul 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

snyk-io-us Bot commented Jul 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Snyk checks have passed. No issues have been found so far.

Uh oh!

edandylytics Jul 20, 2026

Choose a reason for hiding this comment

Uh oh!

johncmerfeld Jul 20, 2026

Choose a reason for hiding this comment

Uh oh!

zabdykadyrova left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

edandylytics commented Jul 20, 2026 •

edited

Loading

snyk-io-us Bot commented Jul 20, 2026 •

edited

Loading