Skip to content

Fix/cross year second pass issues #100

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
johncmerfeld merged 3 commits into from
Jul 16, 2026
Merged

Fix/cross year second pass issues #100

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
ba10b1e
Run second pass if num_unmatched_students in None
theokaufman Jul 15, 2026
c80adfd
conditionally enforce cross year match rate threshold based on first …
theokaufman Jul 16, 2026
3477e4b
remove try/finally from cross_year_pass
theokaufman Jul 16, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 11 additions & 6 deletions executor/executor/executor.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -458,7 +458,8 @@ def orchestrate_earthmover(self):
# if we're here, the first pass of Earthmover was successful
if (self.send_to_ods # i.e. we've only tried matching this year's students so far
and self.cross_year_match_available # and we have access to EDU
and self.num_unmatched_students > 0 # and there are unmatched students from the first pass
and (self.num_unmatched_students is None # and either there were no matches at all
or self.num_unmatched_students > 0) # or there are any remaining unmatched students
):
# then take a second pass with the cross-year roster from EDU
# and thus produce a second output set to be sideloaded
Expand Down Expand Up @@ -539,10 +540,13 @@ def cross_year_pass(self, primary):
self.upload_artifact(artifact.CROSS_YEAR_ROSTER)
os.environ["EDFI_ROSTER_FILE"] = os.path.abspath(config.CROSS_YEAR_ROSTER_PATH)

# Boolean to capture whether our first run met the match rate threshold
met_initial_threshold = self.highest_match_rate >= config.REQUIRED_ID_MATCH_RATE

# If we hit the required match rate on the first pass, constrain to the ID column that won.
# Otherwise, we run again and check against all ID types.
# The bundle always appends studentUniqueId internally, so we pass an empty list when that's what won.
if self.highest_match_rate >= config.REQUIRED_ID_MATCH_RATE:
if met_initial_threshold:

# use only the students who failed to match the primary ID from the first run
unmatched_path = os.path.join(first_run_output_dir, os.path.basename(artifact.UNMATCHED_STUDENTS.path))
Expand All @@ -566,10 +570,11 @@ def cross_year_pass(self, primary):
self.earthmover_run(artifact.EM_RESULTS_X_YEAR.path)
artifact.EM_RESULTS_X_YEAR.needs_upload = True
self.upload_artifact(artifact.EM_RESULTS_X_YEAR)
# Record our new second pass match rate.
# If we don't meet the threshold, halt!
self.record_highest_match_rate()
self.enforce_match_threshold()

if not met_initial_threshold:
# Only enforce our match threshold if the initial run did not hit
self.record_highest_match_rate()
self.enforce_match_threshold()

self.logger.info(f"cross-year pass: match_rates: {load_match_rates()}")
count = count_unmatched_students()
Expand Down