[sclaw]

Closes #136.

Context

SEPEX submits AWS Batch jobs, then keeps each active job in memory while status updates arrive. When a job reaches a terminal status, the status handler closes it and calls RunFinished() to release its completion waiter. On API restart, recovery rebuilds active AWS Batch jobs from database records and checks their current AWS status.

What went wrong

New AWS Batch jobs increment wgRun during Create(), but recovered jobs did not increment it before becoming active. The first terminal update for a recovered job called RunFinished(), which decremented a zero-valued WaitGroup and crashed the API with sync: negative WaitGroup counter. Repeated restarts could reattach the same unfinished jobs and trigger the crash again.

Overlapping completion paths could also attempt cleanup more than once. A status update, recovery, and a dismissal can all lead to job closure.

Changes

• Initialize wgRun before adding a recovered AWS Batch job to the active-job map.
• Complete recovered jobs that are already terminal when recovery checks AWS.
• Make RunFinished() and Close() run once per AWS Batch job, so overlapping completion paths cannot decrement the counter or send cleanup twice.
• Complete the waiter after dismissal cleanup, consistent with terminal status handling.
• Add a regression test that calls RunFinished() concurrently for multiple jobs and verifies each waiter unblocks.

Verification

From api/: go test -race ./jobs -count=1 passed.

Live AWS Batch load and restart behavior was not tested; that still needs deployment-level verification.