fix: stop retries resending an acknowledged MT result

[AliAlfaifi]

Summary

Follow-up to #325. A retry could still send a training-stage result that another attempt had already got acknowledged.

What #325 got wrong

The send guard skips a result already known to have been sent. Retries were made to bypass it, so that the retries of an unacknowledged result would not be turned into no-ops by the cache entry that attempt had just written. But the bypass was unconditional — it also ignored the acknowledged entry, so a retry chain already in flight kept sending after a concurrent attempt succeeded.

# before — a retry ignores the cache entirely
if not getattr(getattr(current_task, "request", None), "retries", 0) and cache.get(cache_key):
    return

Observed in production

Within minutes of the #325 rollout, one (learner, course) pair:

22:17:28,206  Failed  responseCode 110
22:17:28,478  Called  responseCode 100   <- acknowledged, 272ms later
22:17:45,126  Failed  responseCode 110   <- retries of the attempt that failed first
22:18:01,983  Failed  responseCode 110
22:18:06,796  Failed  responseCode 110

Three surplus calls after the result was already accepted. Bounded by max_retries=3, so it is a leak rather than a regression of the 121× duplicate storm #325 fixed — but it defeats the deduplication for exactly the pairs that succeed.

The change

Store the outcome in the cache entry rather than a bare flag, and let the retry bypass apply only to the unacknowledged state:

cached_result = cache.get(cache_key)
retrying = bool(getattr(getattr(current_task, "request", None), "retries", 0))

if cached_result == MT_ACKNOWLEDGED or (cached_result and not retrying):
    return

state	first attempt	retry
acknowledged	skip	skip (was: send)
unacknowledged	skip	send
absent	send	send

Testing

• New case test_retry_is_skipped_once_acknowledged; test_retry_is_not_skipped_by_the_cache updated to seed the unacknowledged state so it still proves retries are not suppressed.
• 633 passed, coverage 96% (--fail-under=93), pylint 10.00/10, pycodestyle, isort and flake8 --max-complexity 10 all clean.

🤖 Generated with Claude Code