fix(sdk): retry transient QDC 5xx during job status/log polling - #1373
Merged
Conversation
QDC's status and log-download endpoints intermittently return a 504 (and occasionally 500) with no fault on the caller's side. A 504's body isn't JSON, and the SDK's try_call() does a bare json.loads() on it without checking the status code first, so the crash usually surfaces as a JSONDecodeError rather than a "status code 5xx" exception. Across the Geniex Bench run that flagged this, this was the single largest cause of failed benchmark cells even though the underlying device job had already succeeded. Also raise the pending-job-quota submit retry budget from 1h to 2h: under a full matrix run the quota stays contested longer than an hour (observed elapsed-before-giveup up to ~54min, with the "N pending" count still fluctuating rather than permanently stuck). Signed-off-by: Mengsheng Wu <mengshengwu@outlook.com>
This PR's own test-sdk-qdc CI job hit exactly the bug this PR is fixing, one call site over: qdc_api.upload_file()'s chunked-upload POST returned a 500 mid-transfer with no retry, crashing the whole job submission. Route it through the same _call_with_retry used for the status/log calls. Signed-off-by: Mengsheng Wu <mengshengwu@outlook.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The
Geniex Benchworkflow has been failing on the large majority of its per-device/per-model jobs. Auditing a recent run (91 failing jobs) showed the dominant cause: QDC's job-status and log-download endpoints intermittently return a 504 (occasionally 500) with no fault on the caller's side, andsdk/benchmark/qdc/_qdc.pyhad no retry around those calls — a single blip crashed the whole benchmark cell even though the device job itself had already completed successfully.A 504's response body isn't JSON, and the QDC SDK's
try_call()does a barejson.loads()on it without checking the status code first, so the actual observed crash is usually aJSONDecodeErrorrather than an exception whose message mentions "status code 5xx" — that alone accounted for 38 of the 91 failing jobs in the audited run.This PR adds a short, bounded retry (
_call_with_retry) around the QDC calls in the polling/download/upload path that can hit this:get_job_status,get_job_log_upload_status,get_job_log_files,download_job_log_files, and (added after this PR's owntest-sdk-qdcCI job hit the identical bug one call site over, mid-review)upload_file. It mirrors the existing retry pattern already used for job submission (_submit_with_retry), just with a much shorter backoff since this is a transient blip, not capacity to wait out.Separately, the existing pending-job-quota submit retry (
_submit_with_retry) was giving up after a 1-hour budget; in the same audited run several jobs were still retrying with a fluctuating (not stuck) pending-job count right up to that limit. Raised the budget to 2 hours, which still leaves comfortable room under the job's 355-minute timeout.Not in scope: a handful of the 91 failures were genuine on-device functional failures unrelated to CI infra (e.g. a VLM
Multimodal generation failederror code onQwen2.5-VL-7B-qairt, and ageniex-benchnon-zero exit on Android for a couple of models) — those need separate model/plugin-level investigation.Test plan
JSONDecodeErrorandstatus code 5xxexceptions from the failing run's logs against_call_with_retrylocally — confirms both transient signatures are retried, a permanent error still raises immediately, and exhaustingTRANSIENT_RETRY_ATTEMPTSstill re-raises.test-sdk-qdc(linux, windows, android) on this PR itself exercises_qdc.pyagainst real QDC devices — one run hit a livestatus code 500during artifact upload before the second commit, confirming the fix is needed and covers this path too.