ci: third-party artifact downloads fail intermittently, reddening unrelated PRs

[jiwenc-nv]

CI fails intermittently on third-party artifact downloads, unrelated to the changes under test. Measured over one afternoon on a 14-PR stack: 22 failed Build Ubuntu runs, 48 failed jobs.

Job	Count	Cause
build-ubuntu	18	Couldn't download depthai-device-fwp.tar.xz
build-ubuntu	3	Couldn't download depthai-bootloader-fwp.tar.xz
build-ubuntu	5	Failed to download CloudXR Runtime SDK (NGC 404)
wheel-gate	22	aggregator only — mirrors build-ubuntu
test-teleop-ros2	observed	CloudXR SDK 404, exit 4

It hits main too: Build Ubuntu failed on c5fe6624c with the CloudXR error, green either side.

Three things make one flaky download expensive:

1. fail-fast: true on pull requests (build-ubuntu.yml:36) turns a single 404 into 1 failure plus 11 cancelled jobs and a red wheel-gate.
2. curl --retry does not retry 4xx. scripts/download_cloudxr_runtime_sdk.sh:105 passes --retry 3 --retry-delay 5, inert against the 404 actually seen.
3. DepthAI device binaries have no mirror or cache — fetched per job by DepthaiDownloader.cmake, so every matrix entry is an independent chance to fail.

Directions: cache or mirror both artifacts, retry 4xx explicitly, or let test-teleop-ros2 skip when the SDK is unavailable.

[jiwenc-nv]

Full sweep of every workflow run on a 14-PR stack, including jobs that failed inside runs still in progress (the earlier numbers only counted runs whose own conclusion was failure, which undercounts).

102 failed jobs. Not one classifies as anything but an artifact download.

Cause	Jobs
depthai-device-fwp download	18
depthai-bootloader-fwp download	7
CloudXR Runtime SDK (NGC 404)	10
wheel-gate — aggregator, no failure of its own	67
unclassified	0

35 genuine job failures: 31 in build-ubuntu, 4 in test-teleop-ros2. 13 of the 14 branches were hit at least once.

Two things the spread shows:

• No matrix entry is safe. Failures land across Release/Debug × 3.11/3.12/3.13 × x64/arm64 fairly evenly (worst: Release, 3.12, x64 at 6). It is the download, not a platform.
• The aggregator dominates the noise. 67 of 102 red jobs are wheel-gate reporting someone else's failure, so the red surface is ~3x the real fault count.

Nine reruns were needed to move three PRs to green; several PRs failed on consecutive attempts.

[jiwenc-nv]

One more observation, offered as a hypothesis rather than a finding.

The five PRs at the bottom of a 14-PR stack went green early, while nine runs were in flight, and have stayed green. The upper nine have cycled red→retry→red ever since, and the retries were being issued in bulk — up to nine PRs at once, each running two test-teleop-ros2 jobs, so roughly 18 simultaneous NGC fetches of the same tarball.

That is a plausible way to cause 404s rather than merely suffer them, and it would explain why failure rates rose as more of the stack became active. Retries here are now capped at two PRs per round to test it.

If concurrency is a factor, two mitigations are cheap and worth more than adding retries:

• cache the CloudXR Runtime SDK and the DepthAI device binaries per runner or in a shared artifact cache, so N concurrent jobs become one fetch;
• back off and retry on 4xx explicitly, since curl --retry ignores those.

Anyone reproducing this should compare failure rate against the number of concurrent workflow runs before assuming the upstream servers are solely at fault.

[jiwenc-nv]

Correction to the framing above: this is not only intermittent flake. It has sustained outage windows.

Measured today:

• Up to 07:04 UTC, test-teleop-ros2 passed on both matrix entries (3.12 x64 and arm64) for five consecutive PRs.
• From roughly 07:10 onward, the same job failed on all nine remaining PRs with curl: (22) 404 on the CloudXR Runtime SDK.
• A deliberate three-PR retry probe at 07:32 came back unanimous: all three failed again on the same 404.

So during a window, retrying is guaranteed to fail — every attempt costs a full CI run and returns nothing. Retry-based mitigation cannot fix this class; only caching or a mirror can.

That also changes the priority of the earlier suggestions. Caching the SDK per runner or in a shared artifact cache is worth more than any retry tuning, because it decouples the build from endpoint availability entirely. A build that has fetched the SDK once should never need it again for the same version.

A cheaper stopgap: make test-teleop-ros2 skip with a clear message when the SDK cannot be fetched, so an unavailable third-party endpoint does not redden PRs that have nothing to do with it.