Add aria2 prefetch + glance-direct import fallback#1252
Merged
Conversation
The wait_for_image() polling loop could block the caller forever. Any status that was neither "active" nor "queued" -- including the terminal "killed"/"deleted" states -- fell through to an unconditional sleep(10) inside a "while True" with no overall deadline, and the except handler set exit_with_error but kept looping, so repeated get_image() failures spun indefinitely. A stalled import therefore had no way to fail fast. Make the wait bounded: - Accept an optional absolute deadline (a time.monotonic() timestamp). When omitted it is derived from self.CONF.import_timeout so any single caller stays bounded; import_image() will pass one shared deadline across all attempts. On the deadline elapsing, log and return None. - Handle the terminal statuses "killed", "deleted" and "pending_delete" explicitly by returning None instead of looping. - Bound consecutive SDK exceptions and return None once the limit is reached rather than spinning. wait_for_image() no longer sets exit_with_error; returning None is now its single "did not reach active" signal and the caller owns the error decision. The existing-image wait in process_image() is updated to set exit_with_error when the wait returns None, preserving today's failure reporting for that path. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Roger Luethi <luethi@osism.tech>
The per-version "checksum" field defined in the image YAML (and rendered
by OSISM's octavia template as "sha256:<hex>") was dropped during
version normalization and never reached import_image(), so it could not
be used to verify a downloaded image. The upcoming aria2 prefetch path
needs it to let aria2c verify and retry on corruption.
Copy "checksum" through the normalization loop, add an optional
"checksum" parameter to import_image(), and pass it from process_image()
via versions[version].get("checksum"). Add a module-level helper
checksum_to_aria2() that converts a "sha256:<hex>" or bare hex digest to
the "<algo>=<hex>" form aria2c's --checksum expects.
No behavioural change yet: the checksum is only threaded through; the
prefetch path that consumes it is added separately. Definitions using
version "latest" fetch their digest dynamically and are not covered here.
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Roger Luethi <luethi@osism.tech>
When Glance imports the octavia amphora image via the web-download
method, glance-api itself pulls the ~345 MB qcow2 from an external object
store over the public internet. When that fetch stalls, the import task
fails asynchronously, the image reverts to "queued", and the manager
eventually exits non-zero -- aborting the deploy before tempest runs.
Add an alternative import path that moves only the fragile internet
fetch off glance-api and onto aria2c running locally, then uploads the
downloaded file via the glance-direct import method. glance-direct runs
the same decompress/convert/store taskflow as web-download, so the
qcow2->raw conversion required by the Ceph/RBD store is preserved.
A new --prefetch option selects the behaviour, validated against the
allowed set:
- never -- today's behaviour, byte-for-byte;
- on-stuck -- (default) web-download first, fall back to aria2 +
glance-direct once if every attempt fails;
- always -- skip web-download and use aria2 + glance-direct directly.
The fallback does not reuse the image left behind by the failed
web-download (which may be queued, killed, deleted or still importing);
it best-effort deletes that leftover and stages a fresh image. A
synchronous web-download create/import error is treated as a failed
attempt so the fallback is still reached, and the fallback's own
create_image() is inside its try so a failure (for example a fixed
image id not yet freed) fails cleanly rather than raising.
Since the download runs on the manager (unlike web-download, which pulls
directly into glance), a HEAD Content-Length vs free-space preflight
aborts before writing when the target filesystem is too small, so a
prefetch cannot silently fill the manager disk; an actual out-of-space
error mid-download is still caught via aria2c's non-zero exit.
import_image() computes one overall deadline from --import-timeout and
shares it across all attempts, reserving headroom for the fallback. The
aria2c download is bounded by the remaining budget (subprocess timeout)
and the deadline is re-checked before staging; the staging upload itself
is a synchronous SDK call and is not bounded, but the import wait after
it is. aria2c is a required runtime dependency; if absent, _download()
logs an error and returns False rather than skipping silently. Distinct
"PREFETCH:" log lines are emitted for monitoring how often the fallback
fires and its outcome.
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Roger Luethi <luethi@osism.tech>
Describe the new --prefetch import path (never / on-stuck / always) and the --import-timeout budget, including the aria2c runtime dependency, the glance-direct requirement on the target cloud, and the manager-side temporary disk space the prefetch download needs. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Roger Luethi <luethi@osism.tech>
ideaship
marked this pull request as ready for review
July 14, 2026 04:44
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
A robust alternative to Glance's
web-downloadimport for images whose sourcedownload is slow or truncating: download the image locally with
aria2c(multi-connection range download + resume + retry), then import via the
glance-directmethod — which runs the same decompress → qcow2→raw convert →Ceph-store taskflow, so conversion and storage are unchanged.
Changes
wait_for_image()against hangs: overall deadline, terminal-statushandling, and SDK-error caps (also fixes a pre-existing forever-hang on the
web-download path).
checksumthrough import so the download can be verified.--prefetch {never,on-stuck,always}(defaulton-stuck) and--import-timeout; onon-stuck, fall back to aria2 + glance-direct after afailed web-download. Includes a free-space preflight (the download is new
manager-side disk) and
PREFETCH:telemetry lines.--prefetch neverreproduces today's behaviour byte-for-byte. 45 unit tests;flake8/black/mypy clean.
Related
🤖 Generated with Claude Code