|
| 1 | +--- |
| 2 | +title: "Singularity 2.3.2 Quick Fix Release" |
| 3 | +category: releases |
| 4 | +permalink: "release-2-3-2" |
| 5 | +targz: "2.3.2.tar.gz" |
| 6 | +--- |
| 7 | + |
| 8 | +<a target="_blank" href="https://github.com/singularityware/singularity/releases/tag/2.3.2">Release 2.3.2 </a> This dot release includes a fix for a change that Docker implemented to their registry RESTful API which broke compatibility with Singularity (among several other low minor fixes). |
| 9 | + |
| 10 | + |
| 11 | +## What happened? |
| 12 | + |
| 13 | + |
| 14 | +**Where does Docker image metadata come from?** |
| 15 | +The Docker Registry serves metadata about images via manifests, where each image has a manifest that tells Singularity important information like environment, labels, and entrypoints and running command. Importantly, the image manifest serves the actual layers that should be obtained to create the image. The manifests come in different flavors, or schema versions. Version 1 serves the primary load of metadata (labels, environment) while the first release of version 2 served layers, and had the addition of size. This addition made it possible to pull an image with Singularity and calculate the size on the fly for images generated with support for this manifest. |
| 16 | + |
| 17 | +Singularity had an old default to retrieve the version 2 (done by way of a header asking for it), and ask it for layers. If the remote registry could only offer version 1, we were still able to obtain a list of layers (under a different key, `fsLayers` instead of `layers`) and metadata via the older (schema 1) manifest. However, with the update, the API version 2 schema can now return a <a href="https://docs.docker.com/registry/spec/manifest-v2-2/#manifest-list"> list of manifests</a>. This meant that when we checked the response for the keys `layers` or `fslayers`, they wouldn't be found, becaues we needed to look under `manifests`, and then do a second call to retrieve the manifest of interest based on a system architecture and OS. This of course broke most `import`, `shell`, `pull`, `exec`, because all of these commands require retrieving the layers. |
| 18 | + |
| 19 | +## The Patch |
| 20 | +A super quick fix would have been to fall back to the version 1 manifest, always, but then we would lose the automatic calculation of size, which is important to many users. The "better" fix is obvious - the code needs to: |
| 21 | + |
| 22 | + - check for a `manifests` key |
| 23 | + - if found, select a default manifest to use |
| 24 | + - retrieve it, and continue! |
| 25 | + - if not found, fall back to version 1 |
| 26 | + |
| 27 | +This means that we've added environment variables `SINGULARITY_DOCKER_OS` and `SINGULARITY_DOCKER_ARCHITECTURE` to specify this choice, with defaults `linux` and `amd64`. This is a pretty exciting change, because it means down the line you (as a user!) can specify the specifics of the layers you want returned, given an image that has this metadata available. We can see the fix running as follows: |
| 28 | + |
| 29 | + |
| 30 | +``` |
| 31 | +DEBUG |
| 32 | +*** STARTING DOCKER IMPORT PYTHON **** |
| 33 | +``` |
| 34 | +This is the initialization of the Docker client, it's parsing the image name provided |
| 35 | +``` |
| 36 | +DEBUG Starting Docker IMPORT, includes env, runscript, and metadata. |
| 37 | +VERBOSE Docker image: ubuntu:14.04 |
| 38 | +VERBOSE2 Specified Docker ENTRYPOINT as %runscript. |
| 39 | +DEBUG Headers found: Content-Type,Accept |
| 40 | +VERBOSE Registry: index.docker.io |
| 41 | +VERBOSE Namespace: library |
| 42 | +VERBOSE Repo Name: ubuntu |
| 43 | +VERBOSE Repo Tag: 14.04 |
| 44 | +VERBOSE Version: None |
| 45 | +``` |
| 46 | +The initial poke to Docker hub asks for tags, to determine if we need some kind of token |
| 47 | +``` |
| 48 | +VERBOSE Obtaining tags: https://index.docker.io/v2/library/ubuntu/tags/list |
| 49 | +DEBUG GET https://index.docker.io/v2/library/ubuntu/tags/list |
| 50 | +``` |
| 51 | + |
| 52 | +401 means that we do, and with the response the API sends back the scope and other details we need to make to request it |
| 53 | + |
| 54 | +``` |
| 55 | +DEBUG Http Error with code 401 |
| 56 | +``` |
| 57 | + |
| 58 | +Here we are requesting it |
| 59 | +``` |
| 60 | +DEBUG GET https://auth.docker.io/token?service=registry.docker.io&expires_in=9000&scope=repository:library/ubuntu:pull |
| 61 | +DEBUG Headers found: Content-Type,Authorization,Accept |
| 62 | +``` |
| 63 | + |
| 64 | +And here is the new bit. Instead of blindly doing one call, we have a function to update two versions of the manifest - the older (with metadata) and some version of the newer (with layers and size) with a fallback to use the version 1 |
| 65 | +``` |
| 66 | +DEBUG Updating manifests. |
| 67 | +
|
| 68 | +# Here is version 2+ |
| 69 | +DEBUG MANIFEST (Primary): not found, making initial call. |
| 70 | +VERBOSE Obtaining manifest: https://index.docker.io/v2/library/busybox/manifests/latest |
| 71 | +DEBUG GET https://index.docker.io/v2/library/busybox/manifests/latest |
| 72 | +
|
| 73 | +# Here is version 1 (Metadata) |
| 74 | +DEBUG MANIFEST (Metadata): not found, making initial call. |
| 75 | +VERBOSE Obtaining manifest: https://index.docker.io/v2/library/busybox/manifests/latest |
| 76 | +DEBUG GET https://index.docker.io/v2/library/busybox/manifests/latest |
| 77 | +``` |
| 78 | +Notice that the two calls are the same - that's because the headers are actually different, to request different ones. |
| 79 | + |
| 80 | +And here is the (new) indication that we found a list |
| 81 | +``` |
| 82 | +DEBUG Image manifest version 2.2 list found. |
| 83 | +``` |
| 84 | +Here is the default architecture / os |
| 85 | +``` |
| 86 | +DEBUG Obtaining architecture: amd64, OS: linux |
| 87 | +``` |
| 88 | +And the specific call to get it |
| 89 | +``` |
| 90 | +VERBOSE Obtaining manifest: https://index.docker.io/v2/library/busybox/manifests/sha256:030fcb92e1487b18c974784dcc110a93147c9fc402188370fbfd17efabffc6af |
| 91 | +DEBUG GET https://index.docker.io/v2/library/busybox/manifests/sha256:030fcb92e1487b18c974784dcc110a93147c9fc402188370fbfd17efabffc6af |
| 92 | +``` |
| 93 | + |
| 94 | +With this fix - we can again use `pull`/`import`, etc, and also have a working `singularity pull docker://busybox` that estimates the size automatically. |
| 95 | + |
| 96 | +Please report any additional bugs to <a target="_blank" href="https://github.com/singularityware/singularity/issues/new">our issues board.</a> |
0 commit comments