You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Singularity is good friends with Docker. You can do all of the following:
9
-
10
-
- Import a Docker image into a Singularity image
11
-
- Use Docker to generate a development environment with Singularity and/or Docker
12
-
- Bootstrap a Docker image as the base for a Singularity image
13
8
14
9
# Import a Docker image into a Singularity Image
15
10
16
11
The core of a Docker image is basically a compressed set of files, a set of `.tar.gz` that (if you look in your <ahref="http://stackoverflow.com/questions/19234831/where-are-docker-images-stored-on-the-host-machine"target="_blank">Docker image folder</a> on your host machine, you will see. We are going to use this local repository for this first set of methods.
17
12
18
-
## Use your local Docker cache
19
-
20
-
For the first methods below, you will access Docker images via the `Docker` command line tool, meaning using the Docker engine, and either images pulled from Docker Hub or your local cache.
21
13
22
-
### docker2singularity.sh: running on your machine
14
+
##Quick Start: Use the Docker Remote API
23
15
24
-
It so happens that Docker has an "export" command to pipe this data out, and Singularity has an "import" command to take them in. Thus, you can do a simple import of a Docker image into a Singularity command by doing:
16
+
### Import Docker to Singularity
17
+
The Docker engine communicates with the Docker Hub via the <ahref="https://docs.docker.com/engine/reference/api/docker_remote_api/"target="_blank">Docker Remote API</a>, and guess what, we can too! The easiest thing to do is create an image, and then pipe a Docker image directly into it from the Docker Registry. This first method does not require having Docker installed on your machine. Let's say that I want to bootstrap tensorflow from Docker. First I should create the tensorflow image:
25
18
26
19
```bash
27
-
# Here is the name of the Singularity image I will create
28
-
image=ubuntu.img
29
-
30
-
# Now I am creating it
31
-
sudo singularity create $image
32
-
33
-
# Now I am exporting a running Docker container into it via a pipe (|)
No bootstrap definition passed, updating container
45
+
Executing Prebootstrap module
46
+
Executing Postbootstrap module
47
+
Done.
35
48
```
36
49
37
-
Where `$container_id` is the id of a running container obtained with `docker ps`. However, there are subtle details like the environment and permissions that this method will miss. It's also the case that most Docker images don't run (and stay running) easily unless you do something like:
50
+
Now I can shell into it, and import tensorflow:
38
51
39
52
```bash
40
-
docker run -d $image tail -f /dev/null
53
+
$ singularity shell tensorflow.img
54
+
Singularity: Invoking an interactive shell within container...
55
+
56
+
Singularity.tensorflow.img> ls
57
+
Singularity.tensorflow.img> python
58
+
Python 2.7.6 (default, Jun 22 2015, 17:58:13)
59
+
[GCC 4.8.2] on linux2
60
+
Type "help", "copyright", "credits" or "license"for more information.
61
+
>>> import tensorflow
62
+
>>>
41
63
```
42
64
43
-
and so early on we created a <ahref="https://github.com/singularityware/docker2singularity/blob/master/docker2singularity.sh"target="_blank">docker2singularity.sh</a>, a script that you can download and run as follows:
65
+
### Use a Spec File
66
+
Do a barrel role! Use a spec file! Many times, you want to bootstrap an image, and then either change the `%runscript` or add additional software or commands in the `%post` section. To achieve this, you can create a specification file. Currently, these are distributed with the naming format `[myfile].def`, however (soon) we will use a standard name, `Singularity` so all specification files can be automatically found. Here is what the spec file would look like for tensorflow:
To produce a Singularity image of "ubuntu:latest" in the present working directory.
73
+
%runscript
74
+
75
+
exec /usr/bin/python "$@"
52
76
77
+
%post
53
78
54
-
### docker2singularity.sh: Dockerized
79
+
echo"Post install stuffs!"
80
+
```
55
81
56
-
We wrapped this entire process into a Docker container itself, which means that you can use a Docker container in a Docker container to export a Docker container into Singularity! Nuts. Full instructions <ahref="https://github.com/singularityware/docker2singularity"target="_blank"> are provided, however here is the gist:
82
+
The solution above would be ideal for saving the specification of an image to build at some runtime.
This is currently the recommended way to convert your Docker images to Singularity. An alternative method that does not require Docker (next) can be used to bootstrap Docker images that are in the `library` namespace of the main Docker registry, and other registries that (currently) do not require a token.
85
+
### Run a Singularity Shell from a Docker image
68
86
69
-
## Use the Docker Remote API
70
-
The Docker engine communicates with the Docker Hub via the <ahref="https://docs.docker.com/engine/reference/api/docker_remote_api/"target="_blank">Docker Remote API</a>, and guess what, we can too!
87
+
Finally, we can achieve a "shell" experience, meaning shelling into Docker image imported into Singularity. We do this by storing the entire image in a temporary location, and then running the same function. You would do something like this:
71
88
72
-
**Please note that this functionality is currently best suited for the [official-images](https://hub.docker.com/explore/) in the Docker registry. Each registry carries it's own permissions, and an image appearing in Docker Hub does not ensure that we can properly obtain it. See more details under troubleshooting below.**
89
+
```bash
90
+
sudo singularity shell docker://ubuntu:latest
91
+
```
73
92
74
-
This will hopefully eventually mean some kind of solution to import Docker containers without needing sudo, however this is still under development. For now, it means importing without needing the Docker engine.
75
93
76
-
### Bootstrap a Docker image to generate a Singularity one
94
+
##Detailed Start: Bootstrapping a Docker image
77
95
A common use case is to want to start with a Docker image, possibly add custom commands, and have a Singularity image when you finish. You can read a bit about <ahref="/bootstrap-image"target="_blank">bootstrapping here</a> to get a sense of adding the custom commands and software. To specify "Docker" as the build source, you simply need this header:
78
96
79
97
```bash
@@ -90,47 +108,63 @@ IncludeCmd: yes
90
108
- ubuntu:latest
91
109
- ubuntu
92
110
93
-
In the case of omitting the tag (latest) it is assumed that you want the latest image. In the case of omitting the namespace (library) it is assumed that you want the common namespace, library.
94
111
112
+
In the case of omitting the tag (latest) it is assumed that you want the latest image. In the case of omitting the namespace (library) it is assumed that you want the common namespace, library. If you have a reason to use the Docker Engine, we also have a method to do this. The benefit of this method would be that you could use an image built locally (in your local cache) that isn't on Docker Hub.
95
113
96
-
### Troubleshooting
97
-
Why won't my image on Docker hub work? There are a few reasons, and this is why we suggest the first method as the current best practice to convert from Docker to Singularity.
98
114
99
-
#### Your Image is Hosted on a Different Registry
100
-
Keep in mind that Docker Hub and the main Docker Registry are not one and the same - an image on Docker Hub may be hosted on another registry! When you run the docker engine (the `docker` command) it is using something called the Remote API, and this API takes care of the details of navigating registries. Since we are implementing this functionality without having the docker engine as a dependency, this means that our default (the docker registry at `registry-1.docker.io` may not in fact be the correct one for your image. For example, the tensorflow image is <ahref="https://hub.docker.com/r/tensorflow/tensorflow/"target="_blank">listed on Docker Hub</a>, however if you look a little deeper into the docs, we find that <ahref="https://www.tensorflow.org/versions/r0.11/get_started/os_setup.html#docker-installation"target="_blank">it's hosted at gcr.io</a>. Thus, if we want a simple way to bootstrap the tensorflow (cpu) Docker image:
115
+
### Using Docker Engine
101
116
102
-
```bash
103
-
Bootstrap: docker
104
-
From: tensorflow/tensorflow:latest
105
-
IncludeCmd: yes
106
-
Registry: gcr.io
107
-
Token: no
117
+
Here we will access Docker images via the `Docker` command line tool, meaning using the Docker engine. As is the Docker standard, the image is first looked for in your local cache, and if not found, is pulled from Docker Hub.
108
118
109
-
%runscript
110
-
111
-
exec /usr/bin/python "$@"
112
119
113
-
%post
120
+
#### docker2singularity.sh: Dockerized
114
121
115
-
chmod 755 /usr/lib/python2.7/dist-packages/.wh*
116
-
```
122
+
We wrapped this entire process into a Docker container itself, which means that you can use a Docker container in a Docker container to export a Docker container into Singularity! Nuts. Full instructions <ahref="https://github.com/singularityware/docker2singularity"target="_blank"> are provided, however here is the gist:
117
123
118
-
Notice that we are specifying a different registry, `gcr.io`, and that we don't need an auth token. If you do not specify this registry, or don't disable the token, the bootstrap will not work.
Your best bet is to start with a Docker [official-image](https://hub.docker.com/explore/) and then add software/commands to it via the `%post` tag, as these images are tested to work well with our Bootstrap spec. In the case that your image is on Docker Hub and the Bootstrap doesn't work, this is likely a permissions issue, and you should fall back to the stable use case with docker2singularity.
133
+
##### How does docker2singularity.sh work?
122
134
123
-
### Import Docker to Singularity without specfile
124
-
A quicker implementation of the above (meaning without a specfile) is to specify the remote Docker image with the import command. For example, to import the Docker image "ubuntu:latest" into my Singularity image "/tmp/Debian.img" I would do:
135
+
How did this come to be? It so happens that Docker has an "export" command to pipe this data out, and Singularity has an "import" command to take them in. Thus, you can do a simple import of a Docker image into a Singularity command by doing:
Where `$container_id` is the id of a running container obtained with `docker ps`. However, there are subtle details like the environment and permissions that this method will miss. It's also the case that most Docker images don't run (and stay running) easily unless you do something like:
149
+
150
+
```bash
151
+
docker run -d $image tail -f /dev/null
152
+
```
131
153
132
-
Finally, we can use the same Docker Remote API (it comes down to the same Python function) to achieve a "shell" experience, meaning shelling into Docker image imported into Singularity. We do this by storing the entire image in a temporary location, and then running the same function. You would do something like this:
154
+
Early on we created a <ahref="https://github.com/singularityware/docker2singularity/blob/master/docker2singularity.sh"target="_blank">docker2singularity.sh</a>, a script that you can download and run as follows:
To produce a Singularity image of "ubuntu:latest" in the present working directory.
163
+
164
+
165
+
## Troubleshooting
166
+
Why won't my image bootstrap work? If you can't find an answer on this site, please <ahref="https://www.github.com/singularityware/singularity/issues"target="_blank">ping us an issue</a>.
167
+
If you've found an answer and you'd like to see it on the site for others to benefit from, then post to us <ahref="https://www.github.com/singularityware/singularityware.github.io/issues"target="_blank">here</a>.
168
+
169
+
## Future
170
+
This entire process will hopefully change in two ways. First, we hope to collapse the image creation and bootstrapping, so you have the option to do them both in one full swing. Second, we hope to eventually figure out some kind of solution to import Docker containers without needing sudo.
0 commit comments