Skip to content

Commit 6c24000

Browse files
GodloveDvsoch
authored andcommitted
Complete bootstrap modules (#134)
* adding some build modules and editing others * added docker build module page * bringing arch debootstrap and yum up to speed with format * added busybox and zypper * few last minute edits * accidentally overwrote build-docker.md, now fixed * corrections based on v's review * typo
1 parent 311cf00 commit 6c24000

10 files changed

Lines changed: 238 additions & 19 deletions
16 KB
Binary file not shown.

pages/docs/user-docs/build-arch.md

Lines changed: 18 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,25 @@
11
---
2-
title: Arch Linux Builds
2+
title: arch bootstrap module
33
sidebar: user_docs
4-
permalink: build-archlinux
4+
permalink: build-arch
55
folder: docs
66
toc: false
77
---
88

9+
This module allows you to build a Arch Linux based container.
10+
11+
{% include toc.html %}
12+
13+
## Overview
14+
Use the `arch` module to specify a base for an Arch Linux based container. Arch Linux uses the aptly named the `pacman` package manager (all puns intended).
15+
16+
## Keywords
17+
```
18+
Bootstrap: arch
19+
```
20+
The **Bootstrap** keyword is always mandatory. It describes the bootstrap module to use.
21+
922
The Arch Linux bootstrap module does not name any additional keywords at this time. By defining the `arch` module, you have essentially given all of the information necessary for that particular bootstrap module to build a core operating system.
23+
24+
## Notes
25+
Arch Linux is, by design, a very stripped down, light-weight OS. You may need to perform a fair amount of configuration to get a usable OS. Please refer to [this README.md](https://github.com/singularityware/singularity/blob/master/examples/arch/README.md) and the [Arch Linux example](https://github.com/singularityware/singularity/blob/master/examples/arch/Singularity) for more info.
Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
---
2+
title: busybox bootstrap module
3+
sidebar: user_docs
4+
permalink: build-busybox
5+
folder: docs
6+
toc: false
7+
---
8+
9+
This module allows you to build a container based on BusyBox.
10+
11+
{% include toc.html %}
12+
13+
## Overview
14+
Use the `busybox` module to specify a BusyBox base for container. You must also specify a URI for the mirror you would like to use.
15+
16+
## Keywords
17+
```
18+
Bootstrap: busybox
19+
```
20+
The **Bootstrap** keyword is always mandatory. It describes the bootstrap module to use.
21+
```
22+
MirrorURL: https://www.busybox.net/downloads/binaries/1.26.1-defconfig-multiarch/busybox-x86_64
23+
```
24+
The **MirrorURL** keyword is mandatory. It specifies a URL to use as a mirror when downloading the OS.
25+
26+
## Notes
27+
You can build a fully functional BusyBox container that only takes up ~600kB of disk space!

pages/docs/user-docs/build-debootstrap.md

Lines changed: 35 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,13 +1,43 @@
11
---
2-
title: Debootstrap Builds
2+
title: debootstrap bootstrap module
33
sidebar: user_docs
44
permalink: build-debootstrap
55
folder: docs
66
toc: false
77
---
88

9-
The Debian bootstrap module is a tool which is used specifically for bootstrapping distributions which utilize the `.deb` package format and `apt-get` repositories. This module will bootstrap any of the Debian and Ubuntu based distributions. When using the `debootstrap` module, the following keywords must also be defined:
9+
This module allows you to build a Debian/Ubuntu style container from a mirror URI.
10+
11+
{% include toc.html %}
12+
13+
## Overview
14+
Use the `debootstrap` module to specify a base for a Debian-like container. You must also specify the OS version and a URI for the mirror you would like to use.
15+
16+
## Keywords
17+
```
18+
Bootstrap: debootstrap
19+
```
20+
The **Bootstrap** keyword is always mandatory. It describes the bootstrap module to use.
21+
```
22+
OSVersion: xenial
23+
```
24+
The **OSVersion** keyword is mandatory. It specifies the OS version you would like to use. For Ubuntu you can use code words like `trusty` (14.04), `xenial` (16.04), and `yakkety` (17.04). For Debian you can use values like `stable`, `oldstable`, `testing`, and `unstable` or code words like `wheezy` (7), `jesse` (8), and `stretch` (9).
25+
```
26+
MirrorURL: http://us.archive.ubuntu.com/ubuntu/
27+
```
28+
The **MirrorURL** keyword is mandatory. It specifies a URL to use as a mirror when downloading the OS.
29+
```
30+
Include: somepackage
31+
```
32+
The **Include** keyword is optional. It allows you to install additional packages into the core operating system. It is a best practice to supply only the bare essentials such that the `%post` section has what it needs to properly complete the build.
33+
34+
## Notes
35+
In order to use the `debootstrap` build module, you must have `debootstrap` installed on your system. On Ubuntu you can install it like so:
36+
```
37+
$ sudo apt-get update && sudo apt-get install debootstrap
38+
```
39+
On CentOS you can install it from the epel repos like so:
40+
```
41+
$ sudo yum update && sudo yum install epel-release && sudo yum install debootstrap.noarch
42+
```
1043

11-
- **MirrorURL**: This is the location where the packages will be downloaded from. When bootstrapping different Debian based distributions of Linux, this will define which varient will be used (e.g. specifying a different URL can be the difference between Debian or Ubuntu).
12-
- **OSVersion**: This keyword must be defined as the alpha-character string associated with the version of the distribution you wish to use. For example, `trusty` or `stable`.
13-
- **Include**: As with the `yum` module, the `Include` keyword will install additional packages into the core operating system and the best practice is to supply only the bare essentials such that the `%inside` scriptlet has what it needs to properly completely the bootstrap.
Lines changed: 47 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,47 @@
1+
---
2+
title: docker bootstrap module
3+
sidebar: user_docs
4+
permalink: build-docker-module
5+
folder: docs
6+
toc: false
7+
---
8+
9+
This module allows you to build a container from image layers hosted on Docker Hub or another Docker registry.
10+
11+
{% include toc.html %}
12+
13+
## Overview
14+
Docker images are comprised of layers that are assembled at runtime to create an image. You can use Docker layers to create a base image, and then add your own custom software. For example, you might use Docker's Ubuntu image layers to create an Ubuntu Singularity container. You could do the same with CentOS, Debian, Arch, Suse, Alpine, BusyBox, etc.
15+
16+
Or maybe you want a container that already has software installed. For instance, maybe you want to build a container that uses CUDA and cuDNN to leverage the GPU, but you don't want to install from scratch. You can start with one of the `nvidia/cuda` containers and install your software on top of that.
17+
18+
Or perhaps you have already invested in Docker and created your own Docker containers. If so, you can seamlessly convert them to Singularity with the `docker` bootstrap module.
19+
20+
## Keywords
21+
```
22+
Bootstrap: docker
23+
```
24+
The **Bootstrap** keyword is always mandatory. It describes the bootstrap module to use.
25+
```
26+
From: <registry>/<namespace>/<container>:<tag>@<digest>
27+
```
28+
The **From** keyword is mandatory. It specifies the container to use as a base. `registry` is optional and defaults to `index.docker.io`. `namespace` is optional and defaults to `library`. This is the correct namespace to use for some official containers (ubuntu for example). `tag` is also optional and will default to `latest`
29+
30+
See [Singularity and Docker](docs-docker#how-do-i-specify-my-docker-image) for more detailed info on using Docker registries.
31+
```
32+
Registry: http://custom_registry
33+
```
34+
The **Registry** keyword is optional. It will default to `index.docker.io`.
35+
```
36+
IncludeCmd: yes
37+
```
38+
The **IncludeCmd** keyword is optional. If included, and if a `%runscript` is not specified, a Docker `CMD` will take precedence over `ENTRYPOINT` and will be used as a runscript. Note that the `IncludeCmd` keyword is considered valid if it is _not empty_! This means that `IncludeCmd: yes` and `IncludeCmd: no` are identical. In both cases the `IncludeCmd` keyword is not empty, so the Docker `CMD` will take precedence over an `ENTRYPOINT`.
39+
40+
See [Singularity and Docker](docs-docker#what-gets-used-as-the-runscript) for more info on order of operations for determining a runscript.
41+
42+
## Notes
43+
Docker containers are stored as a collection of tarballs called layers. When building from a Docker container the layers must be downloaded and then assembled in the proper order to produce a viable file system. Then the file system must be converted to squashfs or ext3 format.
44+
45+
Building from Docker Hub is not considered reproducible because if any of the layers of the image are changed, the container will change. If reproducibility is important to you, consider hosting a base container on Singularity Hub and building from it instead.
46+
47+
For detailed information about setting your build environment see [Build Customization](build-environment).
Lines changed: 18 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,13 +1,27 @@
11
---
2-
title: Local Image Builds
2+
title: localimage bootstrap module
33
sidebar: user_docs
44
permalink: build-localimage
55
folder: docs
66
toc: false
77
---
88

9-
This module allows you to bootstrap a container from an existing Singularity image on your host system. You can use an existing container image as your "base" and then customize it with a new `%post` `%environment`, `%runscript`, etc.
9+
This module allows you to build a container from an existing Singularity container on your host system. The name is somewhat misleading because your container can be in either image or directory format.
1010

11-
This module allows you to bootstrap a container from an existing Singularity image on your host system. You can use an existing container image as your "base," and then add customization. For example, you may have a different base image for several common operating systems, or for different versions of python, or for several different compilers. Instead of building these from scratch each time, you could start with the appropriate local base image and then customize the new image in `%post`, `%environment`, `%runscript`, etc.
11+
{% include toc.html %}
1212

13-
When bootstrapping from a local image, all previous definition files that led to the creation of the current image will be stored in a directory within the container called `/.singularity.d/bootstrap_history`. Singularity will also alert you if environment variables have been changed between the base image and the new image during bootstrap.
13+
## Overview
14+
You can use an existing container image as your "base," and then add customization. This allows you to build multiple images from the same starting point. For example, you may want to build several containers with the same custom python installation, the same custom compiler toolchain, or the same base MPI installation. Instead of building these from scratch each time, you could start with the appropriate local base container and then customize the new container in `%post`, `%environment`, `%runscript`, etc.
15+
16+
## Keywords
17+
```
18+
Bootstrap: localimage
19+
```
20+
The **Bootstrap** keyword is always mandatory. It describes the bootstrap module to use.
21+
```
22+
From: /path/to/container/file/or/directory
23+
```
24+
The **From** keyword is mandatory. It specifies the local container to use as a base.
25+
26+
## Notes
27+
When building from a local container, all previous definition files that led to the creation of the current container will be stored in a directory within the container called `/.singularity.d/bootstrap_history`. Singularity will also alert you if environment variables have been changed between the base image and the new image during bootstrap.

pages/docs/user-docs/build-shub.md

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
---
2+
title: shub bootstrap module
3+
sidebar: user_docs
4+
permalink: build-shub
5+
folder: docs
6+
toc: false
7+
---
8+
9+
This module allows you to build a container from a container hosted on Singularity Hub.
10+
11+
{% include toc.html %}
12+
13+
## Overview
14+
You can use an existing container on Singularity Hub as your "base," and then add customization. This allows you to build multiple images from the same starting point. For example, you may want to build several containers with the same custom python installation, the same custom compiler toolchain, or the same base MPI installation. Instead of building these from scratch each time, you could create a base container on Singularity Hub and then build new containers from that existing base container adding customizations in `%post`, `%environment`, `%runscript`, etc.
15+
16+
## Keywords
17+
```
18+
Bootstrap: shub
19+
```
20+
The **Bootstrap** keyword is always mandatory. It describes the bootstrap module to use.
21+
```
22+
From: shub://<registry>/<username>/<container-name>:<tag>@digest
23+
```
24+
The **From** keyword is mandatory. It specifies the container to use as a base. `registry` is optional and defaults to `singularity-hub.org`. `tag` and `digest` are also optional. `tag` defaults to `latest` and `digest` can be left blank if you want the latest build.
25+
26+
## Notes
27+
When bootstrapping from a Singularity Hub image, all previous definition files that led to the creation of the current image will be stored in a directory within the container called `/.singularity.d/bootstrap_history`. Singularity will also alert you if environment variables have been changed between the base image and the new image during bootstrap.

pages/docs/user-docs/build-yum.md

Lines changed: 32 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -1,16 +1,42 @@
11
---
2-
title: Yum Builds
2+
title: yum bootstrap module
33
sidebar: user_docs
44
permalink: build-yum
55
folder: docs
66
toc: false
77
---
88

9+
This module allows you to build a Red Hat/CentOS/Scientific Linux style container from a mirror URI.
910

10-
The YUM base uses YUM on the host system to bootstrap the core operating system that exists within the container. This module is applicable for bootstrapping distributions like Red Hat, Centos, and Scientific Linux. When using the `yum` bootstrap module, several other keywords may also be necessary to define:
11+
{% include toc.html %}
12+
13+
## Overview
14+
Use the `yum` module to specify a base for a CentOS-like container. You must also specify the URI for the mirror you would like to use.
15+
16+
## Keywords
17+
```
18+
Bootstrap: yum
19+
```
20+
The **Bootstrap** keyword is always mandatory. It describes the bootstrap module to use.
21+
```
22+
OSVersion: 7
23+
```
24+
The **OSVersion** keyword is optional. It specifies the OS version you would like to use. It is only required if you have specified a %{OSVERSION} variable in the `MirrorURL` keyword.
25+
```
26+
MirrorURL: http://mirror.centos.org/centos-%{OSVERSION}/%{OSVERSION}/os/$basearch/
27+
```
28+
The **MirrorURL** keyword is mandatory. It specifies the URL to use as a mirror to download the OS. If you define the `OSVersion` keyword, than you can use it in the URL as in the example above.
29+
```
30+
Include: yum
31+
```
32+
The **Include** keyword is optional. It allows you to install additional packages into the core operating system. It is a best practice to supply only the bare essentials such that the `%post` section has what it needs to properly complete the build. One common package you may want to install when using the `yum` build module is YUM itself.
33+
34+
## Notes
35+
There is a major limitation with using YUM to bootstrap a container. The RPM database that exists within the container will be created using the RPM library and Berkeley DB implementation that exists on the host system. If the RPM implementation inside the container is not compatible with the RPM database that was used to create the container, RPM and YUM commands inside the container may fail. This issue can be easily demonstrated by bootstrapping an older RHEL compatible image by a newer one (e.g. bootstrap a Centos 5 or 6 container from a Centos 7 host).
36+
37+
In order to use the `debootstrap` build module, you must have `yum` installed on your system. It may seem counter-intuitive to install YUM on a system that uses a different package manager, but you can do so. For instance, on Ubuntu you can install it like so:
38+
```
39+
$ sudo apt-get update && sudo apt-get install yum
40+
```
1141

12-
- **MirrorURL**: This is the location where the packages will be downloaded from. When bootstrapping different RHEL/YUM compatible distributions of Linux, this will define which variant will be used (e.g. the only difference in bootstrapping Centos from Scientific Linux is this line.
13-
- **OSVersion**: When using the `yum` bootstrap module, this keyword is conditional and required only if you have specified a %{OSVERSION} variable name in the `MirrorURL` keyword. If the `MirrorURL` definition does not have the %{OSVERSION} variable, `OSVersion` can be omitted from the header field.
14-
- **Include**: By default the core operating system is an extremely minimal base, which may or may not include the means to even install additional packages. The `Include` keyword should define any additional packages which should be used and installed as part of the core operating system bootstrap. The best practice is to keep this keyword usage as minimal as possible such that you can then use the `%inside` scriptlet (explained shortly) to do additional installations. One common package you may want to include here is `yum` itself.
1542

16-
Warning, there is a major limitation with using YUM to bootstrap a container and that is the RPM database that exists within the container will be created using the RPM library and Berkeley DB implementation that exists on the host system. If the RPM implementation inside the container is not compatible with the RPM database that was used to create the container, once the container has been created RPM and YUM commands inside the container may fail. This issue can be easily demonstrated by bootstrapping an older RHEL compatible image by a newer one (e.g. bootstrap a Centos 5 or 6 container from a Centos 7 host).
Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,33 @@
1+
---
2+
title: zypper bootstrap module
3+
sidebar: user_docs
4+
permalink: build-zypper
5+
folder: docs
6+
toc: false
7+
---
8+
9+
This module allows you to build a Suse style container from a mirror URI.
10+
11+
{% include toc.html %}
12+
13+
## Overview
14+
Use the `zypper` module to specify a base for a Suse-like container. You must also specify a URI for the mirror you would like to use.
15+
16+
## Keywords
17+
```
18+
Bootstrap: zypper
19+
```
20+
The **Bootstrap** keyword is always mandatory. It describes the bootstrap module to use.
21+
```
22+
OSVersion: 42.2
23+
```
24+
The **OSVersion** keyword is optional. It specifies the OS version you would like to use. It is only required if you have specified a %{OSVERSION} variable in the `MirrorURL` keyword.
25+
```
26+
MirrorURL: http://download.opensuse.org/distribution/leap/%{OSVERSION}/repo/oss/
27+
```
28+
The **MirrorURL** keyword is mandatory. It specifies a URL to use as a mirror when downloading the OS.
29+
```
30+
Include: somepackage
31+
```
32+
The **Include** keyword is optional. It allows you to install additional packages into the core operating system. It is a best practice to supply only the bare essentials such that the `%post` section has what it needs to properly complete the build. One common package you may want to install when using the `zypper` build module is zypper itself.
33+

pages/docs/user-docs/docs-recipes.md

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -46,9 +46,8 @@ Include: yum
4646
Each build base requires particular details during build time. You can read about them and see examples at the following links:
4747

4848
- [shub](/build-shub) (images hosted on Singularity Hub)
49-
- [docker](/build-docker) (images hosted on Docker Hub)
49+
- [docker](/build-docker-module) (images hosted on Docker Hub)
5050
- [localimage](/build-localimage) (images saved on your machine)
51-
- [self](/build-self) (a snapshot of your build system)
5251
- [yum](/build-yum) (yum based systems such as CentOS and Scientific Linux)
5352
- [debootstrap](/build-debootstrap) (apt based systems such as Debian and Ubuntu)
5453
- [arch](/build-arch) (Arch Linux)

0 commit comments

Comments
 (0)