You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Once Singularity is installed in it's default configuration you may find that there is a SETUID component installed at `$PREFIX/libexec/singularity/sexec-suid`. The purpose of this is to do the require privilege escalation necessary for Singularity to operate properly. There are a few aspects of Singularity's functionality that require escalated privileges:
9
-
7
+
8
+
{% include toc.html %}
9
+
10
+
## Executable Permissions
11
+
Once Singularity is ready to go in your specified `$PREFIX` base (eg, such as `/usr/local`) you may find that there are several `SETUID` root components installed (at `$PREFIX/libexec/singularity/bin/`). Let’s take a look:
Each of the binaries is named accordingly to the action that it is suited for, and generally, each handles the required privilege escalation necessary for Singularity to operate. What specifically requires escalated privileges?
25
+
10
26
1. Mounting (and looping) the Singularity container image
11
27
2. Creation of the necessary namespaces in the kernel
12
28
3. Binding host paths into the container
29
+
30
+
Removing any of these SUID binaries or changing the permissions on them would cause Singularity to utilize the non-SUID workflows. What workflows are we talking about? We tricked you a bit with the command above by limiting to a particular permission - each file with `*-suid` also has a non-suid equivalent:
31
+
32
+
```bash
33
+
/usr/local/libexec/singularity/bin/create
34
+
/usr/local/libexec/singularity/bin/action
35
+
/usr/local/libexec/singularity/bin/copy
36
+
/usr/local/libexec/singularity/bin/mount
37
+
/usr/local/libexec/singularity/bin/import
38
+
/usr/local/libexec/singularity/bin/expand
39
+
/usr/local/libexec/singularity/bin/export
40
+
/usr/local/libexec/singularity/bin/bootstrap
41
+
```
42
+
43
+
While most of these workflows will not properly function without the SUID components, we have provided these fall back executables for sites that wish to limit the SETUID capabilities to the bare essentials/minimum. Under this case, only the `action-suid` would be required (this would still allow `shell`, `exec`, `test`, and `run`). You can find the setting for `allow suid` at the top of the `singularity.conf` file, which is typically located in `$PREFIX/etc/singularity/singularity.conf` or the `etc` directory of the base repo before you install it.
44
+
45
+
```
46
+
# ALLOW SETUID: [BOOL]
47
+
# DEFAULT: yes
48
+
# Should we allow users to utilize the setuid program flow within Singularity?
49
+
# note1: This is the default mode, and to utilize all features, this option
50
+
# will need to be enabled.
51
+
# note2: If this option is disabled, it will rely on the user namespace
52
+
# exclusively which has not been integrated equally between the different
53
+
# Linux distributions.
54
+
allow setuid = yes
55
+
```
56
+
57
+
58
+
## Can I install Singularity as a user?
59
+
If you have ever tried to install Singularity as a user, the install may appear to work, but it is these specific functions that require a root user that may lead to errors in this “local installation.” If we go back to the basic idea that a container requires some management of <ahref="https://en.wikipedia.org/wiki/Linux_namespaces"target="_blank">linux namespaces</a>, it will make sense why a root user is required. It is generally impossible to implement any kind of container system without this level of permissions.
60
+
61
+
62
+
## A Container Permissions Strategy
63
+
As a cluster admin, you want to set up a configuration that is customized for your cluster or shared resource. In the following paragraphs, we will elaborate on this container permissions strategy, giving detail about which users are allowed to run containers, along with image curation and ownership.
64
+
65
+
66
+
### Can I limit usage to specific users?
67
+
In the configuration file (when installed it is located at `$PREFIX/etc/singularity/singularity.conf`) you have complete control over limiting the usage of Singularity to a specific set of users:
68
+
69
+
70
+
```bash
71
+
# LIMIT CONTAINER OWNERS: [STRING]
72
+
# DEFAULT: NULL
73
+
# Only allow containers to be used that are owned by a given user. If this
74
+
# configuration is undefined (commented or set to NULL), all containers are
75
+
# allowed to be used. This feature only applies when Singularity is running in
# Only allow containers to be used that are located within an allowed path
82
+
# prefix. If this configuration is undefined (commented or set to NULL),
83
+
# containers will be allowed to run from anywhere on the file system. This
84
+
# feature only applies when Singularity is running in SUID mode and the user is
85
+
# non-root.
86
+
#limit container paths = /scratch, /tmp, /global
87
+
```
88
+
89
+
For example, if I were to uncomment out the last line of the first section, I could specify a set of usernames that are allowed to use, manage, create, and do anything and everything with containers. Any command issued to Singularity that does not come from one of these users would not be allowed to run.
90
+
91
+
92
+
### Can users have ownership of containers?
93
+
Remember that container ownership is akin to file ownership. If you set strict permissions on a container, the same rules will apply as would a file. This is only one of the many ways that Singularity allows you to control permissions.
94
+
95
+
96
+
## Debugging and Logging
97
+
Singularity offers a very comprehensive auditing mechanism via its debugging output that is printed to the stderr (in your terminal), and also the system log. For each command that is issued, it prints the UID, PID, and location of the command. For example, let’s see what happens if we shell into an image:
98
+
99
+
```
100
+
$ singularity shell nginx.img
101
+
```
102
+
103
+
We can then peek into the system log to see what was recorded:
104
+
105
+
```bash
106
+
cat /var/log/syslog
107
+
Jun 10 00:02:24 vanessa-ThinkPad-T460s kernel: [205148.055781] EXT4-fs (loop1): mounting ext3 file system using the ext4 subsystem
108
+
Jun 10 00:02:24 vanessa-ThinkPad-T460s kernel: [205148.056282] EXT4-fs (loop1): mounted filesystem with ordered data mode. Opts: errors=remount-ro
109
+
Jun 10 00:02:24 vanessa-ThinkPad-T460s Singularity: action-suid (U=1000,P=9673)> USER=vanessa, IMAGE='nginx.img', COMMAND='shell'
110
+
```
111
+
112
+
We can also add the `--debug` argument to the command itself to see verbose output to the terminal, without peeking into syslog (note this is extremely cut down for this example):
Not only do I see all of the configuration options that I (probably forgot about) previously set, I can trace the entire flow of Singularity from the first execution of an action (shell) to the final shell into the container. Also note that the first line shows the exact version of Singularity that I'm using, which has this format:
147
+
148
+
```
149
+
Singularity version: 2.3-master.g499419b
150
+
Singularity version: [version]-[branch].[commit]
151
+
# (The commit id only includes the first 8 characters)
152
+
```
153
+
and if you install from a release, you will likely see:
154
+
155
+
```
156
+
Singularity version: 2.3-dist
157
+
```
158
+
159
+
But not to get sidetracked, we want to look at permissions! Actually, the part that you care about is where the UID of 0 changes to 1000:
13
160
14
-
In general, it is impossible to implement a container system that employs the features that Singularity offers without requiring extended privileges, but if this is a concern to you, the SUID components can be disabled via either the configuration file, changing the physical permissions on the sexec-suid file, or just removing that file. Depending on the kernel you are using and what Singularity features you employ this may (or may not) be an option for you. But first a warning...
161
+
```
162
+
DEBUG [U=0,P=9718] singularity_priv_drop_perm() Dropping to group ID '1000'
163
+
DEBUG [U=0,P=9718] singularity_priv_drop_perm() Dropping real and effective privileges to GID = '1000'
164
+
DEBUG [U=0,P=9718] singularity_priv_drop_perm() Dropping real and effective privileges to UID = '1000'
165
+
DEBUG [U=1000,P=9718] singularity_priv_drop_perm() Confirming we have correct GID
166
+
DEBUG [U=1000,P=9718] singularity_priv_drop_perm() Confirming we have correct UID
167
+
DEBUG [U=1000,P=9718] singularity_priv_drop_perm() Setting NO_NEW_PRIVS to prevent future privilege escalations.
168
+
```
169
+
170
+
In the above output you can see that we escalate permissions to properly mount, and once this has been done, we immediately drop permissions back to the calling user. We recommend that you try a `--debug` command locally on your workstation to see the entire pathway that Singularity follows for securely shell-ing into an image.
15
171
16
-
Many people (ignorantly) claim that the 'user namespace' will solve all of the implementation problems with unprivileged containers. While it does solve some, it is currently feature limited. With time this may change, but even on kernels that have a reasonable feature list implemented, it is known to be very buggy and cause kernel panics. Additionally very few distribution vendors are shipping supported kernels that include this feature. For example, Red Hat considers this a "technology preview" and is only available via a system modification, while other kernels enable it and have been trying to keep up with the bugs it has caused. But, even in it's most stable form, the user namespace does not completely alleviate the necessity of privilege escalation unless you also give up the desire to support images (#1 above).
17
172
173
+
## Why the user namespace isn't the answer to life, everything, and container permissions
174
+
Many people (possibly ignorantly) claim that the "user namespace" will solve all of the implementation problems with unprivileged containers. While it does solve some, it is currently feature limited. With time this may change, but even on kernels that have a reasonable feature list implemented, it is known to be very buggy and cause kernel panics. Additionally very few distribution vendors are shipping supported kernels that include this feature. For example, Red Hat considers this a "technology preview" and is only available via a system modification, while other kernels enable it and have been trying to keep up with the bugs it has caused. But, even in it's most stable form, the user namespace does not completely alleviate the necessity of privilege escalation unless you also give up the desire to support images (#1 above).
175
+
176
+
18
177
### How do other container solutions do it?
19
-
Docker and the like implement a root owned daemon to control the bring up, teardown, and functions of the containers. Users have the ability to control the daemon via a socket (either a UNIX domain socket or network socket). Allowing users to control a root owned daemon process which has the ability to assign network addresses, bind file systems, spawn other scripts and tools, is a large problem to solve and one of the reasons why Docker is not typically used on multi-tenant HPC resources.
178
+
Docker and other container solutions use a root owned daemon to control the bring up, teardown, and functions of the containers. Users have the ability to control the daemon via a socket (either a UNIX domain socket or network socket). Allowing users to control a root owned daemon process which has the ability to assign network addresses, bind file systems, spawn other scripts and tools, is a large problem to solve and one of the reasons why Docker is not typically used on multi-tenant HPC resources.
20
179
180
+
21
181
### Security mitigations
22
182
SUID programs are common targets for attackers because they provide a direct mechanism to gain privileged command execution. These are some of the baseline security mitigations for Singularity:
23
-
183
+
24
184
1. Keep the escalated bits within the code as simple and transparent so it can be easily audit-able
25
185
2. Check the return value of every system call, command, and check and bomb out early if anything looks weird
26
186
3. Make sure that proper permissions of files and directories are owned by root (e.g. the config must be owned by root to work)
@@ -31,27 +191,4 @@ SUID programs are common targets for attackers because they provide a direct mec
31
191
8. Limit all user actions within the container to that single user (disable escalation of privileges within a container)
32
192
9. Even though the user owns the image, it utilizes a POSIX like file system inside so files inside the container owned by root can only be modified by root
33
193
34
-
Additionally Singularity offers a very comprehensive auditing mechanism within it's debugging output by printing UID, PID, and location of every call it is making. For example:
DEBUG [U=0,P=33160] privilege.c:179:singularity_priv_drop() : Dropping privileges to UID=1000, GID=1000
42
-
DEBUG [U=1000,P=33160] privilege.c:191:singularity_priv_drop() : Confirming we have correct UID/GID
43
-
...
44
-
```
45
-
46
-
In the above output you can see that we are starting as UID 1000 (U=1000) and PID 33160 and we are escalating privileges. Once privileges have been increased, Singularity can properly mount /tmp and then it immediately drops permissions back to the calling user.
47
-
48
-
For comparison, the below output is when being called with the user namespace. Notice that I am not able to use the Singularity image format, and instead I am referencing a raw directory which contains the contents of the Singularity image:
49
-
50
-
```
51
-
$ singularity --debug shell -u /tmp/Centos7/
52
-
...
53
-
DEBUG [U=1000,P=111121] privilege.c:142:singularity_priv_escalate(): Not escalating privileges, user namespace enabled
DEBUG [U=1000,P=111121] privilege.c:169:singularity_priv_drop() : Not dropping privileges, user namespace enabled
56
-
...
57
-
```
194
+
If you have questions about security, or want to know more about a particular issue, please <ahref="https://www.github.com/singularityware/singularityware.github.io/issues"target="_blank">open an issue</a>, and we will help to answer your question right away.
0 commit comments