adding docs for checks and daemons

vsoch · vsoch · commit e614168ee05b · 2017-07-26T16:22:50.000-04:00
diff --git a/_data/sidebars/admin_docs.yml b/_data/sidebars/admin_docs.yml
@@ -22,6 +22,10 @@ entries:
       url: /docs-config
       output: web, pdf
 
+    - title: Container Checks
+      url: /docs-admin-checks
+      output: web, pdf
+
     - title: Troubleshooting
       url: /docs-troubleshooting
       output: web, pdf
diff --git a/_data/sidebars/user_docs.yml b/_data/sidebars/user_docs.yml
@@ -31,6 +31,10 @@ entries:
       url: /image-content
       output: web, pdf
 
+    - title: Container Checks
+      url: /docs-user-checks
+      output: web, pdf
+
     - title: User Control
       url: /user-control
       output: web, pdf
@@ -39,6 +43,10 @@ entries:
       url: /docs-mount
       output: web, pdf
 
+    - title: Running Services
+      url: /docs-instances
+      output: web, pdf
+
     - title: Environment and Metadata
       url: /docs-environment-metadata
       output: web, pdf
diff --git a/pages/docs/admin-docs/docs-checks.md b/pages/docs/admin-docs/docs-checks.md
@@ -0,0 +1,73 @@
+---
+title: Singularity Checks
+sidebar: user_docs
+permalink: docs-admin-checks
+folder: docs
+toc: false
+---
+
+New to Singularity 2.4 is the ability to, on demand, run container "checks," which can be anything from a filter for sensitive information, to an analysis of content on the filesystem. Checks are installed with Singularity, managed by the administrator, and [available to the user](/docs-user-checks).
+
+
+## What is a check?
+Broadly, a check is a script that is run over a mounted filesystem, primary with the purpose of checking for some security issue. This process is tighly controlled, meaning that the script names in the [checks](https://github.com/singularityware/singularity/tree/development/libexec/helpers/checks) folder are hard coded into the script [check.sh](https://github.com/singularityware/singularity/blob/development/libexec/helpers/check.sh). The flow of checks is the following:
+
+ - the user calls `singularity check container.img` to invoke [check.exec](https://github.com/singularityware/singularity/blob/development/libexec/cli/check.exec)
+ - specification of `--low` (3), `--med` (2), or `--high` (1) sets the level to perform. The level is a filter, meaning that a level of 3 will include 3,2,1, and a level of 1 (high) will only call checks of high priority.
+ - specification of `-t/--tag` will allow the user (or execution script) to specify a kind of check. This is primarily to allow for extending the checks to do other types of things. For example, for this initial batch, these are all considered `default` checks. The [check.help](https://github.com/singularityware/singularity/blob/development/libexec/cli/check.help) displays examples of how the user specifies a tag:
+
+```
+    # Perform all default checks, these are the same
+    $ singularity check ubuntu.img
+    $ singularity check --tag default ubuntu.img
+
+    # Perform checks with tag "clean"
+    $ singularity check --tag clean ubuntu.img
+```
+ 
+### Adding a Check
+A check should be a bash (or other) script that will perform some action. The following is required:
+
+**Relative to SINGULARITY_ROOTFS**
+The script must perform check actions relative to `$SINGULARITY_ROOTFS`. For example, in python you might change directory to this location:
+
+```
+import os
+base = os.environ["SINGULARITY_ROOTFS"]
+os.chdir(base)
+```
+
+or do the same in bash:
+
+```
+cd $SINGULARITY_ROOTFS
+ls $SINGULARITY_ROOTFS/var
+```
+
+Since we are doing a mount, all checks must be static relative to this base, otherwise you are likely checking the host system.
+
+**Verbose**
+The script should indicate any warning/message to the user if the check is found to have failed. If pass, the check's name and status will be printed, with any relevant information. For more thorough checking, you might want to give more verbose output.
+
+**Return Code**
+The script return code of "success" is defined in [check.sh](check.sh), and other return codes are considered not success. When a non success return code is found, the rest of the checks continue running, and no action is taken. We might want to give some admin an ability to specify a check, a level, and prevent continuation of the build/bootstrap given a fail.
+
+**Check.sh**
+The script level, path, and tags should be added to [check.sh](check.sh) in the following format:
+
+```
+##################################################################################
+# CHECK SCRIPTS
+##################################################################################
+
+#        [SUCCESS] [LEVEL]  [SCRIPT]                                                                         [TAGS]
+execute_check    0    HIGH  "bash $SINGULARITY_libexecdir/singularity/helpers/checks/1-hello-world.sh"       security
+execute_check    0     LOW  "python $SINGULARITY_libexecdir/singularity/helpers/checks/2-cache-content.py"   clean
+execute_check    0    HIGH  "python $SINGULARITY_libexecdir/singularity/helpers/checks/3-cve.py"             security
+```
+
+The function `execute_check` will compare the level (`[LEVEL]`) with the user specified (or default) `SINGULARITY_CHECKLEVEL` and execute the check only given it is under the specified threshold, and (not yet implemented) has the relevant tag. The success code is also set here with `[SUCCESS]`. Currently, we aren't doing anything with `[TAGS]` and thus perform all checks.
+
+
+## How to tell users?
+If you add a custom check that you want for your users to use, you should tell them about it. Better yet, <a href="https://github.com/singularityware/singularity/issues" target="_blank">tell us </a> about it so it can be integrated into the Singularity software for others to use.
diff --git a/pages/docs/user-docs/docs-checks.md b/pages/docs/user-docs/docs-checks.md
@@ -0,0 +1,52 @@
+---
+title: Singularity Checks
+sidebar: user_docs
+permalink: docs-user-checks
+folder: docs
+toc: false
+---
+
+New to Singularity 2.4 is the ability to, on demand, run container "checks," which can be anything from a filter for sensitive information, to an analysis of content on the filesystem. Checks are installed with Singularity and [managed by the administration](/docs-admin-checks), however as a user they are accessible to you for use during bootstrap or on demand:
+
+```
+# Perform all default checks, these are the same
+$ singularity check ubuntu.img
+$ singularity check --tag default ubuntu.img
+
+# Perform checks with tag "clean"
+$ singularity check --tag clean ubuntu.img
+```
+
+## Tags and Organization
+Currently, checks are organized by tag and security level. If you know a specific tag that you want to use, for example "docker" deploys checks for containers with Docker imported layers, you can specify the tag:
+
+```
+USAGE
+
+    -t/--tag       tag to filter checks. default is "default"                      
+                      Available: default, security, docker, clean
+
+
+EXAMPLE
+
+singularity check --tag docker ubuntu.img
+```
+
+If you want to run checks associated with a different security level, you can specify with `--low`, `--med`, or `-high`:
+
+```
+USAGE: singularity [...] check [exec options...] <container path>
+
+This command will run security checks for an image.
+Note that some checks require sudo.
+
+    -l/--low       Specify low threshold (all checks, default) 
+    -m/--med       Perform medium and high checks
+    -h/--high      Perform only checks at level high
+```
+
+Note that some checks will require sudo, and you will be alerted if this is the case and you didn't use it. Finally, if you want to run all default checks, just don't specify a tag or level.
+
+
+## What checks are available?
+Currently, you can view all installable checks [here](https://github.com/singularityware/singularity/blob/development/libexec/helpers/check.sh#L49), and we anticipate adding an ability to view tags that are available, along with your own custom checks. You should also ask your administration if new checks have been added not supported by Singularity. If you want to request adding a new check, please <a href="https://github.com/singularityware/singularity/issues" target="_blank">tell us!</a>.
diff --git a/pages/docs/user-docs/docs-instances.md b/pages/docs/user-docs/docs-instances.md
@@ -0,0 +1,76 @@
+---
+title: Singularity Container Instances
+sidebar: user_docs
+permalink: docs-instances
+folder: docs
+toc: false
+---
+
+New to Singularity 2.4 is the ability to clone your image, meaning you create an instance of it that has its own namespace. Why would you want to do this? It means that your container can be instantiated and then serve a process that your computer has control of. 
+
+## Why container instances?
+Let's say I want to run a web server. With nginx, that is pretty simple, I install nginx and start the service:
+
+```
+apt-get update && apt-get install -y nginx
+service nginx start
+```
+
+With older versions of Singularity, if you were to do something like this, from inside the container you would happily see the service start, and the web server running! But then if you were to log out of the container what would happen?
+
+>> ghost process!!
+
+You would lose control of the process. It would still be running, but you couldn't kill it. This is a called a ghost process, and it means that for running (enduring) services, Singularity was a no starter.
+
+
+## Cloning containers
+With version 2.4, you can do this in a more realistic way. First, let's put the commands of how to start our service into a script. Let's call it a `startscript`. And we can imagine this fitting into a bootstrap recipe file like this:
+
+```
+%startscript
+
+service nginx start
+```
+
+and we will pretend that we are running with sudo, and we don't specify it above. Now let's say we have a container called `nginx.img` and we want to run it! What do we do?
+
+```
+            [action]  [image]    [name of instance]
+singularity clone     nginx.img  service
+```
+
+When I do that, I still have my file `nginx.img` sitting on my Desktop, but now you can think about having actually an instance of it running, which I can now control! Heck, I could do that multiple times, if it made sense for my service:
+
+```
+singularity clone     nginx.img  instance1
+singularity clone     nginx.img  instance2
+singularity clone     nginx.img  instance3
+```
+
+Once you create this instance, you can't do additional things like binds. So if your service requires a special mount or any other kind of connection, do that at the time of the clone:
+
+```
+singularity clone   -B /etc/nginx  nginx.img  instance1
+```
+
+## Starting Services
+Once you have generated instances, you can start them up! You do that with start, directed to the instance name:
+
+```
+singularity start nginx.img instance1
+```
+
+## Listing Services
+You can then easily list services:
+
+```
+singularity list 
+```
+
+## Important Notes
+
+- The instances are linked with your user. So if you clone and start with sudo, that is going to go under root, and you will be confused to call `singularity list` as your user and then not see your services.
+- The only reason to specify the image is because it could be the case that you have two different images with services named equally.
+
+
+This stuff is completely under development and likely to change! <a href="https://github.com/singularityware/singularity/issues" target="_blank"> Join the conversation!</a>.
