FAQ updates

Author: gmkurtzer

pages/docs/overview/faq.md

@@ -8,53 +8,27 @@ folder: docs
 ## General Singularity info
 
 ### Why the name "Singularity"?
+A "Singularity" is an astrophysics phenomenon in which a single point becomes infinitely dense. This type of a singularity can thus contain massive quantities of universe within it and thus encapsulating an infinite amount of data within it.
 
-The name "Singularity" for me (Greg) stems back from my past experience working at a company called <a href="https://en.wikipedia.org/wiki/Linuxcare" target="_blank">Linuxcare</a> where the Linux Boot-able Business Card (LNX-BBC) was developed. The BBC, was a Linux rescue disk which paved the way for all live CD bootable distributions using a loop back file system called the "singularity".
+Additionally, the name "Singularity" for me (Greg) also stems back from my past experience working at a company called <a href="https://en.wikipedia.org/wiki/Linuxcare" target="_blank">Linuxcare</a> where the Linux Boot-able Business Card (LNX-BBC) was developed. The BBC, was a Linux rescue disk which paved the way for all live CD bootable distributions using a compressed single image file system called the "singularity".
 
-This nomenclature represented that all files within the environment were contained within a single file, and for the same reason Singularity emphasizes the same nomenclature. (Thanks LNX-BBC!)
+The name has **NOTHING** to do with Kurzweil's (among others) prediction that artificial intelligence will abruptly have the ability to reprogram itself, surpass that of human intelligence and take control of the planet. If you are interested in this may I suggest the movie **Terminator 2: Judgement Day**.
 
+### What is so special about Singularity?
+While Singularity is a container solution (like many others), Singularity differs in it's primary design goals and architecture:
 
-### Which namespaces are virtualized? Is that select-able?
-
-The goal of Singularity is to run an application within a contained environment such as it was not contained. Thus there is a balance between what to separate and what not to separate. At present the virtualized namespaces are process, mount points, and certain parts of the contained file system.
-
-When you run your Singularity container, you may find that the process IDs start with 1 (one) and increment from there. You will also find that while the file system is contained starting with '/' (root), you can have access outside the container via your starting path. This means that relative paths will resolve outside the container, and fully qualified paths will resolve inside the container.
-
-To achieve this behavior, you will find that several Linux namespaces are separated (PIDS, file systems and descriptors, mounts, and root file system). These can be enabled or disabled by the build and what namespaces the host system supports as well as through environment variables.
-
+1. **Reproducible software stacks:** These must be easily verifiable via checksum or cryptographic signature in such a manner that does not change formats (e.g. splatting a tarball out to disk). By default Singularity uses a container image file which can be checksummed, signed, and thus easily verified and/or validated.
+2. **Mobility of compute:** Singularity must be able to transfer (and store) containers in a manner that works with standard data mobility tools (rsync, scp, gridftp, http, NFS, etc..) and maintain software and data controls compliancy (e.g. HIPPA, nuclear, export, classified, etc..)
+3. **Compatibility with complicated architectures:** The runtime must be immediately compatible with existing HPC, scientific, compute farm and even enterprise architectures any of which maybe running legacy kernel versions (including RHEL6 vintage systems) which do not support advanced namespace features (e.g. the user namespace)
+4. **Security model:** Unlike many other container systems designed to support *trusted users running trusted containers* we must support the opposite model of *untrusted users running untrusted containers*. This changes the security paradigm considerably and increases the breadth of use cases we can support.
 
-### Can't you do this with Docker?
-Singularity has taken off within the scientific computing world because Docker, while fantastic for enterprise services and local use, is a no starter on traditional HPC resources. There are no HPC centers utilizing Docker on their traditional HPC resources because it is inheritiatnly incompatible with HPC.
-
-Singularity has a very different usage model then Docker in that Singularity utilizes complete image files while Docker containers are made up of layers of tar files. With Singularity, you can prefetch an entire container, cache it on shared optimized storage, and run it from there.   Singularity also limits user privileges and access from within the container, making it safe for user's to bring their own containers. It doesn't open up security risks of users being able to control a root owned daemon, and it integrates seemlessly into existing process and resource manager workflows, supports GPU, MPI, architecture independent, among lots of other aspects that Docker does not.
-
-In the end, Docker is designed for micro-service network virtulization and emulation of the full isolation requirements in the legacy of full machine level virtualization platforms (e.g. VMWare, Xen, KVM, etc.). Singularity is designed specifically for the scentific, application and environment virtulization. The right tool for the right job. :)
-
-If you already have a Docker container you can port it directly into a Singularity image!
-
-```bash
-singularity pull docker://godlovedc/lolcow
-singularity build lolcow.simg docker://godlovedc/lolcow
 
-# Shell right in
-singularity shell docker://godlovedc/lolcow
-
-# Or just run
-singularity run docker://godlovedc/lolcow
-```
-
-
-### How does Singularity relate/differ from other container technology? 
-Singularity differs from other container systems in several major ways that impact usability on shared systems. For example, most container systems emulate standard systems in that there is the ability and necessity to escalate to root, run on separate IP/network address, run services, and in some cases even virtually boot the container system.
-
-Singularity on the other hand focuses on the ability to virtualized only what is necessary to achieve run-time application container and portable environments. For instance, you can not obtain root from within a Singularity container.
-
-There are some additional performance and design enhancements which make Singularity also more appropriate in a scheduled HPC environment. The back-end image type is one such feature that negates the need for temporary caching of container images and optimizes meta-data IO (especially on parallel file systems). Another feature is how Singularity interacts with the host operating system to facilitate application work-flows like X11 and MPI.
+### Which namespaces are virtualized? Is that select-able?
+That is up to you!
 
-### How does Singularity relate/differ from statically compiled binaries?
-Statically compiled binaries are a good comparison to what Singularity can do for a single program because it will package up all of the dynamic libraries and package them into a single executable (interpreted) format.
+While some namespaces, like newns (mount) and fs (file system) must be virtualized, all of the others are conditional depending on what you want to do. For example, if you have a workflow that relies on communication between containers (e.g. MPI), it is best to not isolate any more then absolutely necessary to avoid performance regressions. While other tasks are better suited for isolation (e.g. web and data base services).
 
-But because Singularity is actually wrapping operating system files in to a container, you can do much more with it... Such as include other files, scripts, work-flows, pipe lines, data, and multi program processes and package them into a single portable executable format.
+Namespaces are selected via command line usage and system administration configuration.
 
 ### What Linux distributions are you trying to get on-board?
 All of them! Help us out by letting them know you want Singularity to be included!
@@ -63,13 +37,12 @@ All of them! Help us out by letting them know you want Singularity to be include
 ## Basic Singularity usage
 
 ### Do you need administrator privileges to use Singularity?
-You generally do not need admin/sudo to use Singularity containers. As of version 2.3, you can create, import, run, export, and shell without it. You do however need admin/root access to install Singularity and to bootstrap a container (build it from a specification file called `Singularity`). This means that, given that your cluster has Singularity installed, you are empowered to generate your own portable environments.
+You generally do not need admin/sudo to use Singularity containers but you do however need admin/root access to install Singularity and for some container build functions (for example, building from a recipe, or a writable image).
 
-This then defines the work-flow to some extent. If you have a container (whether Singularity or Docker) ready to go, you can run/shell/import without root access. If you want to build and bootstrap, then your Singularity container image must be built and configured on a host where you have root access (this can be a physical system or on a VM or Docker image). And of course once the container image has been configured it can be used on a system where you do not have root access as long as Singularity has been installed there.
+This then defines the work-flow to some extent. If you have a container (whether Singularity or Docker) ready to go, you can run/shell/import without root access. If you want to build a new Singularity container image from scratch it must be built and configured on a host where you have root access (this can be a physical system or on a VM). And of course once the container image has been configured it can be used on a system where you do not have root access as long as Singularity has been installed there.
 
 
 ### What if I don't want to install Singularity on my computer?
-
 If you don't want to build your own images, <a href="https://singularity-hub.org" target="_blank">Singularity Hub</a> will connect to your Github repos with build specification files, and build the containers automatically for you. You can then interact with them easily where Singularity is installed (e.g., on your cluster):
 
 ```bash
@@ -80,7 +53,9 @@ singularity build hello-world.simg shub://vsoch/hello-world # redundant, you wou
 ```
 
 ### Can you edit/modify a Singularity container once it has been instantiated?
-We strongly advocate for reproducibility, so if you build a squashfs container, it is immutable. However, if you build with `--sandbox` or `--writable` you can produce a writable sandbox folder or ext3 image, respectively. From a sandbox you can develop, test, and make changes, and then build into a full squashfs image. We recommend to our users to use the compressed, immutable format for production containers.
+We strongly advocate for reproducibility, so if you build a squashfs container, it is immutable. However, if you build with `--sandbox` or `--writable` you can produce a writable sandbox folder or a writable ext3 image, respectively. From a sandbox you can develop, test, and make changes, and then build or convert it into a standard image.
+
+We recommend to use the default compressed, immutable format for production containers.
 
 ### Can multiple applications be packaged into one Singularity Container?
 Yes! You can even create entire pipe lines and work flows using many applications, binaries, scripts, etc.. The `%runscript` bootstrap section is where you can define what happens when a Singularity container is run, and with the introduction of [modular apps](/docs-apps) you can now even define `%apprun` sections for different entrypoints to your container.
@@ -98,45 +73,45 @@ singularity run --bind /home/vanessa/Desktop:/data container.img
 
 
 ### How does Singularity handle networking?
-As of 2.4, Singularity now has the ability to support instances, each with their own Network namespace, and the starting behavior defined by the `%startscript`. Without running as an instance, Singularity does no network isolation and runs like any other application on the system. It has all of the same networking privileges as any program running as that user. This means that if you run a process in the container directed to localhost, it will show up on your host machine browser as localhost. Take a look at some of our <a href="https://github.com/vsoch/singularity-web" target="_blank">web examples</a> to get a sense for this.
+As of 2.4, Singularity can support the network namespace to a limited degree. At present, we just use it for isolation, but it will soon be more featurefull.
 
 
-### Can a Singularity container be multi-threaded?
+### Can Singularity support daemon processes?
+Singularity has container "instance" support which allows one to start a container process, within its own namespaces, and use that instance like it was a stand alone, isolated system.
+
+At the moment (as above describes), the network (and UTS) namespace is not well supported, so if you spin up a process daemon, it will exist on your host's network. This means you can run a web server, or any other daemon, from within a container like and access it directly from your host.
 
+### Can a Singularity container be multi-threaded?
 Yes. Singularity imposes no limitations on forks, threads or processes in general.
 
 ### Can a Singularity container be suspended or check-pointed?
+Yes and maybe respectively. Any Singularity application can be suspended using standard Linux/Unix signals. Check-pointing requires some preloaded libraries to be automatically loaded with the application but because Singularity escapes the hosts library stack, the checkpoint libraries would not be loaded. If however you wanted to make a Singularity container that can be check-pointed, you would need to install the checkpoint libraries into the Singularity container via the specfile.
 
-Yes and maybe respectively. Any Singularity application can be suspended using standard Linux/Unix signals. Check-pointing requires some preloaded libraries to be automatically loaded with the application but because Singularity escapes the hosts library stack, the checkpoint libraries would not be loaded. If however you wanted to make a Singularity container that can be check-pointed, you would need to install the checkpoint libraries into the Singularity container via the specfile
+On our roadmap is the ability to checkpoint the entire container process thread, and restart it. Keep an eye out for that feature!
 
 
-### Are there any special requirements to use Singularity through a job scheduler?
-
+### Are there any special requirements to use Singularity through an HPC job scheduler?
 Singularity containers can be run via any job scheduler without any modifications to the scheduler configuration or architecture. This is because Singularity containers are designed to be run like any application on the system, so within your job script just call Singularity as you would any other application!
 
 
 ### Does Singularity work in multi-tenant HPC cluster environments?
- 
 Yes! HPC was one of the primary use cases in mind when Singularity was created.
 
 Most people that are currently integrating containers on HPC resources do it by creating virtual clusters within the physical host cluster. This precludes the virtual cluster from having access to the host cluster's high performance fabric, file systems and other investments which make an HPC system high performance.
 
 Singularity on the other hand allows one to keep the high performance in High Performance Computing by containerizing applications and supporting a runtime which seamlessly interfaces with the host system and existing environments.
 
 ### Can I run X11 apps through Singularity?
-
 Yes. This works exactly as you would expect it to.
 
 ### Can I containerize my MPI application with Singularity and run it properly on an HPC system?
-
 Yes! HPC was one of the primary use cases in mind when Singularity was created.
 
 While we know for a fact that Singularity can support multiple MPI implementations, we have spent a considerable effort working with Open MPI as well as adding a Singularity module into Open MPI (v2) such that running at extreme scale will be as efficient as possible.
 
 note: We have seen no major performance impact from running a job in a Singularity container.
 
 ### Why do we call 'mpirun' from outside the container (rather then inside)?
-
 With Singularity, the MPI usage model is to call 'mpirun' from outside the container, and reference the container from your 'mpirun' command. Usage would look like this:
 
 ```bash
@@ -149,9 +124,7 @@ In the end, we do not gain anything by calling 'mpirun' from within the containe
 
 See the Singularity on HPC page for more details.
 
-
 ### Does Singularity support containers that require GPUs?
-
 Yes. Many users run GPU dependant code within Singularity containers.  The
 experimental `--nv` option allows you to leverage host GPUs without installing 
 system level drivers into your container. See the [`exec`](/docs-exec#a-gpu-example) command for
@@ -160,7 +133,6 @@ an example.
 ## Container portability
 
 ### Are Singularity containers kernel dependent?
-
 No, never. But sometimes yes.
 
 Singularity is using standard container principals and methods so if you are leveraging any kernel version specific or external patches/module functionality (e.g. OFED), then yes there maybe kernel dependencies you will need to consider.
@@ -172,19 +144,16 @@ There is also some level of glibc forward compatibility that must be taken into
 note: If you require kernel dependent features, a container platform is probably not the right solution for you.
 
 ### Can a Singularity container resolve GLIBC version mismatches?
-
 Yes. Singularity containers contain their own library stack (including the Glibc version that they require to run).
 
 ### What is the performance trade off when running an application native or through Singularity?
-
 So far we have not identified any appreciable regressions of performance (even in parallel applications running across nodes with InfiniBand). There is a small start-up cost to create and tear-down the container, which has been measured to be anywhere from 10 - 20 thousandths of a second.
 
 ## Misc
 
 The following are miscellaneous questions.
 
 ### Are there any special security concerns that Singularity introduces?
-
 No and yes.
 
 While Singularity containers always run as the user launching them, there are some aspects of the container execution which requires escalation of privileges. This escalation is achieved via a SUID portion of code. Once the container environment has been instantiated, all escalated privileges are dropped completely, before running any programs within the container.
@@ -193,14 +162,10 @@ Additionally, there are precautions within the container context to mitigate any
 
 You can read more about the Singularity <a href="/docs-security">security overview here</a>.
 
-
 ## Troubleshooting
-
 A little bit of help.
 
-
 ### How to use Singularity with GRSecurity enabled kernels
-
 To run Singularity on a GRSecurity enabled kernel, you must disable several security features:
 
 ```bash
@@ -252,11 +217,10 @@ Also, make sure you are using an up-to-date Singularity to bootstrap your contai
 
 
 ### Error running Singularity with sudo
-
 This fix solves the following error when Singularity is installed into the default compiled prefix of /usr/local:
 
 ```bash
-$ sudo singularity create /tmp/centos.img
+$ sudo singularity instance.start container.img daemon1
 sudo: singularity: command not found
 ```