Merge branch 'master' of https://github.com/singularityware/singularityware.github.io

just doing a standard git pull upstream master results in merge??

Author: GodloveD

_posts/news/2017-09-13-docker-manifest-v2.md

@@ -0,0 +1,17 @@
+---
+title:  "Announcement: Problem downloading from Docker Hub to be resolved soon"
+category: news
+permalink: 2017-docker-problem
+---
+
+To all Singularity users,
+ 
+On Tuesday September 12, Docker released a new version of Docker image metadata.  This means that any new images built on Docker Hub cannot currently be downloaded using a singularity `pull` or other commands like `shell`, `exec`, and `bootstrap` when updated Docker registries are queried.
+ 
+Vanessa (`@v`) has created an interim fix for the problem and we have merged it into the development branch.  Pending further testing we plan to merge this fix into master and create a new minor release (2.3.2).  We will make another announcement as soon as it is ready to install. 
+ 
+Thanks for your patience! 
+ 
+The Singularity team
+
+{% include links.html %}

_posts/news/2017-09-26-scif-draft.md

@@ -0,0 +1,30 @@
+---
+title:  "Standard Container Integration Format (SCI-F) Final Draft"
+category: news
+permalink: 2017-scif-contribute
+---
+
+Hi Singularity Community!
+
+Thanks to everyone that provided comments on the early draft for SCI-F. The goals were initially:
+
+ - write specification draft (this was done via a Google Doc)
+ - implement into Singularity (now in [development branch](https://github.com/singularityware/singularityware.github.io/blob/docs/2.4/pages/docs/user-docs/docs-apps.md))
+ - get feedback on both, adjust implementation and draft
+ - do several (N=4) implemented use cases for SCI-F, write up ([available here](http://containers-ftw.org/apps/category/#Example))
+ - make an interactive, open source repository to contribute and share SCI-F apps [http://containers-ftw.org/apps/](http://containers-ftw.org/apps/)
+ - write up the (almost) "final" draft formally [http://containers-ftw.org/SCI-F/](http://containers-ftw.org/SCI-F/)
+
+I'm (`@vsoch`) happy to report that I've finished the above, and I'd like to ask the following:
+
+ - if you contributed, please check the AUTHORS file in the SCI-F repository to make sure that I added you, and spelled your name correctly (apologies in advance).
+ - Any further suggestions, additions, or entire new contributions can be added to the draft by way of pull request. I'm not in any rush, and if you have a good contribution, I want to help.
+ - If you have not contributed yet and would like to, see the ideas linked on the [draft abstract page](http://containers-ftw.org/SCI-F/). 
+
+The plan is (as of now) to submit this no later than the end of October, and I want to make sure everyone that wants to contribute has had ample chance. Indeed, with Singularity there are many ways to cook your container, and SCi-F definitely exemplifies that.
+
+![egg](/images/logo/three-ways.png)
+
+Thanks again everyone!
+
+{% include links.html %}

_posts/recipes/2017-01-26-singularity-hub-tutorial.md

@@ -30,7 +30,7 @@ The easiest thing to do is to install Singularity on your local workstation:
     sudo make install
 ```
 
-If you have the unfortunate situation of using a Mac, or just need a virtual machine, then you will want to follow the instructions <a href="http://singularity.lbl.gov/install-mac" target="_blank">here</a>. Basically, you need to install vagrant, virtual box, and then do this:
+If you are using a Mac, or just need a virtual machine, then you will want to follow the instructions <a href="http://singularity.lbl.gov/install-mac" target="_blank">here</a>. Basically, you need to install vagrant, virtual box, and then do this:
 
 ```bash
 vagrant init ubuntu/trusty64

_posts/releases/2017-09-15-release-2-3-2.md

@@ -0,0 +1,96 @@
+---
+title:  "Singularity 2.3.2 Quick Fix Release"
+category: releases
+permalink: "release-2-3-2"
+targz: "2.3.2.tar.gz"
+---
+
+<a target="_blank" href="https://github.com/singularityware/singularity/releases/tag/2.3.2">Release 2.3.2 </a> This dot release includes a fix for a change that Docker implemented to their registry RESTful API which broke compatibility with Singularity (among several other low minor fixes).
+
+
+## What happened?
+
+
+**Where does Docker image metadata come from?**
+The Docker Registry serves metadata about images via manifests, where each image has a manifest that tells Singularity important information like environment, labels, and entrypoints and running command. Importantly, the image manifest serves the actual layers that should be obtained to create the image. The manifests come in different flavors, or schema versions. Version 1 serves the primary load of metadata (labels, environment) while the first release of version 2 served layers, and had the addition of size. This addition made it possible to pull an image with Singularity and calculate the size on the fly for images generated with support for this manifest.
+
+Singularity had an old default to retrieve the version 2 (done by way of a header asking for it), and ask it for layers. If the remote registry could only offer version 1, we were still able to obtain a list of layers (under a different key, `fsLayers` instead of `layers`) and metadata via the older (schema 1) manifest. However, with the update, the API version 2 schema can now return a <a href="https://docs.docker.com/registry/spec/manifest-v2-2/#manifest-list"> list of manifests</a>. This meant that when we checked the response for the keys `layers` or `fslayers`, they wouldn't be found, becaues we needed to look under `manifests`, and then do a second call to retrieve the manifest of interest based on a system architecture and OS. This of course broke most `import`, `shell`, `pull`, `exec`, because all of these commands require retrieving the layers.
+
+## The Patch
+A super quick fix would have been to fall back to the version 1 manifest, always, but then we would lose the automatic calculation of size, which is important to many users. The "better" fix is obvious - the code needs to:
+
+ - check for a `manifests` key
+ - if found, select a default manifest to use
+ - retrieve it, and continue!
+ - if not found, fall back to version 1
+
+This means that we've added environment variables `SINGULARITY_DOCKER_OS` and `SINGULARITY_DOCKER_ARCHITECTURE` to specify this choice, with defaults `linux` and `amd64`. This is a pretty exciting change, because it means down the line you (as a user!) can specify the specifics of the layers you want returned, given an image that has this metadata available. We can see the fix running as follows:
+
+
+```
+DEBUG 
+*** STARTING DOCKER IMPORT PYTHON  ****
+```
+This is the initialization of the Docker client, it's parsing the image name provided
+```
+DEBUG Starting Docker IMPORT, includes env, runscript, and metadata.
+VERBOSE Docker image: ubuntu:14.04
+VERBOSE2 Specified Docker ENTRYPOINT as %runscript.
+DEBUG Headers found: Content-Type,Accept
+VERBOSE Registry: index.docker.io
+VERBOSE Namespace: library
+VERBOSE Repo Name: ubuntu
+VERBOSE Repo Tag: 14.04
+VERBOSE Version: None
+```
+The initial poke to Docker hub asks for tags, to determine if we need some kind of token
+```
+VERBOSE Obtaining tags: https://index.docker.io/v2/library/ubuntu/tags/list
+DEBUG GET https://index.docker.io/v2/library/ubuntu/tags/list
+```
+
+401 means that we do, and with the response the API sends back the scope and other details we need to make to request it
+
+```
+DEBUG Http Error with code 401
+```
+
+Here we are requesting it
+```
+DEBUG GET https://auth.docker.io/token?service=registry.docker.io&expires_in=9000&scope=repository:library/ubuntu:pull
+DEBUG Headers found: Content-Type,Authorization,Accept
+```
+
+And here is the new bit. Instead of blindly doing one call, we have a function to update two versions of the manifest - the older (with metadata) and some version of the newer (with layers and size) with a fallback to use the version 1
+```
+DEBUG Updating manifests.
+
+# Here is version 2+
+DEBUG MANIFEST (Primary): not found, making initial call.
+VERBOSE Obtaining manifest: https://index.docker.io/v2/library/busybox/manifests/latest
+DEBUG GET https://index.docker.io/v2/library/busybox/manifests/latest
+
+# Here is version 1 (Metadata)
+DEBUG MANIFEST (Metadata): not found, making initial call.
+VERBOSE Obtaining manifest: https://index.docker.io/v2/library/busybox/manifests/latest
+DEBUG GET https://index.docker.io/v2/library/busybox/manifests/latest
+```
+Notice that the two calls are the same - that's because the headers are actually different, to request different ones.
+
+And here is the (new) indication that we found a list
+```
+DEBUG Image manifest version 2.2 list found.
+```
+Here is the default architecture / os
+```
+DEBUG Obtaining architecture: amd64, OS: linux
+```
+And the specific call to get it
+```
+VERBOSE Obtaining manifest: https://index.docker.io/v2/library/busybox/manifests/sha256:030fcb92e1487b18c974784dcc110a93147c9fc402188370fbfd17efabffc6af
+DEBUG GET https://index.docker.io/v2/library/busybox/manifests/sha256:030fcb92e1487b18c974784dcc110a93147c9fc402188370fbfd17efabffc6af
+```
+
+With this fix - we can again use `pull`/`import`, etc, and also have a working `singularity pull docker://busybox` that estimates the size automatically.
+
+Please report any additional bugs to <a target="_blank" href="https://github.com/singularityware/singularity/issues/new">our issues board.</a>

images/logo/three-ways.png

@@ -16,7 +16,7 @@ This nomenclature represented that all files within the environment were contain
 
 ### Which namespaces are virtualized? Is that select-able?
 
-The goal of Singularity is to run an application within a contained environment such as it was not contained. Thus there is a balance between what to separate and what not to separate. At present the virtualized namespaces are process, mount points, and certain parts of the contained file system.
+The goal of Singularity is to run an application within a contained environment as if it was not contained. Thus there is a balance between what to separate and what not to separate. At present the virtualized namespaces are process, mount points, and certain parts of the contained file system.
 
 When you run your Singularity container, you may find that the process IDs start with 1 (one) and increment from there. You will also find that while the file system is contained starting with '/' (root), you can have access outside the container via your starting path. This means that relative paths will resolve outside the container, and fully qualified paths will resolve inside the container.
 
@@ -25,10 +25,10 @@ To achieve this behavior, you will find that several Linux namespaces are separa
 
 ### Can't you do this with Docker?
 
-No, not even close. If that was true, there would be no use for Singularity. But in fact, Singularity has taken off within the scientific computing world! Furthermore, if Docker can be used on traditional HPC resources, it would be; but it is not! There are no HPC centers utilizing Docker on their traditional HPC resources because it is inheritiatnly incompatible with HPC.
+No, not even close. If that was true, there would be no use for Singularity. But in fact, Singularity has taken off within the scientific computing world! Furthermore, if Docker can be used on traditional HPC resources, it would be; but it is not! There are no HPC centers utilizing Docker on their traditional HPC resources because it is inherently incompatible with HPC.
 
-Singularity has a very different usage model then Docker in that Singularity utilizes complete image files while Docker containers are made up of layers of tar files. With Singularity, you can prefetch an entire container, cache it on shared optimized storage, and run it from there.   Singularity also limits user privileges and access from within the container, making it safe for
-user's to bring their own containers. It doesn't open up security risks of users being able to control a root owned daemon, and it integrates seemlessly into existing process and resource manager workflows, supports GPU, MPI, architecture independent, among lots of other aspects that Docker does not.
+Singularity has a very different usage model than Docker in that Singularity utilizes complete image files while Docker containers are made up of layers of tar files. With Singularity, you can prefetch an entire container, cache it on shared optimized storage, and run it from there. Singularity also limits user privileges and access from within the container, making it safe for
+users to bring their own containers. It doesn't open up security risks of users being able to control a root owned daemon, and it integrates seemlessly into existing process and resource manager workflows, supports GPU, MPI, architecture independence, among lots of other aspects that Docker does not.
 
 In the end, Docker is designed for micro-service network virtulization and emulation of the full isolation requirements in the legacy of full machine level virtualization platforms (e.g. VMWare, Xen, KVM, etc.). Singularity is designed specifically for the scentific, application and environment virtulization. The right tool for the right job. :)
 

pages/docs/overview/faq.md

@@ -81,6 +81,18 @@ Progress |===================================| 100.0%
 Done. Container is at: /tmp/vsoch-hello-world-master.img
 ```
 
+### Pull by commit
+You can also pull different versions of your container by using their commit id (`version`). 
+
+```
+singularity pull shub://vsoch/hello-world@42e1f04ed80217895f8c960bdde6bef4d34fab59
+Progress |===================================| 100.0%
+Done. Container is at: ./vsoch-hello-world-master.img
+```
+
+In this example, the first build of this container will be pulled.
+
+
 ## Docker
 Docker pull is similar (on the surface) to a Singularity Hub pull, and we would do the following: