Detailed Singularity walk-through

This walk-through is based on our Singularity tutorial and consists of several exercises that will guide you through the entire process of installing Singularity and creating images to actually running them on a different system (Peregrine). The solutions and explanations can be found at the bottom of the page. The walk-through assumes that you have some Ubuntu-like system with root privileges (for the tutorial we use Xubuntu VMs) and that the container will be run with a regular user account on Peregrine.

At the end of this walk-through, you will be able to submit a job on Peregrine that uses Singularity to run the following Python-pandas script, which takes an input file and processes it:

script.py

# Based on https://www.kaggle.com/antgoldbloom/exploring-climate-change-data
 
import pandas as pd
import sys
 
# Check usage
if len(sys.argv) < 2:
    print("Usage: %s <csv file>" % sys.argv[0])
    sys.exit()
 
csvfile = sys.argv[1]
 
# Sort all cities on average temperature
dfTempByCity = pd.read_csv(csvfile,index_col='dt',parse_dates=[0])
cities_2012_avg = dfTempByCity[dfTempByCity.index.year == 2012][['City','Country','AverageTemperature']].groupby(['City','Country']).mean().sort_values('AverageTemperature',ascending=False)
 
print("Cities with highest average temperature in 2012:")
print(cities_2012_avg.head())
 
print("")
 
print("Cities with lowest average temperature in 2012:")
print(cities_2012_avg.tail())
 
#Temperature in Groningen using a 12 month rolling mean to smooth out seasonality
plot = pd.Series.rolling(dfTempByCity[dfTempByCity['City'] == 'Groningen']['AverageTemperature'],window=12).mean().plot(x=dfTempByCity.index)
plot.get_figure().savefig('plot.png')

The required dataset can be downloaded from Kaggle (please note you will need to login to access the data) and we will refer to it as inputfile.csv. For the exercises we assume that both files are stored in /data/<peregrine_username> on Peregrine.

Install Singularity on your machine. You can use the instructions from the Singularity website. On Peregrine we use the latest stable version, so it is best to use the same one on your own machine.

Solution

Create a blank Singularity image and then populate it by using the import function to import a Python 3.5.3 image from the Docker Hub. Finally, shell into your container and test it:

• verify which operating system is installed;
• check your username, user id and groups;
• check the disk space usage;
• do environment variables like $PATH get transferred from the host to the container?

Solution

We need to add a few more things to our container. Before doing this, we have to expand its size. Use the expand function to add 1GB of space to the image. Then shell into it with write permission, update the packages through your package manager and finally install the Python packages pandas and matplotlib.

Test the container by running the following command in it:

python3.5 script.py inputfile.csv

Solution

Now that the container is ready, we are going to run it on Peregrine. First copy the image to Peregrine; because of its size, store it in your personal directory on /data. Then log in to the interactive node of Peregrine and try to shell into your container using the --contain option: what does it do? What happens if you store a file on /home and leave the container? Also try to use the --bind option to mount your /data/<peregrine_username> directory in the container: make sure that you can find all the necessary files (Python script and input data) in your container.

Solution

Now we want to run the Python script in the container through a SLURM job. Write a job script that requests a few minutes of wall clock time and runs the job in the short partition, and use the singularity exec function to run the Python script. You can store the job script in the same directory as all the other files. Finally, submit the job and check the results when it is done.

Solution

In the previous exercises we have manually set up the container image, which is not a very reproducible way of working. A definition file allows you to automate all these steps; the Singularity bootstrap command uses a definition file to bootstrap an image. Write a definition file that contains all the steps to generate the same container as you had before, create a blank image and bootstrap it using your definition file. Finally, copy it to Peregrine, run it again and verify that it works.

Solution

Singularity also allows you to make use of GPUs in your containers. In this exercise we are going to test this. First, you will need to compile some CUDA application. You can for instance use the Hello World code from the following website:
http://computer-graphics.se/hello-world-for-cuda.html

Load the most recent CUDA module and compile it with the nvcc compiler. In order to run it on a GPU node, write a job script that requests one GPU, runs the job in the “gpu” partition and requests an appropriate amount of wall time (for the Hello World example 1 minute is fine). The job script only has to run your compiled application through the container.
Note that your application needs the NVIDIA drivers in order to run on the GPU. On the GPU nodes these can be found in /usr/lib64/nvidia; you will need to make them available in your container as well.

Hint:
use a bind mount and adjust your $LD_LIBRARY_PATH.

Solution

You may have to install these build dependencies first:

sudo apt install build-essential autoconf libtool wget

In order to install Singularity 2.2.1, run the following commands:

wget https://github.com/singularityware/singularity/releases/download/2.2.1/singularity-2.2.1.tar.gz
tar xzvf singularity-2.2.1.tar.gz
cd singularity-2.2.1
./configure --prefix=/usr/local
make
sudo make install

Back to exercise 1
To exercise 2

Create a blank image tutorial.img:

sudo singularity create tutorial.img

Import the Docker image python:3.5.3 into your tutorial.img:

sudo singularity import tutorial.img docker://python:3.5.3

Shell into the container:

singularity shell --shell /bin/bash tutorial.img

Test it by running these commands:

cat /etc/os-release
id
df -h
env
exit

Back to exercise 2
To exercise 3

Expand the size of the image by 1024MB:

sudo singularity expand --size 1024 tutorial.img

Shell into the container (--shell /bin/bash will use a Bash shell instead of a Dash shell) as root (sudo) and with write permission (-w):

sudo singularity shell --shell /bin/bash -w tutorial.img

Run the following commands to update the package list, upgrade packages, and install pandas and matplotlib through pip3:

apt update
apt upgrade
pip3 install pandas matplotlib
exit

In order to run the Python script, either shell into the container and run the command, or use Singularity exec:

singularity exec tutorial.img python3.5 script.py inputfile.csv

Back to exercise 3
To exercise 4

First, we have to create a /data/<peregrine_username> directory that will serve as a mount point for Peregrine’s /data/<peregrine_username>:

sudo singularity shell --shell /bin/bash --writable tutorial.img

Run these commands in the container:

mkdir -p /data/<peregrine_username>
exit

Now copy the image to your data directory on Peregrine, log in to Peregrine, go to the data directory, and start your container:

scp tutorial.img <peregrine_username>@peregrine.hpc.rug.nl:/data/<peregrine_username>/SingularityTutorial
ssh <peregrine_username>@peregine.hpc.rug.nl
cd /data/$USER/SingularityTutorial
singularity shell --shell /bin/bash tutorial.img

Test the container by running some commands, e.g.:

pwd
ls -l $HOME
exit

Find out what the --contain option for shell does:

singularity shell --contain --shell /bin/bash tutorial.img
    pwd
    ls -l $HOME
    touch $HOME/testfile.txt
    exit

And do the same for the --bind option:

singularity shell --bind /data/$USER/:/data/$USER --shell /bin/bash tutorial.img
    ls -lh /data/$USER
    exit

Back to exercise 4
To exercise 5

Open a text editor to create a jobscript.sh, e.g.:

nano jobscript.sh

The contents of the job script should look like this:

#!/bin/bash
#SBATCH --time=00:05:00
#SBATCH --partition=short
 
singularity exec --bind /data/$USER/:/data/$USER/ tutorial.img python3.5 script.py inputfile.csv

We did not specify any resource requirements, but the defaults are fine (1 core, 2GB of memory).

Submit the job using:

sbatch jobscript.sh

Finally, study the output file:

less slurm-*.out

Back to exercise 5
To exercise 6

We need to have the debootstrap command on the host machine. On Ubuntu you can install this using:

sudo apt install debootstrap

Create a blank image (of about 1GB) and bootstrap it using:

sudo singularity create --size 1024 tutorial.img
sudo singularity bootstrap tutorial.img pandas.def

The pandas.def is the definition file, that should look like this:

BootStrap: debootstrap
OSVersion: zesty
MirrorURL: http://nl.archive.ubuntu.com/ubuntu
 
%setup
    echo "Looking in directory '$SINGULARITY_ROOTFS' for /bin/sh"
    if [ ! -x "$SINGULARITY_ROOTFS/bin/sh" ]; then
        echo "Hrmm, this container does not have /bin/sh installed..."
        exit 1
    fi
    exit 0
 
 
%runscript
    echo "Welcome to Ubuntu 17.04"
    exec python3.5 $@
 
%post
    echo "Adding multiverse repo"
    sed -i 's/$/ universe/' /etc/apt/sources.list
    sed -i 's/$/ multiverse/' /etc/apt/sources.list
    apt update
    apt upgrade
    apt install -y python3 python3-pip
    pip3 install numpy pandas matplotlib
    mkdir -p /data/<peregrine_username>

Finally, copy the image to Peregrine and run it (either interactively or through a batch job as before):

scp tutorial.img <peregrine_username>@peregrine.hpc.rug.nl:/data/<peregrine_username>/SingularityTutorial
ssh <peregrine_username>@peregine.hpc.rug.nl
cd /data/$USER/SingularityTutorial
singularity run --bind /data/$USER/:/data/$USER/ tutorial.img script.py inputfile.csv

Back to exercise 6
To exercise 7

Create the mount point in the container:

sudo singularity shell --shell /bin/bash --writable tutorial.img
    mkdir /usr/lib64/nvidia
    exit

Copy it to Peregrine, log in, change to the right directory:

scp tutorial.img <peregrine_username>@peregrine.hpc.rug.nl:/data/<peregrine_username>/SingularityTutorial
ssh <peregrine_username>@peregine.hpc.rug.nl
cd /data/$USER/SingularityTutorial

Now put the CUDA source code in a file, e.g. hello_world.cu, and compile it:

module load CUDA/8.0.61
nvcc -o hello_world hello_world.cu

Create a job script using your favorite text editor

nano jobscript_gpu.sh

with the following contents:

#!/bin/bash
#SBATCH --time=00:05:00
#SBATCH --partition=gpu
#SBATCH --gres=gpu:1
 
export LD_LIBRARY_PATH=/usr/lib64/nvidia:$LD_LIBRARY_PATH
singularity exec --bind /usr/lib64/nvidia:/usr/lib64/nvidia --bind /data/$USER/:/data/$USER tutorial.img ./hello_world

Setting the LD_LIBRARY_PATH to /usr/lib64/nvidia is required to let your application find the drivers. The --bind option will make sure that the Nvidia drivers from the host are mounted in your container. We also use a bind mount to make your data directory available. In this case you could also store the CUDA application in your home directory, which is automatically mounted; in that case you don’t need the second bind option.

Finally, submit your job using:

sbatch jobscript_gpu.sh

And check the results in the slurm-*.out file.

Back to exercise 7