This is an old revision of the document!
Software environment
On Hábrók a lot of software has been made available. In order to allow multiple versions of software to be available the software is arranged in so called modules.
module system
By default none of these modules is loaded, which means that the software will not yet be available on the command line prompt. Throughout this page we will use R as an example. Normally the command R should fire up a command line session for R:
[username@habrok ~]$ R -bash: R: command not found [username@habrok ~]$
As you can see R cannot be found. In order to find the options that are available and add the software to our environment we have to make use of the module system. The command module allows for finding and loading software into the environment. For this the command has to be used together with a subcommand to specify the action we want to perform.
The subcommands are:
availfor showing a list of available softwareloadoraddfor loading software into the environmentlistfor showing the currently loaded modulesrmordelfor removing a module from the environmentpurgefor removing all loaded modules from the environmentsavefor saving a set of loaded modules into a configuration for restoring at a later timerestorefor restoring a saved set of modules
module avail
One way to find information about software is to run the command module with the subcommand avail. If we run it without extra arguments we get:
[username@habrok ~]$ module avail -------------------- /usr/share/lmod/lmod/modulefiles/Core -------------------- lmod settarg ---------------------------- /software/modules/bio ---------------------------- ABCtoolbox/1.0 ABySS/2.0.2-foss-2018a ADMIXTURE/1.3.0 AUGUSTUS/3.3.1-foss-2018a AdmixTools/2017_06_29-foss-2016a Annocript/2.0.1-foss-2018a-Perl-5.26.1 BBMap/37.93-foss-2018a BCFtools/1.3-foss-2016a BCFtools/1.9-foss-2018a (D) BEDTools/2.25.0-foss-2016a BEDTools/2.27.1-foss-2018a BEDTools/2.27.1-GCCcore-6.4.0 (D) BLAST+/2.3.0-foss-2016a-Python-2.7.11 BLAST+/2.7.1-foss-2018a (D) BLAST/2.2.22-Linux_x86_64 BLAST/2.2.26-Linux_x86_64 (D) BOLT-LMM/2.3.4 --More--
We can use the spacebar to scroll through the extensive list. As you can see the list contains software packages. For each package one or more versions is available.
In order to bring down the list to certain packages only we can specify an argument to module avail. If we specify 'R', or 'r' as an argument it will show us all packages with an 'r' in its name. This will not help much, so we have to be more precise. A more useful search string would be 'r/3':
[username@habrok ~]$ module avail r/3 ---------------------------- /software/modules/bio ---------------------------- HMMER/3.1b2-foss-2018a --------------------------- /software/modules/lang ---------------------------- R/3.2.3-foss-2016a R/3.3.1-foss-2016a R/3.3.3-foss-2016a-X11-20160819 R/3.4.2-foss-2016a-X11-20160819 R/3.4.4-foss-2018a-X11-20180131 R/3.5.0-foss-2018a-X11-20180131 R/3.6.1-foss-2018a (D) Tkinter/3.6.4-foss-2018a-Python-3.6.4 Tkinter/3.6.4-fosscuda-2018a-Python-3.6.4 Tkinter/3.7.2-GCCcore-8.2.0 Tkinter/3.7.4-GCCcore-8.3.0 (D) --------------------------- /software/modules/math ---------------------------- MPFR/3.1.4-foss-2016a Where: D: Default Module --More--
Now we can see several versions of R that are available. One of the things that are shown is that the packages that have '(D)' behind the name are the default versions.
module load/add
In order to load R, we can use the load or equivalent add subcommand for module. Note that we have to be precise with respect to upper- and lowercase now.
[username@habrok ~]$ module load R [username@habrok ~]$ R --version R version 3.6.1 (2019-07-05) -- "Action of the Toes" Copyright (C) 2019 The R Foundation for Statistical Computing Platform: x86_64-pc-linux-gnu (64-bit) R is free software and comes with ABSOLUTELY NO WARRANTY. You are welcome to redistribute it under the terms of the GNU General Public License versions 2 or 3. For more information about these matters see https://www.gnu.org/licenses/. [username@habrok ~]$
As can be seen from the last output R version 3.6.1 is now in our environment. If no version is specified the default version of the package is loaded.
CAVEAT Although packages have default versions, these defaults are subject to change without notice. It is therefore better to be explicit in the version that is to be loaded, especially in jobscripts. Otherwise you may find out that jobscripts break when the default suddenly changes.
For loading a specific version the full version has to be specified like:
[username@habrok ~]$ module load R/3.5.0-foss-2018a-X11-20180131 [username@habrok ~]$ R --version R version 3.5.0 (2018-04-23) -- "Joy in Playing" Copyright (C) 2018 The R Foundation for Statistical Computing Platform: x86_64-pc-linux-gnu (64-bit) R is free software and comes with ABSOLUTELY NO WARRANTY. You are welcome to redistribute it under the terms of the GNU General Public License versions 2 or 3. For more information about these matters see http://www.gnu.org/licenses/. [username@habrok ~]$
module spider
Another way to find out more information about available packages is through module spider. In this uppercase R has to be used as well:
[username@habrok ~]$ module spider R
---------------------------------------------------------------------------
R:
---------------------------------------------------------------------------
Description:
R is a free software environment for statistical computing and
graphics.
Versions:
R/3.2.3-foss-2016a
R/3.3.1-foss-2016a
R/3.3.3-foss-2016a-X11-20160819
R/3.4.2-foss-2016a-X11-20160819
R/3.4.4-foss-2018a-X11-20180131
R/3.5.0-foss-2018a-X11-20180131
R/3.6.1-foss-2018a
Other possible modules matches:
ADMIXTURE ANTLR AmberTools Annocript Armadillo BioPerl ...
---------------------------------------------------------------------------
To find other possible module matches execute:
--More--
This command shows some extra information on the package.
module list
In order to find out which modules have been loaded the subcommand list can be used:
[username@habrok ~]$ module list Currently Loaded Modules: 1) GCCcore/6.4.0 20) nettle/3.4-foss-2018a 39) LibTIFF/4.0.9-GCCcore-6.4.0 2) binutils/2.28-GCCcore-6.4.0 21) libdrm/2.4.91-GCCcore-6.4.0 40) Java/1.8.0_162 3) GCC/6.4.0-2.28 22) ncurses/6.0-GCCcore-6.4.0 41) Tk/8.6.8-foss-2018a 4) numactl/2.0.11-GCCcore-6.4.0 23) LLVM/5.0.1-GCCcore-6.4.0 42) cURL/7.58.0-GCCcore-6.4.0 5) hwloc/1.11.8-GCCcore-6.4.0 24) Mesa/17.3.6-foss-2018a 43) libxml2/2.9.7-GCCcore-6.4.0 6) OpenMPI/2.1.2-GCC-6.4.0-2.28 25) libGLU/9.0.0-foss-2018a 44) Python/3.6.4-foss-2018a 7) OpenBLAS/0.2.20-GCC-6.4.0-2.28 26) pixman/0.34.0-GCCcore-6.4.0 45) Szip/2.1.1-GCCcore-6.4.0 8) gompi/2018a 27) libffi/3.2.1-GCCcore-6.4.0 46) HDF5/1.10.1-foss-2018a 9) FFTW/3.3.7-gompi-2018a 28) XZ/5.2.3-GCCcore-6.4.0 47) netCDF/4.6.0-foss-2018a 10) ScaLAPACK/2.0.2-gompi-2018a-OpenBLAS-0.2.20 29) gettext/0.19.8.1-GCCcore-6.4.0 48) GEOS/3.6.2-foss-2018a-Python-3.6.4 11) foss/2018a 30) PCRE/8.41-GCCcore-6.4.0 49) JasPer/2.0.14-GCCcore-6.4.0 12) bzip2/1.0.6-GCCcore-6.4.0 31) util-linux/2.31-GCCcore-6.4.0 50) PROJ/5.0.0-foss-2018a 13) expat/2.2.5-GCCcore-6.4.0 32) GLib/2.54.3-GCCcore-6.4.0 51) libgeotiff/1.4.2-foss-2018a 14) zlib/1.2.11-GCCcore-6.4.0 33) cairo/1.14.12-GCCcore-6.4.0 52) GDAL/2.2.3-foss-2018a-Python-3.6.4 15) libpng/1.6.34-GCCcore-6.4.0 34) libreadline/7.0-GCCcore-6.4.0 53) NLopt/2.4.2-foss-2018a 16) freetype/2.9-GCCcore-6.4.0 35) Tcl/8.6.8-GCCcore-6.4.0 54) libsndfile/1.0.28-GCCcore-6.4.0 17) fontconfig/2.12.6-GCCcore-6.4.0 36) SQLite/3.21.0-GCCcore-6.4.0 55) ICU/61.1-GCCcore-6.4.0 18) X11/20180131-GCCcore-6.4.0 37) NASM/2.13.03-GCCcore-6.4.0 56) R/3.5.0-foss-2018a-X11-20180131 19) GMP/6.1.2-GCCcore-6.4.0 38) libjpeg-turbo/1.5.3-GCCcore-6.4.0
As you can see not only R 3.5.0 has been loaded, but also a lot of other modules. These modules are all the tools and libraries this version of R depends on.
One of the issues that you will probably encounter is that if you want to use different modules, and load these that you run into conflitcs with respect to the underlying dependencies. More on this later.
module del/rm
In order to remove a certain module from the environment the rm or equivalent del subcommand can be used:
[username@habrok ~]$ module rm R [username@habrok ~]$ module list Currently Loaded Modules: 1) GCCcore/6.4.0 20) nettle/3.4-foss-2018a 39) LibTIFF/4.0.9-GCCcore-6.4.0 2) binutils/2.28-GCCcore-6.4.0 21) libdrm/2.4.91-GCCcore-6.4.0 40) Java/1.8.0_162 3) GCC/6.4.0-2.28 22) ncurses/6.0-GCCcore-6.4.0 41) Tk/8.6.8-foss-2018a 4) numactl/2.0.11-GCCcore-6.4.0 23) LLVM/5.0.1-GCCcore-6.4.0 42) cURL/7.58.0-GCCcore-6.4.0 5) hwloc/1.11.8-GCCcore-6.4.0 24) Mesa/17.3.6-foss-2018a 43) libxml2/2.9.7-GCCcore-6.4.0 6) OpenMPI/2.1.2-GCC-6.4.0-2.28 25) libGLU/9.0.0-foss-2018a 44) Python/3.6.4-foss-2018a 7) OpenBLAS/0.2.20-GCC-6.4.0-2.28 26) pixman/0.34.0-GCCcore-6.4.0 45) Szip/2.1.1-GCCcore-6.4.0 8) gompi/2018a 27) libffi/3.2.1-GCCcore-6.4.0 46) HDF5/1.10.1-foss-2018a 9) FFTW/3.3.7-gompi-2018a 28) XZ/5.2.3-GCCcore-6.4.0 47) netCDF/4.6.0-foss-2018a 10) ScaLAPACK/2.0.2-gompi-2018a-OpenBLAS-0.2.20 29) gettext/0.19.8.1-GCCcore-6.4.0 48) GEOS/3.6.2-foss-2018a-Python-3.6.4 11) foss/2018a 30) PCRE/8.41-GCCcore-6.4.0 49) JasPer/2.0.14-GCCcore-6.4.0 12) bzip2/1.0.6-GCCcore-6.4.0 31) util-linux/2.31-GCCcore-6.4.0 50) PROJ/5.0.0-foss-2018a 13) expat/2.2.5-GCCcore-6.4.0 32) GLib/2.54.3-GCCcore-6.4.0 51) libgeotiff/1.4.2-foss-2018a 14) zlib/1.2.11-GCCcore-6.4.0 33) cairo/1.14.12-GCCcore-6.4.0 52) GDAL/2.2.3-foss-2018a-Python-3.6.4 15) libpng/1.6.34-GCCcore-6.4.0 34) libreadline/7.0-GCCcore-6.4.0 53) NLopt/2.4.2-foss-2018a 16) freetype/2.9-GCCcore-6.4.0 35) Tcl/8.6.8-GCCcore-6.4.0 54) libsndfile/1.0.28-GCCcore-6.4.0 17) fontconfig/2.12.6-GCCcore-6.4.0 36) SQLite/3.21.0-GCCcore-6.4.0 55) ICU/61.1-GCCcore-6.4.0 18) X11/20180131-GCCcore-6.4.0 37) NASM/2.13.03-GCCcore-6.4.0 19) GMP/6.1.2-GCCcore-6.4.0 38) libjpeg-turbo/1.5.3-GCCcore-6.4.0
As you can see R has been removed from the environment. This is not true, however, for the dependencies that were loaded as well. In most cases the rm or del subcommand is therefore not very useful.
module purge
In order to completely clear the list of loaded modules the subcommand purge must be used. This is an effective way of making sure all modules are gone, before loading a new package.
[username@habrok ~]$ module purge [username@habrok ~]$ module list No modules loaded [username@habrok ~]$
module save
If you have a set of modules that you need to use more often, you can save this set into a configuration which is stored for reuse later. This can be done using the subcommand restore.
[username@habrok ~]$ module load BLAST+/2.7.1-foss-2018a [username@habrok ~]$ module load Beast/2.6.1-foss-2018a [username@habrok ~]$ module save my_favorite_modules Saved current collection of modules to: "my_favorite_modules" [username@habrok ~]$
module restore
In order to restore a set of saved modules into the environment the subcommand restore has to be used:
[username@habrok ~]$ module purge [username@habrok ~]$ module list No modules loaded [username@habrok ~]$ module restore my_favorite_modules Restoring modules from user's my_favorite_modules [username@habrok ~]$ module list Currently Loaded Modules: 1) GCCcore/6.4.0 7) OpenBLAS/0.2.20-GCC-6.4.0-2.28 13) bzip2/1.0.6-GCCcore-6.4.0 19) libjpeg-turbo/1.5.3-GCCcore-6.4.0 2) binutils/2.28-GCCcore-6.4.0 8) gompi/2018a 14) PCRE/8.41-GCCcore-6.4.0 20) LMDB/0.9.21-GCCcore-6.4.0 3) GCC/6.4.0-2.28 9) FFTW/3.3.7-gompi-2018a 15) Boost/1.66.0-foss-2018a 21) BLAST+/2.7.1-foss-2018a 4) numactl/2.0.11-GCCcore-6.4.0 10) ScaLAPACK/2.0.2-gompi-2018a-OpenBLAS-0.2.20 16) GMP/6.1.2-GCCcore-6.4.0 22) Java/1.8.0_162 5) hwloc/1.11.8-GCCcore-6.4.0 11) foss/2018a 17) libpng/1.6.34-GCCcore-6.4.0 23) beagle-lib/3.0.1-foss-2018a 6) OpenMPI/2.1.2-GCC-6.4.0-2.28 12) zlib/1.2.11-GCCcore-6.4.0 18) NASM/2.13.03-GCCcore-6.4.0 24) Beast/2.6.1-foss-2018a [username@habrok ~]$
Next section: Transferring data