Differences

This shows you the differences between two versions of the page.

--- habrok:python:loading_python [2026/03/19 13:44] – created vlada
+++ habrok:python:loading_python [2026/03/19 13:45] (current) – removed vlada
@@ Line 1: / Line 1: @@
-====== Loading Python ======
-Python on Hábrók is not available by default, it is provided through the module system, which allows multiple versions to coexist on the cluster without interfering with each other. Before you can use Python or install any packages, you need to load the version you want to work with. The main take home message is: load a Python module for the version you need, then create a virtual environment for any packages not already available through the module system. The rest of this page explains how to do both.
-===== Loading a Python module =====
-On Hábrók, several versions of Python are available through the module system. To see which versions are available, run:
-<code bash>
-module avail Python
-</code>
-To load a specific version, for example:
-<code bash>
-module load Python/3.9.6-GCCcore-11.2.0
-</code>
-You can verify the version that was loaded with:
-<code bash>
-python3 --version
-</code>
-===== Libraries within the Python module =====
-<WRAP center round tip>
-**TLDR**
-If you need to use a specific Python library on Hábrók, don't just ''pip install'' it, as what you will get will not be an optimized version. First, check whether the library is already available from the specific version of Python you loaded.
-If it is not, check whether the library is installed on Hábrók as a module with ''module avail library_name''. When using multiple libraries via the module system, pay attention to the Python and toolchain versions. Only if you've not been able to find the library, should you consider installing it via ''pip'' and a virtual environment.
-</WRAP>
-The Python ecosystem is extremely varied, with a lot of libraries for all sorts of purposes, from web servers, to numerical computing, and everything in between and to the sides.
-As a Python user, you would usually install these libraries with ''pip'', the Python Package Installer. You can still do that on Hábrók, as we have detailed in the [[habrok_internal:drafts:python:environments:pip_venv|pip + venv]] page, but this is not always the best way, because ''pip'' doesn't optimize the libraries for the particular machines they would be running on. In an HPC environment, performance is key, especially for numerical libraries.
-The Python module itself comes with a host of libraries already installed (optimally), so that is the first place to look for a specific library. You can check what is included with:
-<code bash>
-module whatis Python/3.7.4-GCCcore-8.3.0
-</code>
-which gives the following output:
-<code>
-Python/3.7.4-GCCcore-8.3.0                            : Description: Python is a programming language that lets you work more quickly and integrate your systems more effectively.
-Python/3.7.4-GCCcore-8.3.0                            : Homepage: https://python.org/
-Python/3.7.4-GCCcore-8.3.0                            : URL: https://python.org/
-Python/3.7.4-GCCcore-8.3.0                            : Extensions: alabaster-0.7.12, asn1crypto-0.24.0, atomicwrites-1.3.0, attrs-19.1.0, Babel-2.7.0, bcrypt-3.1.7, bitstring-3.1.6, blist-1.3.6, certifi-2019.9.11, cffi-1.12.3, chardet-3.0.4, Click-7.0, cryptography-2.7, Cython-0.29.13, deap-1.3.0, decorator-4.4.0, docopt-0.6.2, docutils-0.15.2, ecdsa-0.13.2, future-0.17.1, idna-2.8, imagesize-1.1.0, importlib_metadata-0.22, ipaddress-1.0.22, Jinja2-2.10.1, joblib-0.13.2, liac-arff-2.4.0, MarkupSafe-1.1.1, mock-3.0.5, more-itertools-7.2.0, netaddr-0.7.19, netifaces-0.10.9, nose-1.3.7, packaging-19.1, paramiko-2.6.0, pathlib2-2.3.4, paycheck-1.0.2, pbr-5.4.3, pip-19.2.3, pluggy-0.13.0, psutil-5.6.3, py-1.8.0, py_expression_eval-0.3.9, pyasn1-0.4.7, pycparser-2.19, pycrypto-2.6.1, Pygments-2.4.2, PyNaCl-1.3.0, pyparsing-2.4.2, pytest-5.1.2, python-dateutil-2.8.0, pytz-2019.2, requests-2.22.0, scandir-1.10.0, setuptools-41.2.0, setuptools_scm-3.3.3, six-1.12.0, snowballstemmer-1.9.1, Sphinx-2.2.0, sphinxcontrib-applehelp-1.0.1, sphinxcontrib-devhelp-1.0.1, sphinxcontrib-htmlhelp-1.0.2, sphinxcontrib-jsmath-1.0.1, sphinxcontrib-qthelp-1.0.2, sphinxcontrib-serializinghtml-1.1.3, sphinxcontrib-websupport-1.1.2, tabulate-0.8.3, ujson-1.35, urllib3-1.25.3, virtualenv-16.7.5, wcwidth-0.1.7, wheel-0.33.6, xlrd-1.2.0, zipp-0.6.0
-</code>
-All these libraries will be available to you when you load the Python module with ''module load Python/3.7.4-GCCcore-8.3.0''.
-===== Libraries as modules =====
-If the library you want is not listed here, it might be that we have it installed as a module on Hábrók, in an **optimized** version. We've done this for several common libraries, and we **strongly encourage** you to use these modules, rather than ''pip install'' the libraries. Doing so can speed up your computation significantly. Below, we present a list of the Python libraries which are installed as modules on Hábrók:
-  * TensorFlow
-  * SciPy-bundle: numpy, scipy, pandas, mpi4py, mpmath
-  * scikit-learn, scikit-image
-  * matplotlib
-  * PyTorch
-  * Numba
-  * Tkinter (only usable with portal or X server forwarding)
-  * h5py
-<WRAP center round important>
-**Caveat** Unfortunately packages installed outside of the main Python installation, and reached through the environment variable ''PYTHONPATH'' cannot be upgraded inside a virtual environment. This issue is often encountered with the packages from ''SciPy-bundle''.
-In these cases it may be better to stick to only loading the Python module itself in the version you need and installing additional modules yourself in the virtual environment.
-Another trick, if you do want to use a module which imports ''SciPy-bundle'' is to unload the ''SciPy-bundle'' afterwards and to install numpy, scipy, pandas and any other required packages in your virtual environment instead.
-</WRAP>
-This is not an exhaustive list, please check with ''module avail'' to see if a module you are looking for is available before installing it with ''pip''. This only applies to large, well known libraries, however, so don't make a pain for yourself searching every single package you intend to import.
-To find out which versions of these libraries are available on Hábrók, you can use the ''module avail'' command, e.g.
-<code bash>
-module avail TensorFlow
-</code>
-which will produce something like the following output:
-<code>
------------------------------------------------- /software/modules/lib ------------------------------------------------
-   TensorFlow/1.6.0-foss-2018a-Python-3.6.4-CUDA-9.1.85    TensorFlow/1.12.0-foss-2018a-Python-2.7.14
-   TensorFlow/1.6.0-foss-2018a-Python-3.6.4                TensorFlow/1.12.0-foss-2018a-Python-3.6.4
-   TensorFlow/1.8.0-foss-2018a-Python-3.6.4                TensorFlow/1.12.0-fosscuda-2018a-Python-2.7.14
-   TensorFlow/1.8.0-fosscuda-2018a-Python-3.6.4            TensorFlow/1.12.0-fosscuda-2018a-Python-3.6.4
-   TensorFlow/1.9.0-foss-2018a-Python-3.6.4-CUDA-9.1.85    TensorFlow/1.15.2-fosscuda-2019b-Python-3.7.4
-   TensorFlow/1.9.0-foss-2018a-Python-3.6.4                TensorFlow/2.0.0-foss-2019a-Python-3.7.2
-   TensorFlow/1.10.1-foss-2018a-Python-3.6.4               TensorFlow/2.1.0-fosscuda-2019b-Python-3.7.4
-   TensorFlow/1.10.1-fosscuda-2018a-Python-2.7.14          TensorFlow/2.2.0-fosscuda-2019b-Python-3.7.4
-   TensorFlow/1.10.1-fosscuda-2018a-Python-3.6.4           TensorFlow/2.3.1-fosscuda-2019b-Python-3.7.4   (D)
-  Where:
-   D:  Default Module
-Use "module spider" to find all possible modules and extensions.
-Use "module keyword key1 key2 ..." to search for all possible modules matching any of the "keys".
-</code>
-You can then load a specific version with ''module load'', e.g.:
-<code bash>
-module load TensorFlow/2.3.1-fosscuda-2019b-Python-3.7.4
-</code>
-TensorFlow loads a bunch of other modules on which it depends. You can check which modules are loaded with
-<code bash>
-module list
-</code>
-and that will give you the following list of almost 50 modules:
-<code>
-Currently Loaded Modules:
-) GCCcore/8.3.0                    25) GMP/6.1.2-GCCcore-8.3.0
-) zlib/1.2.11-GCCcore-8.3.0        26) libffi/3.2.1-GCCcore-8.3.0
-) binutils/2.32-GCCcore-8.3.0      27) Python/3.7.4-GCCcore-8.3.0
-) GCC/8.3.0                        28) SciPy-bundle/2019.10-fosscuda-2019b-Python-3.7.4
-) CUDA/10.1.243-GCC-8.3.0          29) Szip/2.1.1-GCCcore-8.3.0
-) gcccuda/2019b                    30) HDF5/1.10.5-gompic-2019b
-) numactl/2.0.12-GCCcore-8.3.0     31) h5py/2.10.0-fosscuda-2019b-Python-3.7.4
-) XZ/5.2.4-GCCcore-8.3.0           32) cURL/7.66.0-GCCcore-8.3.0
-) libxml2/2.9.9-GCCcore-8.3.0      33) double-conversion/3.1.4-GCCcore-8.3.0
-) libpciaccess/0.14-GCCcore-8.3.0  34) flatbuffers/1.12.0-GCCcore-8.3.0
-) hwloc/1.11.12-GCCcore-8.3.0      35) giflib/5.2.1-GCCcore-8.3.0
-) OpenMPI/3.1.4-gcccuda-2019b      36) ICU/64.2-GCCcore-8.3.0
-) OpenBLAS/0.3.7-GCC-8.3.0         37) JsonCpp/1.9.3-GCCcore-8.3.0
-) gompic/2019b                     38) NASM/2.14.02-GCCcore-8.3.0
-) FFTW/3.3.8-gompic-2019b          39) libjpeg-turbo/2.0.3-GCCcore-8.3.0
-) ScaLAPACK/2.0.2-gompic-2019b     40) LMDB/0.9.24-GCCcore-8.3.0
-) fosscuda/2019b                   41) nsync/1.24.0-GCCcore-8.3.0
-) cuDNN/7.6.4.38-gcccuda-2019b     42) PCRE/8.43-GCCcore-8.3.0
-) NCCL/2.4.8-gcccuda-2019b         43) protobuf/3.10.0-GCCcore-8.3.0
-) bzip2/1.0.8-GCCcore-8.3.0        44) protobuf-python/3.10.0-fosscuda-2019b-Python-3.7.4
-) ncurses/6.1-GCCcore-8.3.0        45) libpng/1.6.37-GCCcore-8.3.0
-) libreadline/8.0-GCCcore-8.3.0    46) snappy/1.1.7-GCCcore-8.3.0
-) Tcl/8.6.9-GCCcore-8.3.0          47) SWIG/4.0.1-GCCcore-8.3.0
-) SQLite/3.29.0-GCCcore-8.3.0      48) TensorFlow/2.3.1-fosscuda-2019b-Python-3.7.4
-</code>
-As you can see, several of the associated Python modules that we listed above have also been loaded, e.g. ''SciPy-bundle'', as well as a specific version of Python itself, i.e. ''Python/3.7.4-GCCcore-8.3.0''.
-Associated Python modules behave just like every other module on Hábrók, which means that you need to pay careful attention to toolchain versions, ''fosscuda/2019b'', and Python versions.
-<WRAP center round important>
-**IMPORTANT**
-Make sure that all the associated Python modules you load use the same Python and toolchain versions. Using different versions of these will most likely lead to conflicts.
-</WRAP>