Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Next revision		 Previous revision
habrok:examples:python [2022/05/25 12:36] – external edit 127.0.0.1habrok:examples:python [2024/01/29 10:16] (current) – [GPU] pedro
Line 14: Line 14:
  
 ==== Python Virtual Environments ==== ==== Python Virtual Environments ====
-On Peregrine, we have several versions of Python installed (Six versions for Python 3 only!). In addition to these bare-bones Python installations, we also have optimized versions of a handful of common Python packages (''scipy'', ''matplotlib'', etc.). However, the Python ecosystem is so large and varied, that we have no hope of installing cluster-optimized versions of even the most common Python packages. +On Hábrók, we have several versions of Python installed (Six versions for Python 3 only!). In addition to these bare-bones Python installations, we also have optimized versions of a handful of common Python packages (''scipy'', ''matplotlib'', etc.). However, the Python ecosystem is so large and varied, that we have no hope of installing cluster-optimized versions of even the most common Python packages. 
  
-As a regular user on Peregrine, **you** have the power to build you own //Python Virtual Environment//, where you can install any and all Python packages you need. A Python Virtual Environment simply consists of a folder saved somewhere you have access to, and which will contain your own copy of Python, as well as all the packages you install in the Virtual Environment. You can build several Virtual Environments (for example one for each project you're working on), each residing in its own folder, and not interfering with each other. To use any of them, you simply tell the system which folder to use. You can therefore easily switch between these Virtual Environments. +As a regular user on Hábrók, **you** have the power to build you own //Python Virtual Environment//, where you can install any and all Python packages you need. A Python Virtual Environment simply consists of a folder saved somewhere you have access to, and which will contain your own copy of Python, as well as all the packages you install in the Virtual Environment. You can build several Virtual Environments (for example one for each project you're working on), each residing in its own folder, and not interfering with each other. To use any of them, you simply tell the system which folder to use. You can therefore easily switch between these Virtual Environments. 
  
 Below, we show a short and hopefully simple guide to setting up and using a Python Virtual Environment, using the ''%%venv%%'' Python package. Below, we show a short and hopefully simple guide to setting up and using a Python Virtual Environment, using the ''%%venv%%'' Python package.
  
 === Building the Python Virtual Environment === === Building the Python Virtual Environment ===
-Before setting up a Python Virtual Environment, you need to first load a specific version of Python. In this example, we will use the latest version of Python, ''3.8.6'', that is available on Peregrine, but this should work for older versions as well. If you cannot follow these instructions for a specific version of Python, please let us know, and we will add special instructions for that version.+Before setting up a Python Virtual Environment, you need to first load a specific version of Python. In this example, we will use the latest version of Python, ''3.9.6'', that is available on Hábrók, but this should work for older versions as well. If you cannot follow these instructions for a specific version of Python, please let us know, and we will add special instructions for that version.
  
 We load the Python module: We load the Python module:
 <code bash> <code bash>
-module load Python/3.8.6-GCCcore-10.2.0+module load Python/3.9.6-GCCcore-11.2.0
 </code> </code>
  
Line 31: Line 31:
 <code bash> <code bash>
 python3 --version python3 --version
-Python 3.8.6+Python 3.9.6
 </code> </code>
  
 which is what we wanted. which is what we wanted.
  
-Now, we need to decide where to save the folder that contains the Python Virtual Environment we're going to build. There is no restriction on this, as long as you have the permissions, but we suggest saving it in your ''/data'' folder, since the disk and file quota on ''/data'' is larger than on ''/home'', and each Python Virtual Environment can contain several hundred files, depending on how many packages you install. Therefore, we will place all environments in ''/data/$USER/.envs''. +Now, we need to decide where to save the folder that contains the Python Virtual Environment we're going to build. There is no restriction on this, as long as you have the permissions, but we suggest saving it in your home directory, since this storage works best for directories containing many files, and each Python Virtual Environment can contain several hundred files (or more), depending on how many packages you install. Therefore, we will place all environments in ''$HOME/venvs''. 
  
 It is easy to build a Python Virtual Environment: It is easy to build a Python Virtual Environment:
 <code bash> <code bash>
-python3 -m venv /data/$USER/.envs/first_env+python3 -m venv $HOME/venvs/first_env
 </code> </code>
  
Line 49: Line 49:
 The Python Virtual Environment is now built, but we can't use it yet, first we need to //activate// it. We do this with the following command: The Python Virtual Environment is now built, but we can't use it yet, first we need to //activate// it. We do this with the following command:
 <code bash> <code bash>
-source /data/$USER/.envs/first_env/bin/activate+source $HOME/venvs/first_env/bin/activate
 </code> </code>
  
-and this will change the prompt of the command line from something like ''[p123456@peregrine ~]$'' to something like ''(first_env) [p123456@peregrine ~]$''. This is a really useful feature, allowing you to see, at a glance, which Python Virtual Environment you are working with.+and this will change the prompt of the command line from something like ''[p123456@login1 ~]$'' to something like ''(first_env) [p123456@login1 ~]$''. This is a really useful feature, allowing you to see, at a glance, which Python Virtual Environment you are working with.
  
-The environment we just built and activated is a pristine one, and it only contains the Python packages that were available in the ''Python/3.8.6-GCCcore-10.2.0'' module. However, we can now populate the environment with whatever packages we want to use in this particular project, by installing them. Before installing any additional package in the Python Virtual Environment, it might be a good idea to update ''pip'', the Python Package Installer, and wheel which is used to install binary packages:+The environment we just built and activated is a pristine one, and it only contains the Python packages that were available in the ''Python/3.9.6-GCCcore-11.2.0'' module. However, we can now populate the environment with whatever packages we want to use in this particular project, by installing them. Before installing any additional package in the Python Virtual Environment, it might be a good idea to update ''pip'', the Python Package Installer, and wheel which is used to install binary packages:
 <code bash> <code bash>
 pip install --upgrade pip pip install --upgrade pip
Line 67: Line 67:
 </code> </code>
  
-where ''package_name'' is the name of the Python package you want to install. This will install the package into the Python Virtual Environment folder ''/data/$USER/.envs/first_env'', and the package will be available every time we activate this particular environment in the future. +where ''package_name'' is the name of the Python package you want to install. This will install the package into the Python Virtual Environment folder ''$HOME/venvs/first_env'', and the package will be available every time we activate this particular environment in the future. 
  
 It is considered good practice to save the names of all the packages you wish to install in a text file (usually called ''requirements.txt'') and use that file to install the packages all at once with the command: It is considered good practice to save the names of all the packages you wish to install in a text file (usually called ''requirements.txt'') and use that file to install the packages all at once with the command:
Line 86: Line 86:
 #!/bin/bash #!/bin/bash
 #SBATCH --time=00:01:00 #SBATCH --time=00:01:00
-#SBATCH --partition=short+#SBATCH --partition=regular
  
 module purge module purge
-module load Python/3.8.6-GCCcore-10.2.0+module load Python/3.9.6-GCCcore-11.2.0
  
-source /data/$USER/.envs/first_env/bin/activate+source $HOME/venvs/first_env/bin/activate
  
 python3 --version python3 --version
Line 103: Line 103:
 </code> </code>
  
-This jobscript will first purge your module environment, then load the correct version of Python (you always have to load the Python module before activating your Python Virtual Environment), and then it activates your environment. Once the environment is activated, we check the version of Python, and the location of the Python executable, which should be ''/data/$USER/.envs/first_env/bin/python3'', the location of your environment. In place of these commands which only give you some information, you can, of course, run your own Python scripts.+This jobscript will first purge your module environment, then load the correct version of Python (you always have to load the Python module before activating your Python Virtual Environment), and then it activates your environment. Once the environment is activated, we check the version of Python, and the location of the Python executable, which should be ''$HOME/venvs/first_env/bin/python3'', the location of your environment. In place of these commands which only give you some information, you can, of course, run your own Python scripts.
  
 Deactivating the Python Virtual Environment isn't strictly necessary, since the job ends after that in any case. Deactivating the Python Virtual Environment isn't strictly necessary, since the job ends after that in any case.
Line 115: Line 115:
 ** TLDR ** ** TLDR **
  
-If you need to use a specific Python library on Peregrine, don't just ''pip install'' it, as what you will get will not be an optimized version. First, check whether the library is already available from the specific version of Python you loaded.  +If you need to use a specific Python library on Hábrók, don't just ''pip install'' it, as what you will get will not be an optimized version. First, check whether the library is already available from the specific version of Python you loaded.  
-If it is not, check whether the library is installed on Peregrine as a module with ''module avail library_name''. When using multiple libraries via the module system, pay attention to the Python and toolchain versions. Only if you've not been able to find the library, should you consider installing it via ''pip'' and a virtual environment.+If it is not, check whether the library is installed on Hábrók as a module with ''module avail library_name''. When using multiple libraries via the module system, pay attention to the Python and toolchain versions. Only if you've not been able to find the library, should you consider installing it via ''pip'' and a virtual environment.
 ---- ----
 The Python ecosystem is extremely varied, with a lot of libraries for all sorts of purposes, from web servers, to numerical computing, and everything in between and to the sides.  The Python ecosystem is extremely varied, with a lot of libraries for all sorts of purposes, from web servers, to numerical computing, and everything in between and to the sides. 
  
-As a Python user, you would usually install these libraries with ''pip'', the Python Package Installer. You can still do that on Peregrine, as we have detailed above, but this is not always the best way, because ''pip'' doesn't optimize the libraries for the particular machines they would be running on. In an HPC environment, performance is key, especially for numerical libraries. +As a Python user, you would usually install these libraries with ''pip'', the Python Package Installer. You can still do that on Hábrók, as we have detailed above, but this is not always the best way, because ''pip'' doesn't optimize the libraries for the particular machines they would be running on. In an HPC environment, performance is key, especially for numerical libraries. 
  
 ==== Libraries within the Python module ==== ==== Libraries within the Python module ====
Line 141: Line 141:
 ==== Libraries as modules ==== ==== Libraries as modules ====
  
-If the library you want is not listed here, it might be that we have it installed as a module on Peregrine, in an **optimized** version. We've done this for several common libraries, and we **strongly encourage** you to use these modules, rather than ''pip install'' the libraries. Doing so can speed up your computation significantly. Below, we present a list of the Python libraries which are installed as modules on Peregrine:+If the library you want is not listed here, it might be that we have it installed as a module on Hábrók, in an **optimized** version. We've done this for several common libraries, and we **strongly encourage** you to use these modules, rather than ''pip install'' the libraries. Doing so can speed up your computation significantly. Below, we present a list of the Python libraries which are installed as modules on Hábrók:
  
   * TensorFlow   * TensorFlow
Line 154: Line 154:
 This is not an exhaustive list, please check with ''module avail'' to see if a module you are looking for is available before installing it with ''pip''. This only applies to large, well known libraries, however, so don't make a pain for yourself searching every single package you intend to import. This is not an exhaustive list, please check with ''module avail'' to see if a module you are looking for is available before installing it with ''pip''. This only applies to large, well known libraries, however, so don't make a pain for yourself searching every single package you intend to import.
  
-To find out which versions of these libraries are available on Peregrine, you can use the ''module avail'' command, e.g.+To find out which versions of these libraries are available on Hábrók, you can use the ''module avail'' command, e.g.
 <code bash> <code bash>
 module avail TensorFlow module avail TensorFlow
Line 221: Line 221:
 As you can see, several of the associated Python modules that we listed above have also been loaded, e.g. ''SciPy-bundle'', as well as a specific version of Python itself, i.e. ''Python/3.7.4-GCCcore-8.3.0''. As you can see, several of the associated Python modules that we listed above have also been loaded, e.g. ''SciPy-bundle'', as well as a specific version of Python itself, i.e. ''Python/3.7.4-GCCcore-8.3.0''.
  
-Associated Python modules behave just like every other module on Peregrine, which means that you need to pay careful attention to toolchain versions, ''fosscuda/2019b'', and Python versions. +Associated Python modules behave just like every other module on Hábrók, which means that you need to pay careful attention to toolchain versions, ''fosscuda/2019b'', and Python versions. 
  
 **IMPORTANT** **IMPORTANT**
Line 234: Line 234:
 ==== Single CPU ==== ==== Single CPU ====
  
-We take the following python script example, in order to run a Python script on the Peregrine cluster:+We take the following python script example, in order to run a Python script on the Hábrók cluster:
  
 <file python python_example.py> <file python python_example.py>
Line 275: Line 275:
 ==== Multiple CPUs ==== ==== Multiple CPUs ====
  
-In this example we request 10 CPUs from the Peregrine cluster to do a simple calculation on, using all the requested CPUs. In this example an array from 0 to 10 is created and each CPU does a simple computation on each value in this array.\\+In this example we request 10 CPUs from the Hábrók cluster to do a simple calculation on, using all the requested CPUs. In this example an array from 0 to 10 is created and each CPU does a simple computation on each value in this array.\\
 Requesting resources in a batch script is done as follows: Requesting resources in a batch script is done as follows:
  
Line 317: Line 317:
 ==== GPU ==== ==== GPU ====
  
-This example shows how to submit a Python GPU job to the Peregrine cluster. In this example we make use of the pycuda library, which can be installed by typing this in the terminal:+This example shows how to submit a Python GPU job to the Hábrók cluster. In this example we make use of the pycuda library, which can be installed by typing this in the terminal:
  
 <code bash> <code bash>
-module load Python/3.6.4-foss-2018a +module load Python/3.10.4-GCCcore-11.3.0 
-module load CUDA/9.1.85 +module load CUDA/11.7.0 
-module load Boost/1.66.0-foss-2018a-Python-3.6.4+module load Boost/1.79.0-GCC-11.3.0
 pip install pycuda --user pip install pycuda --user
 </code> </code>
Line 335: Line 335:
 #SBATCH --gres=gpu:1 #SBATCH --gres=gpu:1
 #SBATCH --mem=8000 #SBATCH --mem=8000
-module load Python/3.6.4-foss-2018a+module load Python/3.10.4-GCCcore-11.3.0 
 +module load CUDA/11.7.0 
 +module load Boost/1.79.0-GCC-11.3.0
 python ./python_gpu.py python ./python_gpu.py
 </code> </code>
Line 382: Line 384:
 === Avoiding I/O Bottlenecks === === Avoiding I/O Bottlenecks ===
  
-If you are using the GPU with, say, many small image files you may notice that your jobs can take a long time to complete as the images are being read to GPU sequentially. In this case you can bypass the issue by copying your data (as an archive) to the local storage on the GPU node. To do this follow the instructions on the [[peregrine:advanced_job_management:many_file_jobs|Many File Jobs]], it describes the process in more detail.+If you are using the GPU with, say, many small image files you may notice that your jobs can take a long time to complete as the images are being read to GPU sequentially. In this case you can bypass the issue by copying your data (as an archive) to the local storage on the GPU node. To do this follow the instructions on the [[habrok:advanced_job_management:many_file_jobs|Many File Jobs]], it describes the process in more detail.
  
 ==== Multiple Nodes ==== ==== Multiple Nodes ====
  
-The next example shows how a MPI job for Python is run on the Peregrine cluster. In the SLURM batch script below three nodes are requested to do an array scatter from the master node to all nodes requested. The batch file is named ''%%python_mpi_batch.sh%%''.+The next example shows how a MPI job for Python is run on the Hábrók cluster. In the SLURM batch script below three nodes are requested to do an array scatter from the master node to all nodes requested. The batch file is named ''%%python_mpi_batch.sh%%''.
  
 <code bash python_mpi_batch.sh> <code bash python_mpi_batch.sh>
Line 454: Line 456:
 <code> <code>
 ############################################################################### ###############################################################################
-Peregrine Cluster+Hábrók Cluster
 Job 1150286 for user 'p275545' Job 1150286 for user 'p275545'
 Finished at: Wed Apr 25 11:19:30 CEST 2018 Finished at: Wed Apr 25 11:19:30 CEST 2018
Line 484: Line 486:
 ================= =================
  
-Please see this page if you want to acknowledge Peregrine in your publications:+Please see this page if you want to acknowledge Hábrók in your publications:
  
-https://wiki.hpc.rug.nl/peregrine/additional_information/scientific_output+https://wiki.hpc.rug.nl/habrok/additional_information/scientific_output
  
 ################################################################################ ################################################################################
Line 495: Line 497:
 **My program output does not appear as expected when I submit a job.** **My program output does not appear as expected when I submit a job.**
  
-There are two possibilities here. First; your program is not reaching the line where you expect to output something, this is something you will have to solve yourself and preferably test on a local machine. Second; python has buffered your output, then an unexpected crash ate the buffer. This makes debugging certain programs on Peregrine very tough because you might have produced some output but it is not shown because of the way Python handles output to the terminal or in this case the ''.job'' file. The easiest solution is to run python with the ''-u'' flag, so ''python -u <my_script> <other_arguments>''. There are also some solutions including logging and outputting to stderr instead of stdout.+There are two possibilities here. First; your program is not reaching the line where you expect to output something, this is something you will have to solve yourself and preferably test on a local machine. Second; python has buffered your output, then an unexpected crash ate the buffer. This makes debugging certain programs on Hábrók very tough because you might have produced some output but it is not shown because of the way Python handles output to the terminal or in this case the ''.job'' file. The easiest solution is to run python with the ''-u'' flag, so ''python -u <my_script> <other_arguments>''. There are also some solutions including logging and outputting to stderr instead of stdout.