Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision		 Previous revision
habrok:examples:python [2023/03/22 12:57] – fokkehabrok:examples:python [2024/01/29 10:16] (current) – [GPU] pedro
Line 21: Line 21:
  
 === Building the Python Virtual Environment === === Building the Python Virtual Environment ===
-Before setting up a Python Virtual Environment, you need to first load a specific version of Python. In this example, we will use the latest version of Python, ''3.8.6'', that is available on Hábrók, but this should work for older versions as well. If you cannot follow these instructions for a specific version of Python, please let us know, and we will add special instructions for that version.+Before setting up a Python Virtual Environment, you need to first load a specific version of Python. In this example, we will use the latest version of Python, ''3.9.6'', that is available on Hábrók, but this should work for older versions as well. If you cannot follow these instructions for a specific version of Python, please let us know, and we will add special instructions for that version.
  
 We load the Python module: We load the Python module:
 <code bash> <code bash>
-module load Python/3.8.6-GCCcore-10.2.0+module load Python/3.9.6-GCCcore-11.2.0
 </code> </code>
  
Line 31: Line 31:
 <code bash> <code bash>
 python3 --version python3 --version
-Python 3.8.6+Python 3.9.6
 </code> </code>
  
 which is what we wanted. which is what we wanted.
  
-Now, we need to decide where to save the folder that contains the Python Virtual Environment we're going to build. There is no restriction on this, as long as you have the permissions, but we suggest saving it in your ''/data'' folder, since the disk and file quota on ''/data'' is larger than on ''/home'', and each Python Virtual Environment can contain several hundred files, depending on how many packages you install. Therefore, we will place all environments in ''/data/$USER/.envs''. +Now, we need to decide where to save the folder that contains the Python Virtual Environment we're going to build. There is no restriction on this, as long as you have the permissions, but we suggest saving it in your home directory, since this storage works best for directories containing many files, and each Python Virtual Environment can contain several hundred files (or more), depending on how many packages you install. Therefore, we will place all environments in ''$HOME/venvs''. 
  
 It is easy to build a Python Virtual Environment: It is easy to build a Python Virtual Environment:
 <code bash> <code bash>
-python3 -m venv /data/$USER/.envs/first_env+python3 -m venv $HOME/venvs/first_env
 </code> </code>
  
Line 49: Line 49:
 The Python Virtual Environment is now built, but we can't use it yet, first we need to //activate// it. We do this with the following command: The Python Virtual Environment is now built, but we can't use it yet, first we need to //activate// it. We do this with the following command:
 <code bash> <code bash>
-source /data/$USER/.envs/first_env/bin/activate+source $HOME/venvs/first_env/bin/activate
 </code> </code>
  
 and this will change the prompt of the command line from something like ''[p123456@login1 ~]$'' to something like ''(first_env) [p123456@login1 ~]$''. This is a really useful feature, allowing you to see, at a glance, which Python Virtual Environment you are working with. and this will change the prompt of the command line from something like ''[p123456@login1 ~]$'' to something like ''(first_env) [p123456@login1 ~]$''. This is a really useful feature, allowing you to see, at a glance, which Python Virtual Environment you are working with.
  
-The environment we just built and activated is a pristine one, and it only contains the Python packages that were available in the ''Python/3.8.6-GCCcore-10.2.0'' module. However, we can now populate the environment with whatever packages we want to use in this particular project, by installing them. Before installing any additional package in the Python Virtual Environment, it might be a good idea to update ''pip'', the Python Package Installer, and wheel which is used to install binary packages:+The environment we just built and activated is a pristine one, and it only contains the Python packages that were available in the ''Python/3.9.6-GCCcore-11.2.0'' module. However, we can now populate the environment with whatever packages we want to use in this particular project, by installing them. Before installing any additional package in the Python Virtual Environment, it might be a good idea to update ''pip'', the Python Package Installer, and wheel which is used to install binary packages:
 <code bash> <code bash>
 pip install --upgrade pip pip install --upgrade pip
Line 67: Line 67:
 </code> </code>
  
-where ''package_name'' is the name of the Python package you want to install. This will install the package into the Python Virtual Environment folder ''/data/$USER/.envs/first_env'', and the package will be available every time we activate this particular environment in the future. +where ''package_name'' is the name of the Python package you want to install. This will install the package into the Python Virtual Environment folder ''$HOME/venvs/first_env'', and the package will be available every time we activate this particular environment in the future. 
  
 It is considered good practice to save the names of all the packages you wish to install in a text file (usually called ''requirements.txt'') and use that file to install the packages all at once with the command: It is considered good practice to save the names of all the packages you wish to install in a text file (usually called ''requirements.txt'') and use that file to install the packages all at once with the command:
Line 86: Line 86:
 #!/bin/bash #!/bin/bash
 #SBATCH --time=00:01:00 #SBATCH --time=00:01:00
-#SBATCH --partition=short+#SBATCH --partition=regular
  
 module purge module purge
-module load Python/3.8.6-GCCcore-10.2.0+module load Python/3.9.6-GCCcore-11.2.0
  
-source /data/$USER/.envs/first_env/bin/activate+source $HOME/venvs/first_env/bin/activate
  
 python3 --version python3 --version
Line 103: Line 103:
 </code> </code>
  
-This jobscript will first purge your module environment, then load the correct version of Python (you always have to load the Python module before activating your Python Virtual Environment), and then it activates your environment. Once the environment is activated, we check the version of Python, and the location of the Python executable, which should be ''/data/$USER/.envs/first_env/bin/python3'', the location of your environment. In place of these commands which only give you some information, you can, of course, run your own Python scripts.+This jobscript will first purge your module environment, then load the correct version of Python (you always have to load the Python module before activating your Python Virtual Environment), and then it activates your environment. Once the environment is activated, we check the version of Python, and the location of the Python executable, which should be ''$HOME/venvs/first_env/bin/python3'', the location of your environment. In place of these commands which only give you some information, you can, of course, run your own Python scripts.
  
 Deactivating the Python Virtual Environment isn't strictly necessary, since the job ends after that in any case. Deactivating the Python Virtual Environment isn't strictly necessary, since the job ends after that in any case.
Line 320: Line 320:
  
 <code bash> <code bash>
-module load Python/3.6.4-foss-2018a +module load Python/3.10.4-GCCcore-11.3.0 
-module load CUDA/9.1.85 +module load CUDA/11.7.0 
-module load Boost/1.66.0-foss-2018a-Python-3.6.4+module load Boost/1.79.0-GCC-11.3.0
 pip install pycuda --user pip install pycuda --user
 </code> </code>
Line 335: Line 335:
 #SBATCH --gres=gpu:1 #SBATCH --gres=gpu:1
 #SBATCH --mem=8000 #SBATCH --mem=8000
-module load Python/3.6.4-foss-2018a+module load Python/3.10.4-GCCcore-11.3.0 
 +module load CUDA/11.7.0 
 +module load Boost/1.79.0-GCC-11.3.0
 python ./python_gpu.py python ./python_gpu.py
 </code> </code>