====== Python Environments ====== On Hábrók, you cannot install Python packages globally. Instead, you work in isolated **Python environments** — each environment is essentially a folder in your home directory that contains its own copy of Python and any packages you install into it. This means you can have multiple environments for different projects, each with their own set of packages and versions, without them interfering with each other. Before creating any environment, you always need to load a Python module first. See the [[.:loading_python|Loading Python]] page for details. ===== Which tool should I use? ===== There are three tools available for creating and managing Python environments on Hábrók: ^ Feature ^ pip + venv ^ conda / mamba ^ uv ^ | Package installation | Yes | Yes | Yes | | Virtual environment management | Manual | Built-in | Built-in | | Lock file support | No (needs pip-tools) | Yes (via conda-lock) | Yes | | Non-Python dependencies | No | Yes | No | | Speed |Baseline | Slower than pip | 10-100x faster than pip | | Project Structure | No | No | Yes | ===== pip + venv ===== On Hábrók, we have several versions of Python installed (Six versions for Python 3 only!). In addition to these bare-bones Python installations, we also have optimized versions of a handful of common Python packages (''scipy'', ''matplotlib'', etc.). However, the Python ecosystem is so large and varied, that we have no hope of installing cluster-optimized versions of even the most common Python packages. As a regular user on Hábrók, **you** have the power to build you own //Python Virtual Environment//, where you can install any and all Python packages you need. A Python Virtual Environment simply consists of a folder saved somewhere you have access to, and which will contain your own copy of Python, as well as all the packages you install in the Virtual Environment. You can build several Virtual Environments (for example one for each project you're working on), each residing in its own folder, and not interfering with each other. To use any of them, you simply tell the system which folder to use. You can therefore easily switch between these Virtual Environments. Below, we show a short guide to setting up and using a Python Virtual Environment, using the ''%%venv%%'' Python package. ==== Building the Python Virtual Environment ==== Before setting up a Python Virtual Environment, you need to first load a specific version of Python. In this example, we will use the latest version of Python, ''3.13.5'', that is available on Hábrók, but this should work for older versions as well. If you cannot follow these instructions for a specific version of Python, please let us know, and we will add special instructions for that version. We load the Python module: module load Python/3.13.5-GCCcore-14.3.0 and check that we have the right version: python3 --version Python 3.13.5 which is what we wanted. Now, we need to decide where to save the folder that contains the Python Virtual Environment we're going to build. There is no restriction on this, as long as you have the permissions, but we suggest saving it in your home directory, since this storage works best for directories containing many files, and each Python Virtual Environment can contain several hundred files (or more), depending on how many packages you install. Therefore, we will place all environments in ''$HOME/venvs''. It is easy to build a Python Virtual Environment: python3 -m venv $HOME/venvs/first_env where ''first_env'' is the name of the environment, as well as of the folder it resides in. Give it a good descriptive name, otherwise you'll be sorry when you have 10-20 different environments. You can create as many environments as you need, each in its own folder. A common pattern is one environment per project, so that different projects can use different package versions without conflicting. For example: python3 -m venv $HOME/venvs/project_a python3 -m venv $HOME/venvs/project_b You can see all your environments at any time with: ls $HOME/venvs/ ==== Using the Python Virtual Environment ==== The Python Virtual Environment is now built, but we can't use it yet, first we need to //activate// it. We do this with the following command: source $HOME/venvs/first_env/bin/activate and this will change the prompt of the command line from something like ''[p123456@login1 ~]$'' to something like ''(first_env) [p123456@login1 ~]$''. This is a really useful feature, allowing you to see, at a glance, which Python Virtual Environment you are working with. The environment we just built and activated is a pristine one, and it only contains the Python packages that were available in the ''Python/3.13.5-GCCcore-14.3.0'' module. However, we can now populate the environment with whatever packages we want to use in this particular project, by installing them. Before installing any additional package in the Python Virtual Environment, it might be a good idea to update ''pip'', the Python Package Installer, and wheel which is used to install binary packages: pip install --upgrade pip pip install --upgrade wheel This is not strictly necessary, but it is recommended, especially for older version of Python, which also come with older versions of ''pip''. Having up-to-date versions makes sure pip and wheel work with the latest package formats. We are now ready to install additional Python packages into our Python Virtual Environment. This is as simple as pip install package_name where ''package_name'' is the name of the Python package you want to install. This will install the package into the Python Virtual Environment folder ''$HOME/venvs/first_env'', and the package will be available every time we activate this particular environment in the future. It is considered good practice to save the names of all the packages you wish to install in a text file (usually called ''requirements.txt'') and use that file to install the packages all at once with the command: pip install -r requirements.txt A typical ''requirements.txt'' file would look something like keras tqdm==4.67.1 where you can also specify a particular version of a certain packages, as for ''tqdm''. To check what is currently installed in your active environment use: pip list Or to check the details of a specific package, including its version and dependencies: pip show package_name To deactivate the environment when you are done: deactivate Note that deactivation is not strictly necessary at the end of a job script, since the environment is automatically deactivated when the job finishes. If you no longer need an environment, or if it has become broken and you want to start fresh, you can simply delete its folder: rm -rf $HOME/venvs/first_env How do we use the Python Virtual Environment we just build in a job script? Here's an example of such a jobscript: #!/bin/bash #SBATCH --time=00:01:00 #SBATCH --partition=regular module purge module load Python/3.13.5-GCCcore-14.3.0 source $HOME/venvs/first_env/bin/activate python3 --version which python3 deactivate which you can submit with sbatch jobscript.sh This jobscript will first purge your module environment, then load the correct version of Python (you always have to load the Python module before activating your Python Virtual Environment), and then it activates your environment. Once the environment is activated, we check the version of Python, and the location of the Python executable, which should be ''$HOME/venvs/first_env/bin/python3'', the location of your environment. In place of these commands which only give you some information, you can, of course, run your own Python scripts. Deactivating the Python Virtual Environment isn't strictly necessary, since the job ends after that in any case. Unfortunately, packages installed outside of the main Python installation and reached through the environment variable ''PYTHONPATH'' cannot be upgraded inside a virtual environment. This issue is often encountered with packages from ''SciPy-bundle''. In these cases it may be better to stick to only loading the Python module itself and installing numpy, scipy, pandas and any other required packages yourself in the virtual environment. Another trick, if you do want to use a module which imports ''SciPy-bundle'', is to unload ''SciPy-bundle'' afterwards and install numpy, scipy, pandas and any other required packages in your virtual environment instead. ===== uv ===== [[https://docs.astral.sh/uv/|uv]] is a modern Python package and project manager written in Rust. It is designed as a complete replacement for pip, pip-tools and venv, unifying them into a single tool. uv is significantly faster than pip because it was built with performance in mind from the ground up — it uses parallel downloading, aggressive caching, and a highly optimised dependency resolver. The key thing to understand is that uv is not just a faster pip. It is a full project manager that manages your project dependencies declaratively in a ''pyproject.toml'' file, automatically handles virtual environments, and generates a ''uv.lock'' file to ensure reproducible environments. You normally do not need to manually create or activate a virtual environment when using the uv project workflow. For more details and instructions see [[habrok_internal:drafts:uv|uv]] page. ===== Mamba ===== Mamba is a separate environment and package manager that can also manage non-Python dependencies. While installation instructions are available, we generally discourage its use on Hábrók and recommend using pip + venv or uv instead. For installation instructions and caveats see the [[habrok:examples:mamba|Mamba]] page.