Differences

This shows you the differences between two versions of the page.

--- habrok:examples:ollama [2025/10/15 11:42] – pedro
+++ habrok:examples:ollama [2026/08/27 12:52] (current) – camarocico
@@ Line 1: / Line 1: @@
-====== Ollama on Hábrók ======
+====== Ollama on Habrok ======
-You can run an LLM on Hábrók with Ollama in a Jupyter environment by using the [[https://portal.hb.hpc.rug.nl/pun/sys/dashboard/batch_connect/sys/hb-ollama/session_contexts/new|Ollama (Jupyter)]] Interactive App on the [[https://portal.hb.hpc.rug.nl|Web Portal]].
+You can run an LLM on Habrok with Ollama in a Jupyter environment by using the [[https://portal.hb.hpc.rug.nl/pun/sys/dashboard/batch_connect/sys/hb-ollama/session_contexts/new|Ollama (Jupyter)]] Interactive App on the [[https://portal.hb.hpc.rug.nl|Web Portal]].
-===== Setting up the virtual environment =====
-To be able to use the app, you first need to set up a Python virtual environment. The version of Ollama installed on Hábrók is ''ollama/0.6.0-GCCcore-12.3.0'', which means that the Python virtual environment needs to use a version of Python using the same ''GCCcore-12.3.0'' toolchain, which is ''Python/3.11.3-GCCcore-12.3.0''. Other versions of Python might work as well, but toolchain compatibilities can sometimes be an issue.
+===== Choosing a Python environment =====
-<code shell>
+The app starts Jupyter inside a Python virtual environment and loads ''%%Python/3.13.1-GCCcore-14.2.0%%''. There are two options:
-module load Python/3.11.3-GCCcore-12.3.0
-python3 -m venv $HOME/venvs/ollama
-</code>
-Once the virtual environment has been built, you need to install ''jupyter'' and ''ollama'' within the virtual environment; optionally, you can also install additional packages, such as ''openai'':
+  * **Default**: a read-only environment maintained by the HPC team at ''%%/scratch/public/venvs/ollama%%'', providing the ''%%jupyter%%'', ''%%jupyterlab%%'', ''%%ollama%%'' and ''%%openai%%'' packages. There is nothing to set up, but you cannot install extra packages into it.
+  * **Custom virtual environment**: your own ''%%venv%%'', for when you need additional packages. Build it against the same Python module so that it matches the environment the app runs in:
 <code bash>
+module purge
+module load Python/3.13.1-GCCcore-14.2.0
+python3 -m venv $HOME/venvs/ollama
 source $HOME/venvs/ollama/bin/activate
 pip install --upgrade pip
@@ Line 19: / Line 19: @@
 </code>
-Finally, to make sure that the Jupyter Notebook is aware of your virtual environment, you need to create a Jupyter kernel:
+Because Jupyter is started from inside the selected environment, its default ''%%Python 3%%'' kernel already sees these packages, and you do not need to register a separate kernel. If you followed earlier instructions and ran ''%%ipykernel install --user --name=ollama%%'', that kernel still points at your old environment; remove it with ''%%jupyter kernelspec remove ollama%%'' to avoid confusion.
-<code bash>
-python3 -m ipykernel install --user --name=ollama --display-name="Ollama"
+===== Choosing an Ollama version =====
-</code>
+Only CUDA-enabled builds of Ollama are offered, because the CPU-only modules would run inference on the CPU without any warning, even on a GPU node. The default, ''%%ollama/0.24.0-GCCcore-14.2.0-CUDA-12.8.0%%'', is the right choice unless you specifically need to reproduce older results. The list of versions is filtered by the node type you select, because the software stack is built separately for each CPU architecture:
+  * **A100 nodes** (Intel Ice Lake): all three versions are available.
+  * **V100 nodes** (Intel Skylake): Ollama 0.24.0 is not built for this architecture, so only the 0.6.0 builds are offered. Nothing is lost by this, as V100s are Volta GPUs and every listed build supports them.
+The RTX PRO 6000 nodes are **not currently offered**. The Ollama modules in the Zen 3 tree that serves those nodes are compiled for Volta and Ada GPUs only, and the RTX PRO 6000 is Blackwell. Ollama does not report this as an error: it drops the GPU and runs the model on the CPU, which for a 12B model is slow enough to look like a frozen session. The node type will come back once the module is rebuilt for that GPU architecture.
+The Ollama server runs independently of the Python environment - it is a self-contained binary that the ''%%ollama%%'' and ''%%openai%%'' packages talk to over HTTP - so any Ollama version can be combined with any virtual environment. If the server fails to start, the session log contains the last lines of ''%%ollama.log%%''.
+**The Python ''%%ollama%%'' package and the Ollama server are versioned independently.** At the time of writing the client is at ''%%0.6.2%%'' while the server module is ''%%0.24.0%%''; this is not a mismatch, it is simply two release streams that both happen to sit in ''%%0.x%%''. The client is a thin wrapper around the server's HTTP API, so the two version numbers are not expected to agree. A client that is newer than the server can expose options the server does not understand yet, but the ''%%chat%%'', ''%%generate%%'', ''%%pull%%'' and ''%%list%%'' calls used below, and the OpenAI-compatible endpoint, work across that gap.
 ===== Choosing a folder for the models =====
 Another important choice when running the app is where the Ollama models should be saved; there are two options, with advantages and drawbacks:
-  * **Custom directory**: This is a folder on the shared filesystem (we recommend ''/scratch/$USER'' since the models are quite large) where the models can be downloaded and saved for use in the future. This way, you only need to download a model once, but it will take quite a bit of time to save the model files to the shared filesystem. It might also be slower to use a model saved here.
-  * **Temporary directory**: This is a folder on the local disc of the node running the job, and is it considerably faster to save a downloaded model here; the drawback is that this is not persistent storage, and the model files will have to be downloaded for each session. However, it might also be a bit faster when using the model.
+  * **Custom directory**: This is a folder on the shared filesystem (we recommend ''%%/scratch/$USER%%'' since the models are quite large) where the models can be downloaded and saved for use in the future. This way, you only need to download a model once, but it will take quite a bit of time to save the model files to the shared filesystem. It might also be slower to use a model saved here.
+  * **Temporary directory**: This is a folder on the local disk of the node running the job, and is it considerably faster to save a downloaded model here; the drawback is that this is not persistent storage, and the model files will have to be downloaded for each session. However, it might also be a bit faster when using the model.
+===== Choosing how much memory to request =====
+The ''%%Memory%%'' field asks for **host RAM**, not GPU memory. Ollama reads the entire model file into RAM before handing it to the GPU, so the amount requested has to cover the model itself. A good rule of thumb is the model's on-disk size plus about 8 GB; a 12B model is roughly 8 GB on disk, so 16 GB is a sensible starting point.
+Requesting too little does not produce a clear error. The server starts normally, the model downloads, and only the first inference request fails:
+<code>
+InternalServerError: Error code: 500 - {'error': {'message': 'llama runner
+process has terminated: signal: killed', ...
+</code>
+That ''%%signal: killed%%'' is the job being terminated for exceeding its memory allocation, not a problem with the model. If you see it, start a new session with more memory.
+===== If generation is unexpectedly slow =====
+Ollama falls back to running a model on the CPU whenever it cannot use the GPU, and it does so without reporting an error. A 12B model on the CPU produces a few tokens per second, so the symptom is a request that appears to hang rather than one that fails.
+To check which it is, look at ''%%ollama.log%%'' in the session directory:
+<code bash>
+grep -E 'offloaded|model weights device' ollama.log
+</code>
+''%%offloaded 49/49 layers to GPU%%'' and ''%%model weights device=CUDA0%%'' mean the GPU is being used. ''%%offloaded 0/49%%'' and ''%%device=CPU%%'' mean it is not, and the session is worth restarting rather than waiting on. The app checks for the known cause of this - a module built for different GPU architectures than the node provides - and refuses to start rather than let the job run on the CPU unnoticed.
 ===== Simple usage example =====
-To use Ollama in the Jupyter app, you need first to open a new notebook in the Jupyter app, and choose the **Ollama** Jupyter kernel built when setting up the virtual environment. Here is a small example which first imports the necessary packages:
+To use Ollama in the Jupyter app, open a new notebook using the **Python 3** kernel. Here is a small example which first imports the necessary packages:
 <code python>
 import os
@@ Line 37: / Line 76: @@
 from openai import OpenAI
 </code>
 then downloads a model from Ollama:
 <code python>
 ollama.pull("gemma3:12b")
 </code>
 and also lists all currently downloaded models:
 <code python>
 for model in ollama.list().models:
@@ Line 48: / Line 91: @@
 It then creates a OpenAI API client:
 <code python>
 client = OpenAI(
@@ Line 54: / Line 98: @@
 )
 </code>
 and interacts with the LLM:
 <code python>
 response = client.chat.completions.create(
@@ Line 73: / Line 119: @@
 The model can, if desired, be deleted:
 <code python>
 ollama.delete("gemma3:12b")