Ollama on Habrok

You can run an LLM on Habrok with Ollama in a Jupyter environment by using the Ollama (Jupyter) Interactive App on the Web Portal.

The app starts Jupyter inside a Python virtual environment and loads Python/3.13.1-GCCcore-14.2.0. There are two options:

• Default: a read-only environment maintained by the HPC team at /scratch/public/venvs/ollama, providing the jupyter, jupyterlab, ollama and openai packages. There is nothing to set up, but you cannot install extra packages into it.
• Custom virtual environment: your own venv, for when you need additional packages. Build it against the same Python module so that it matches the environment the app runs in:

module purge
module load Python/3.13.1-GCCcore-14.2.0
python3 -m venv $HOME/venvs/ollama
source $HOME/venvs/ollama/bin/activate
pip install --upgrade pip
pip install jupyter ollama openai

Because Jupyter is started from inside the selected environment, its default Python 3 kernel already sees these packages, and you do not need to register a separate kernel. If you followed earlier instructions and ran ipykernel install --user --name=ollama, that kernel still points at your old environment; remove it with jupyter kernelspec remove ollama to avoid confusion.

Only CUDA-enabled builds of Ollama are offered, because the CPU-only modules would run inference on the CPU without any warning, even on a GPU node. The default, ollama/0.24.0-GCCcore-14.2.0-CUDA-12.8.0, is the right choice unless you specifically need to reproduce older results. The list of versions is filtered by the node type you select, because the software stack is built separately for each CPU architecture:

• A100 nodes (Intel Ice Lake): all three versions are available.
• V100 nodes (Intel Skylake): Ollama 0.24.0 is not built for this architecture, so only the 0.6.0 builds are offered. Nothing is lost by this, as V100s are Volta GPUs and every listed build supports them.

The RTX PRO 6000 nodes are not currently offered. The Ollama modules in the Zen 3 tree that serves those nodes are compiled for Volta and Ada GPUs only, and the RTX PRO 6000 is Blackwell. Ollama does not report this as an error: it drops the GPU and runs the model on the CPU, which for a 12B model is slow enough to look like a frozen session. The node type will come back once the module is rebuilt for that GPU architecture.

The Ollama server runs independently of the Python environment - it is a self-contained binary that the ollama and openai packages talk to over HTTP - so any Ollama version can be combined with any virtual environment. If the server fails to start, the session log contains the last lines of ollama.log.

The Python ollama package and the Ollama server are versioned independently. At the time of writing the client is at 0.6.2 while the server module is 0.24.0; this is not a mismatch, it is simply two release streams that both happen to sit in 0.x. The client is a thin wrapper around the server's HTTP API, so the two version numbers are not expected to agree. A client that is newer than the server can expose options the server does not understand yet, but the chat, generate, pull and list calls used below, and the OpenAI-compatible endpoint, work across that gap.

Another important choice when running the app is where the Ollama models should be saved; there are two options, with advantages and drawbacks:

• Custom directory: This is a folder on the shared filesystem (we recommend /scratch/$USER since the models are quite large) where the models can be downloaded and saved for use in the future. This way, you only need to download a model once, but it will take quite a bit of time to save the model files to the shared filesystem. It might also be slower to use a model saved here.
• Temporary directory: This is a folder on the local disk of the node running the job, and is it considerably faster to save a downloaded model here; the drawback is that this is not persistent storage, and the model files will have to be downloaded for each session. However, it might also be a bit faster when using the model.

The Memory field asks for host RAM, not GPU memory. Ollama reads the entire model file into RAM before handing it to the GPU, so the amount requested has to cover the model itself. A good rule of thumb is the model's on-disk size plus about 8 GB; a 12B model is roughly 8 GB on disk, so 16 GB is a sensible starting point.

Requesting too little does not produce a clear error. The server starts normally, the model downloads, and only the first inference request fails:

InternalServerError: Error code: 500 - {'error': {'message': 'llama runner
process has terminated: signal: killed', ...

That signal: killed is the job being terminated for exceeding its memory allocation, not a problem with the model. If you see it, start a new session with more memory.

Ollama falls back to running a model on the CPU whenever it cannot use the GPU, and it does so without reporting an error. A 12B model on the CPU produces a few tokens per second, so the symptom is a request that appears to hang rather than one that fails.

To check which it is, look at ollama.log in the session directory:

grep -E 'offloaded|model weights device' ollama.log

offloaded 49/49 layers to GPU and model weights device=CUDA0 mean the GPU is being used. offloaded 0/49 and device=CPU mean it is not, and the session is worth restarting rather than waiting on. The app checks for the known cause of this - a module built for different GPU architectures than the node provides - and refuses to start rather than let the job run on the CPU unnoticed.

To use Ollama in the Jupyter app, open a new notebook using the Python 3 kernel. Here is a small example which first imports the necessary packages:

import os
import ollama
 
from openai import OpenAI

then downloads a model from Ollama:

ollama.pull("gemma3:12b")

and also lists all currently downloaded models:

for model in ollama.list().models:
    print(model.model)

It then creates a OpenAI API client:

client = OpenAI(
    base_url=f"http://{os.environ['OLLAMA_HOST']}/v1",
    api_key="ollama"
)

and interacts with the LLM:

response = client.chat.completions.create(
    model="gemma3:12b",
    messages = [
        {
            "role": "system",
            "content": "You are a friendly dog"
        },
        {
            "role": "user",
            "content": "Would you like a bone?"
        }
    ]
)
print(response.choices[0].message.content)

The model can, if desired, be deleted:

ollama.delete("gemma3:12b")

You can find more info on how to use the Ollama Python library on their GitHub page.