Ollama on Habrok

You can run an LLM on Habrok with Ollama in a Jupyter environment by using the Ollama (Jupyter) Interactive App on the Web Portal.

To be able to use the app, you first need to set up a Python virtual environment. The version of Ollama installed on Habrok is ollama/0.6.0-GCCcore-12.3.0, which means that the Python virtual environment needs to use a version of Python using the same GCCcore-12.3.0 toolchain, which is Python/3.11.3-GCCcore-12.3.0. Other versions of Python might work as well, but toolchain compatibilities can sometimes be an issue.

module load Python/3.11.3-GCCcore-12.3.0
python3 -m venv $HOME/venvs/ollama

Once the virtual environment has been built, you need to install jupyter and ollama within the virtual environment; optionally, you can also install additional packages, such as openai:

source $HOME/venvs/ollama/bin/activate
pip install --upgrade pip
pip install jupyter ollama openai

Finally, to make sure that the Jupyter Notebook is aware of your virtual environment, you need to create a Jupyter kernel:

python3 -m ipykernel install --user --name=ollama --display-name="Ollama"

Another important choice when running the app is where the Ollama models should be saved; there are two options, with advantages and drawbacks:

• Custom directory: This is a folder on the shared filesystem (we recommend /scratch/$USER since the models are quite large) where the models can be downloaded and saved for use in the future. This way, you only need to download a model once, but it will take quite a bit of time to save the model files to the shared filesystem. It might also be slower to use a model saved here.
• Temporary directory: This is a folder on the local disc of the node running the job, and is it considerably faster to save a downloaded model here; the drawback is that this is not persistent storage, and the model files will have to be downloaded for each session. However, it might also be a bit faster when using the model.

To use Ollama in the Jupyter app, you need first to open a new notebook in the Jupyter app, and choose the Ollama Jupyter kernel built when setting up the virtual environment. Here is a small example which first imports the necessary packages:

import os
import ollama
 
from openai import OpenAI

then downloads a model from Ollama:

ollama.pull("gemma3:12b")

and also lists all currently downloaded models:

for model in ollama.list().models:
    print(model.model)

It then creates a OpenAI API client:

client = OpenAI(
    base_url=f"http://{os.environ['OLLAMA_HOST']}/v1",
    api_key="ollama"
)

and interacts with the LLM:

response = client.chat.completions.create(
    model="gemma3:12b",
    messages = [
        {
            "role": "system",
            "content": "You are a friendly dog"
        },
        {
            "role": "user",
            "content": "Would you like a bone?"
        }
    ]
)
print(response.choices[0].message.content)

The model can, if desired, be deleted:

ollama.delete("gemma3:12b")

You can find more info on how to use the Ollama Python library on their GitHub page.