Hábrók Changelog

This page records major changes to Hábrók, which are mostly carried out during scheduled maintenance periods. This changelog is kept since August 13th 2025, although changes have been made during past scheduled maintenance periods and on-demand as necessary.

Starting 2nd phase of migration from Alma Linux 8 to Alma Linux 9. See gradual_migration_of_compute_and_logininteractive_nodes for the details. This introduces the following changes:

• The special reservation for Alma 9 nodes has been removed and they are now part of the general pool
• We will migrate more compute and GPU nodes to Alma Linux 9
• We will start migrating half of the login, interactive and interactive GPU nodes to Alma Linux 9. We have started with reinstalling interactive2.hb.hpc.rug.nl

The following changes were applied during the scheduled maintenance window:

• Applied firmware updates to all nodes.
• Upgraded Slurm (Hábrók's job scheduler) to version 25.11.7.
• Upgraded Lustre (/scratch and /projects filesystem servers) to version 2.15.8.
• Moved V100 nodes to other network switch.
• Rebooted all nodes.
• Completed validation tests.

• A new GPU node containing a single RTX PRO 6000 GPU and a AMD EPYC 9555P Zen5 GPU is now available. Users belonging to the Orion project have priority access. General users are limited to the gpushort partition in this node, see more details about Group specific partitions.

• The GPU drivers have been updated to fix the issues described at https://nvidia.custhelp.com/app/answers/detail/a_id/5821 This means that all GPUs except the V100 GPUs are at driver version 595.71.05. The V100 nodes have driver 580.159.04, as the later releases like versions 595 are no longer supported on those cards.
• The driver updates mean that CUDA 13 is now supported on all GPUs except the V100 GPUs. Although the 580.159.04 driver states support for CUDA 13.0, it is not supported on the V100 GPUs. This means that on the V100 you will need to limit yourself to CUDA 12 based software.

• On the night of 2026-04-29 a security vulnerability – CVE-2026-31431 – was published that affected most Linux systems in operation. This vulnerability allowed obtaining root privileges on systems by running a script that exploited a flaw in Linux's kernel. To prevent this vulnerability from being exploited, we brought down all Hábrók access nodes on the night of 2026-04-29, prevented new jobs from starting in any queue and disabled the web portal. On 2026-04-30, we applied mitigating measures that prevent this vulnerability from being exploited. All idle nodes were immediately patched and rebooted and a mitigating measure that blocks the vulnerability was applied to nodes that still had jobs running, to prevent malicious code from being executed until they are rebooted with a patch. Due to this, these nodes won't accept new jobs and will reboot as soon as they are empty.
• As of 16:30 the access nodes are available again, the queues are resumed, and users can access the system as usual. Capacity will be reduced until all nodes have finished rebooting.
• You can follow the timeline of this incident on our status page issue.

• interactive1.hb.hpc.rug.nl needed to be reinstalled after a failure. This means its ssh host key has changed. See Host keys for more information about host keys.

• New GPU nodes have been made available. See Nvidia RTX Pro 6000 GPU Nodes for details.
• The default number of threads for OpenBLAS and MKL has been set to 1. This to prevent their automatic parallelization to interfere with the parallelization of the software using these libraries. See Controlling the number of threads for OpenBLAS and Intel MKL for details.

• The Web Portal has been updated from Open OnDemand v4.0.1 to Open OnDemand v4.1.0.

• Similarly to what was done on 2026-01-06, the memory of the compute nodes was again reduced slightly because the issue was not resolved yet. This means that you may need to adjust your memory request if you want to use all available memory on a node.

• The memory on the compute nodes available to jobs has been reduced to prevent nodes from running out of memory. This caused system services to crash, causing issues on affected nodes. This means that you may need to adjust your memory request if you want to use all available memory on a node.
• The nodes bought by GELIFES are out of support and as such the gelifes partition has been decommissioned.

• Long and intensive processes running on the login, interactive, and interactive GPU nodes are now automatically killed to ensure everyone can access these resources fairly. See details here.

• The maintenance for 15 September had to be extended because of a vulnerability in the Linux kernel, which needed to be patched urgently. In order to patch the issue more quickly we changed from Rocky Linux to Alma Linux on the compute and user interface nodes. Since both Linux distributions are derived from the same release of Redhat Enterprise Linux this does not affect any other installed software.
• The Lmod package for the module system needed to be downgraded as the latest release introduced issues, causing the module system to fail.
• We've equipped 24 regular nodes with an Cornelis Networks Omnipath adapter and migrated those to the parallel partition as "omni" nodes. We've extended the regularshort and regularmedium partition to share nodes with the parallel partitions. This extends the available capacity for jobs running over more than one node, that require a fast, low-latency interconnect.

• A new node with 2 NVIDIA L40S GPUs has been added to the gpushort partition. The node was procured by the Computer-Aided Organic Synthesis group and their dedicated "caos" partitions have a higher priority than gpushort.
• Updated jobinfo to also report GPU usage on digitallab and caos partitions. This also fixes the hints for these nodes.

The following lists the main changes applied in the Hábrók in the maintenance period from 11th August 2025 to 13th August 2025. These changes aim to generally improve the stability of the system, include previously released software patches and optimize performance of some aspects of the cluster.

• Update Slurm from version 24.05.8-1 to 24.11.6-2
• Update PMIx from version 3.2.5-1 to 4.2.9-1
• Update Lustre servers and clients from version 2.15.6 to 2.15.7
  • Relevant for /scratch
• Run filesystem check on Lustre servers
  • Relevant for /scratch
• Update BIOS from version 2.17.4 to 2.19.0
• Update NVIDIA drivers from version 560.35.05 to 575.57.08 and supported CUDA versions from version 12.6.3 to 12.9.1
• Update Rocky kernel to version 4.18.0-553.64.1
• Update Omni-Path user libraries
  • Relevant for the parallel partition and A100 GPU nodes
• Improve NFS server settings and configuration (higher thread count and mount filesystem as async)
  • Relevant for /home and /userapps performance
• Use upstream EESSI module for initializing the EESSI software stack
• Update several installed packages

A record of the real-time updates given during the maintenance period can be found in the associated status page entry.