Group specific partitions

Some groups and institutes have bought their own extensions of Peregrine and Hábrók. These nodes have been put into special partitions. They share the same storage and software environment as all the other nodes.

Since these nodes have been paid for by the group or institute, their users are often the only ones who can access them. To separate the usage between the group specific nodes and the rest of the cluster a separate account has been made in the scheduler for each of these groups, the other regular account being ‘users’.

Only users which have an account in this group managed account are able to submit to their partition. These people will also have an account in ‘users’, like any other Hábrók user. Jobs submitted in the special partition will be accounted for in these special accounts, and jobs in the other partitions will be accounted for in the regular “users” account.

This sounds a bit complicated, but the main thing to take home is:

- Submit to the dedicated partition to get to the group specific nodes.
- Usage (and priorities) of these nodes are handled separately from the usage and priorities of the rest of the cluster.

Users with the coordinator role in one of the special accounts can add users to the account using the following command:

sacctmgr add user USERNAME account=ACCOUNT fairshare=1

Where USERNAME should be changed into the userid that is to be added to the account ACCOUNT. ACCOUNT must be changed into the name of the account, e.g. gelifes, digitallab, caos. The fairshare should be by default set to 1.

In order to verify/check if a user has already been added to the account, the column “Account” in the output of the following command should show a row with “users” and one with the special account:

sacctmgr show -s user USERNAME

Coordinators can be added to the special accounts using:

sacctmgr add coordinator name=USERNAME account=ACCOUNT

In order to modify an existing user the following can be used:

sacctmgr modify user name=USERNAME account=ACCOUNT set fairshare=1

The node themselves are 64 core AMD EPYC 7601 nodes, running at 2.2 GHz, with 512GB of memory. These should be suitable for most of the GELIFES workloads. There is also quite a lot (16 TB) of temporary local scratch space per node, available for jobs through the use $TMPDIR in the jobs scripts.

The gelifes partition has two types of limits: one on the number of jobs per user, and one on the number of cores allocated to different job lengths.

Note that jobs from all users contribute to the maximum number of cores that can be allocated to these jobs. This prevents the partition from being filled with long jobs, which would lead to higher waiting times for jobs. If this limit is reached, waiting jobs will get a Reason: QOSGrpCpuLimit.

Since the instruction set of the AMD CPUs in the gelifes partition is compatible with that of the standard Intel based nodes, the software from these nodes is used on the gelifes partition. There is an issue with the intel compiler based software, however. Whereas the software based on the GNU compilers (foss toolchains) works fine, the software based on the intel toolchain does not work. This because the intel compiler introduces a CPU check in the code, which fails for the AMD nodes.
Our advice is therefore to make use of the foss toolchains, as all relevant software should be available in the foss toolchains.

You can request access to the gelifes account, giving access to the GELIFES nodes by contacting Joke Bakker from GELIFES.

Users with the coordinator role in the gelifes account can add users to the account using the following command:

sacctmgr add user USERNAME account=gelifes fairshare=1

Where USERNAME should be changed into the userid that is to be added to the account gelifes. The fairshare should be by default set to 1.

In order to verify/check if a user has already been added to the gelifes account, the column “Account” in the output of the following command should show a row with “users” and one with “gelifes”:

sacctmgr show -s user USERNAME

Coordinators can be added to the gelifes account using:

sacctmgr add coordinator name=USERNAME account=gelifes

In order to modify an existing user the following can be used:

sacctmgr modify user name=USERNAME account=gelifes set fairshare=1