Cluster description

The Hábrók cluster has been delivered by Dell and consists of the following parts:

• 119 standard nodes with the following components:
  • 128 cores @ 2.45 GHz (two AMD 7763 CPUs)
  • 512 GB memory
  • 3.84 TB internal SSD disk space

• 24 nodes for multi-node jobs with the following components:
  • 128 cores @ 2.45 GHz (two AMD 7763 CPUs)
  • 512 GB memory
  • 3.84 TB internal SSD disk space
  • 100 Gbps Omni-Path link

• 4 big memory nodes with the following components:
  • 80 cores @ 2.3 GHz (two Intel Xeon Platinum 8380 CPUs)
  • 4096 GB memory
  • 15.36 TB internal SSD disk space

• 6 GPU nodes with the following components:
  • 64 cores @ 2.6 GHz
  • 512 GB memory
  • 4 Nvidia A100 GPU accelerator cards
  • 12.8 TB internal SSD NVMe disk space
  • 100 Gbps Omni-Path link

• 36 GPU nodes with the following components:
  • 6 cores @ 2.7 GHz (12 cores with hyperthreading)
  • 128 GB memory
  • 1 Nvidia V100 GPU accelerator card

• 15 nodes with the following components:
  • 64 cores @ 2.2 GHz (two AMD EPYC 7601 CPUs)
  • 512 GB memory
  • 16 TB internal disk space
  • Only accessible by GELIFES users, see GELIFES Partition

• A 100 Gbps non-blocking Omni-Path network for 24 compute and 6 GPU nodes
  • Useful for parallel processing over multiple computers
  • Connects the system to the bulk storage
• Two 25 Gbps Ethernet networks
  • Used for accessing the storage areas and for job communication
  • Can also be useful to access remote data more quickly

• The cluster has 617 TB of formatted storage available. This storage is set up using the Lustre parallel file system and split into three file systems: /home, /data and /scratch. See storage for more information.

The layout of a modern Xeon E5 2600v3 system is shown in the figure (copied from Intel) below:

Our standard systems have two sockets with a Xeon E5 2680v3 processor. Each processor has 12 CPU cores running at 2.5GHz. When not all cores of a processor are used the cores can be run at a higher clockspeed (at most 3.3 GHz).

In principle each CPU core of a Xeon processor can run multiple threads (programs) simultaneously. This is called hyperthreading. This feature has been disabled for the Peregrine cluster as the performance benefits are minimal and it introduces additional complexity to the scheduling system.

Each processor has its own memory controller and is connected to its own set of memory. For our systems this means that each processor has direct access to 64 GB of the 128 GB in the system.

When a processor wants to access the memory of another processor it has to use the QPI connections between the processors. This connection is much slower than the connection to the local memory. This means that it is important that processes running on one of the processors use the memory local to that processor!

NOTE You can still request all the memory on a machine, even with a single core.

Each processor also has its own links to a PCIe 3.0 bus which connects to the network cards and accelerator cards (Infiniband, 10Gbps Ethernet, GPU, Xeon Phi). This means that one processor is connected to the network card(s) and the other can only reach the network card through the first one.