Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision		 Previous revision
habrok:introduction:workflow [2022/12/20 13:23] – [scratch] fokkehabrok:introduction:workflow [2023/03/22 13:03] (current) – fokke
Line 26: Line 26:
 ===== Data ===== ===== Data =====
  
-For most applications users need to work with data. Data can be parameters for a program that needs to be run, for example to set up a simulation. It can be input data that needs to be analyzed. And finally running simulations or data analysis will result in data containing the results of the computations.+For most applications users need to work with data. Data can be parameters for a program that needs to be run, for example to set up a simulation. It can be input data that needs to be analyzed. And, finally, running simulations or data analyses will result in data containing the results of the computations.
  
 Hábrók has its own storage system, which is decoupled from the desktop storage systems the university has. Although it would be nice to be able to access data from your desktop system directly on Hábrók, currently this is not possible. Technically this would be challenging, and there would also be performance issues, when people start to do more heavy processing on the desktop storage systems. Hábrók has its own storage system, which is decoupled from the desktop storage systems the university has. Although it would be nice to be able to access data from your desktop system directly on Hábrók, currently this is not possible. Technically this would be challenging, and there would also be performance issues, when people start to do more heavy processing on the desktop storage systems.
Line 45: Line 45:
 ==== scratch ==== ==== scratch ====
  
-For larger data sets each user has access to a space on the scratch file system. **This area is only meant for data ready for or recent results from processing. It is not meant for long term storage. THERE IS NO BACKUP!! **+For larger data sets each user has access to a space on the scratch file system. \\ 
 +**This area is only meant for data ready for processing, or recent results from processing. It is not meant for long term storage. THERE IS NO BACKUP!! **
  
 ==== local disks ==== ==== local disks ====
  
-The Peregrine nodes also have local disks that can only be used by calculations running on that specific machine. This implies that this also is temporary space. +The Hábrók nodes also have local disks that can only be used by calculations running on that specific machine. This implies that this also is temporary space. These local disks are based on solid state (SSD) storage. This means that they are for most use cases much faster than the scratch area. 
- +
-These local disks are based on solid state (SSD) storage. This means that they are for most use cases much faster than the scratch area. +
  
 We therefore advise users to copy their input data sets from the scratch area to the local disk at the beginning of the job. The job can then run using the local disk for reading and writing data. At the end of the job the output has to be written back to the scratch area.   We therefore advise users to copy their input data sets from the scratch area to the local disk at the beginning of the job. The job can then run using the local disk for reading and writing data. At the end of the job the output has to be written back to the scratch area.  
-This step  especially important if your input data is read multiple times or consists of many (>1000) files. Similar guidelines are applicable to output data.+This step is especially important if your input data is read multiple times or consists of many (>1000) files. Similar guidelines are applicable to output data.
  
  
Line 61: Line 60:
 Besides the storage directly available on all nodes of the cluster some external storage areas can be accessed from the login nodes. These areas are described below. Besides the storage directly available on all nodes of the cluster some external storage areas can be accessed from the login nodes. These areas are described below.
  
-==== data handling ==== +==== Project storage ====
- +
-On the login node the "data handling" file systems are mounted. Users can get access to storage on data handling based on a fair use model. The storage is allocated on request. +
  
-**WiP**+On the login nodes the ''/projects'' file system is mounted. Users can get access to storage on this storage area based on a fair use model. Additional storage is allocated on request and above a certain threshold costs are involved.
  
 ==== RDMS ==== ==== RDMS ====
  
-**WiP**+The Research Data Management system can be used from the Hábrók nodes. You can find more information about the RDMS on its dedicated wiki pages: 
 +https://wiki.hpc.rug.nl/rdms/start
  
  
 ---- ----
-**Next section: [[peregrine:connecting_to_the_system:connecting]]**+**Next section: [[habrok:connecting_to_the_system:connecting]]**