| Both sides previous revision Previous revision Next revision | Previous revision |
| rdms:bestpractices [2026/05/21 12:48] – burcu | rdms:bestpractices [2026/08/31 07:32] (current) – burcu |
|---|
| * ''%%--retries%% <int>'': This function can be used in combination with the restart files. It specifies the number of automated retries of the transfer. | * ''%%--retries%% <int>'': This function can be used in combination with the restart files. It specifies the number of automated retries of the transfer. |
| |
| **Note:** For the transfer of a large amount of data, especially single big files, we recommend **not to use** the ''-K'' flag. The flag leads to calculating, storing, and comparing the checksums of the file(s) during transfer. This can sometimes take a very long time and also result in timeouts. We recommend that you instead either wait for the automated checksum calculation to finish or force checksum calculation after the transfer by using the ''ichksum'' command. | **Note:** For the transfer of a large amount of data, especially single big files, we recommend **not using ** the ''-K'' flag. The flag leads to calculating, storing, and comparing the checksums of the file(s) during transfer. This can sometimes take a very long time and also result in timeouts. We recommend that you instead either wait for the automated checksum calculation to finish or force checksum calculation after the transfer by using the ''ichksum'' command. |
| |
| ===== Bundling of Data Sets ===== | ===== Bundling of Data Sets ===== |
| - Upload the bundled format to the RDMS. | - Upload the bundled format to the RDMS. |
| - (Optional) Add metadata if desired. | - (Optional) Add metadata if desired. |
| - (Optional) Unbundle the data on the RDMS. | |
| |
| For extraction on the RDMS, CLI users can use the ''ibun -x'' command as also described in the [[https://wiki.hpc.rug.nl/rdms/access/linux/createprofile#icommands_for_metda_data_management|iCommands for (Meta)data Management]] section of this wiki. | **Note:** The ''ibun '' command does not support symlinks. It is therefore recommended to dereference symlinks upon local creation of the archives. For the ''tar'' command, this can be achieved via the additional ''-h'' flag. |
| | |
| For RDMS web interface users, the "Uncompress tar" function, accessible via right-click on a ''*.tar'' file, enables extraction. Currently, this function supports only ''*.tar'' formats. | |
| | |
| **Note:** The ''ibun'' command does not support symlinks. It is therefore recommended to dereference symlinks upon local creation of the archives. For the ''tar'' command, this can be achieved via the additional ''-h'' flag. | |
| |
| ==== Choosing a Data Compression Format ==== | ==== Choosing a Data Compression Format ==== |
| * ''.7z'' | * ''.7z'' |
| |
| From our experiences, ''*.tar.zst'' which uses [[https://en.wikipedia.org/wiki/Zstd|Zstandard compression]] delivers a very good compromise between achieved compression and compression time. | From our experience, ''*.tar.zst'', which uses [[https://en.wikipedia.org/wiki/Zstd|Zstandard compression]], delivers a very good compromise between achieved compression and compression time. |
| |
| **Notes:** | **Notes:** |
| * Not all compression types can be extracted via ''ibun'' on the RDMS side if needed. From the above-listed formats, ''*.7z'' does not work. In these cases, the file needs to be downloaded first before being able to be extracted. | * In certain cases, it makes sense not to bundle the whole data set into one package, but rather in suitable sub-packages. For example, if those constitute defined subsets of the data, it makes sense to bundle them. |
| * In general, for archived data sets, it is also recommended not to extract them on the RDMS, but rather keep them in their bundled (and compressed) format for long-term storage. | |
| * In certain cases, it makes sense to not bundle the whole data set into one package, but rather in suitable sub-packages. For example, if those constitute defined subsets of the data, it makes sense to bundle. | |
| * Also note that for bundled and compressed formats, it is not easy to directly see the content of the archives (exception: content of ''*.tar'' which can be previewed in the [[rdms:webapp:databrowser|data browser of the web interface]]). For cases where the bundled, and potentially compressed, data set is still of a big size, it is recommended to **create a list of files/folders in the archive locally before bundling and then upload this with the bundled data set.** In these cases, the text file, which is much smaller than the data set, can be downloaded first, and it can be used to check if the respective data set contains the searched-for data. How these lists of files/folders are created depends on your system. Linux users can, for example, use the ''find'' or ''tree'' commands for that, while Windows users can achieve similar results via the ''dir'' command (Windows command prompt) or ''Get-ChildItem'' (Windows PowerShell). | * Also note that for bundled and compressed formats, it is not easy to directly see the content of the archives (exception: content of ''*.tar'' which can be previewed in the [[rdms:webapp:databrowser|data browser of the web interface]]). For cases where the bundled, and potentially compressed, data set is still of a big size, it is recommended to **create a list of files/folders in the archive locally before bundling and then upload this with the bundled data set.** In these cases, the text file, which is much smaller than the data set, can be downloaded first, and it can be used to check if the respective data set contains the searched-for data. How these lists of files/folders are created depends on your system. Linux users can, for example, use the ''find'' or ''tree'' commands for that, while Windows users can achieve similar results via the ''dir'' command (Windows command prompt) or ''Get-ChildItem'' (Windows PowerShell). |
| |
| </code> | </code> |
| |
| This command will check the specified location for files which have a replica status of 2 ("read-locked") or 3 ("write-locked"), and then output it in the format: | This command will check the specified location for files that have a replica status of 2 ("read-locked") or 3 ("write-locked"), and then output them in the format: |
| |
| <code> | <code> |