Differences

This shows you the differences between two versions of the page.

--- rdms:bestpractices [2024/09/09 07:40] – ↷ Page name changed from rdms:best to rdms:bestpractices jelte
+++ rdms:bestpractices [2025/12/09 10:41] (current) – [Transferring Huge Data Sets] slight rephrasing and Grammarly check giulio
@@ Line 1: / Line 1: @@
-{{indexmenu_n>8}}
+{{indexmenu_n>9}}
 ====== Best Practices ======
 This section presents a selection of best practices for using the RDMS. Adhering to these best practices will ensure the most optimal user experience with the RDMS.
@@ Line 57: / Line 57: @@
     publication.final version.odt
 </code>
+===== Transferring Large Data Sets =====
+For the transfer of very big data sets, especially those containing files in the realm of several 100GBs and more, we recommend using the [[.:access:linux:icommands|iCommands]] CLI tool. We specifically recommend to use the ''iput'' (upload) and ''iget'' (download) commands with the following parameters:
+<code>
+# Upload: Single Big file
+$ iput -T --lfrestart /path/to/lfRestartFile --retries 3 /path/to/big/local/file /rug/home/destination_collection/
+# Upload: Big folder
+$ iput -r -T -X /path/to/restartfile --lfrestart /path/to/lfRestartFile --retries 3 /path/to/big/local/folder /rug/home/destination_collection/
+# Download: Single big file
+$ iget -T --lfrestart /path/to/lfRestartFile --retries 3 /path/to/big/local/file /rug/home/destination_collection/
+# Download: Big folder
+$ iget -r -T -X /path/to/restartfile --lfrestart /path/to/lfRestartFile --retries 3 /path/to/big/local/folder /rug/home/destination_collection/
+</code>
+The additional parameters used for the `iput` command have the following function:
+  * ''-T'': Renew the socket connection after 10 minutes. This can be useful for big data transfers to prevent events like the firewall canceling the connection.
+  * ''-X /path/to/restartfile'': When this parameter is used, the command writes restart information to the specified restart file. This file contains information on how many of the files were already uploaded and what the last uploaded file was. It is especially useful for transferring folders with multiple files.
+  * ''%%--lfrestart%% /path/to/lfRestartFile'': When this parameter is used, the command writes different restart information for large files to ''/path/to/lfRestartFile''. If the transfer fails, this allows you to continue the transfer from the point where it failed for the large file.
+  * ''%%--retries%% <int>'': This function can be used in combination with the restart files. It specifies the number of automated retries of the transfer.
+**Note:** For the transfer of a large amount of data, especially single big files, we recommend **not to use** the ''-K'' flag. The flag leads to calculating, storing, and comparing the checksums of the file(s) during transfer. This can sometimes take a very long time and also result in timeouts. We recommend that you instead either wait for the automated checksum calculation to finish or force checksum calculation after the transfer by using the ''ichksum'' command.
 ===== Bundling of Data Sets =====
-To improve the performance of the RDMS, it is recommended to store data sets in numerous small files in a structured format like ''*.tar'', ''*.tar.gz'', ''*.tar.bz'', or ''*.zip''. This significantly improves transfer rates as the system engages in multi-threaded transfers after reaching a minimal file size threshold (32 MB). Transferring multiple smaller files furthermore results in big overhead, diminishing performance.
+To improve the performance of the RDMS, it is recommended to store data sets in a structured format like ''*.tar'', ''*.tar.gz'', ''*.zip'', or similar (see below for more info about data compression) instead of individual files/folder. This significantly improves transfer rates as the system engages in multi-threaded transfers after reaching a minimal file size threshold (32 MB). Transferring multiple smaller files furthermore results in big overhead, diminishing performance.
 Best practices to handle such cases are:
@@ Line 74: / Line 99: @@
 For RDMS web interface users, the "Uncompress tar" function, accessible via right-click on a ''*.tar'' file, enables extraction. Currently, this function supports only ''*.tar'' formats.
+**Note:** The ''ibun'' command does not support symlinks. It is therefore recommended to dereference symlinks upon local creation of the archives. For the ''tar'' command, this can be achieved via the additional ''-h'' flag.
+==== Choosing a Data Compression Formats ====
+While the bundling of data without extra compression (''*.tar'') is already very helpful to increase the performance of data transfers, additional compression is often useful as this can reduce the data size tremendously. There are different possibilities of compression, for example:
+  * ''*.tar.gz''
+  * ''*.tar.bz2''
+  * ''*.tar.xz''
+  * ''*.tar.zst''
+  * ''*.zip''
+  * ''.7z''
+From our experiences, ''*.tar.zst'' which uses [[https://en.wikipedia.org/wiki/Zstd|Zstandard compression]] delivers a very good compromise between achieved compression and compression time.
+**Notes:**
+  * Not all compression types can be extracted via ''ibun'' on the RDMS side if needed. From the above listed formats, ''*.7z'' does not work. In this cases, the file needs to be downloaded first before being able to extract.
+  * In general, for archived data sets, it is also recommended to not extract them on the RDMS, but rather keep them in their bundled (and compressed) format for long-term storage.
+  * In certain cases, it makes sense to not bundle the whole data set into one package, but rather in suitable sub-packages. For example if those constitute of defined subsets of the data where it makes sense to bundle.
+  * Also note that for bundled and compressed formats, it is not easy to directly see the content of the archives (exception: content of ''*.tar'' which can be previewed in the [[rdms:webapp:databrowser|data browser of the web interface]]). For cases where the bundled, and potentially compressed, data set is still of a big size, it is recommended to **create a list of files/folders in the archive locally before bundling and then upload this with the bundled data set.** In this cases, the text file, which is much smaller than data set, can be downloaded first and it can be used to check if the respective data set contains the searched for data. How these lists of files/folders are created depends on your system. Linux users can, for example, use the ''find'' or ''tree'' commands for that while Windows users can achieve similar results via the ''dir'' command (Windows command prompt) or  ''Get-ChildItem'' (Windows Powershell).
+Please contact [[rdms-support@rug.nl|rdms-support@rug.nl]] if you are not sure how to bundle/compress your data sets for long-term storage.
 ===== Locked Files (HIERARCHY_ERROR) =====
@@ Line 89: / Line 135: @@
 <code>
- $ iquest "status: %s, name: %s/%s" "SELECT DATA_REPL_STATUS, COLL_NAME, DATA_NAME WHERE COLL_NAME LIKE '/rug/home/path/to/folder%' AND DATA_REPL_STATUS > '1'"
+ $ iquest "status: %s, name: %s/%s" "SELECT DATA_REPL_STATUS, COLL_NAME, DATA_NAME WHERE COLL_NAME LIKE '/rug/home/path/to/folder%' AND DATA_REPL_STATUS <> '1'"
 </code>