| Both sides previous revision Previous revision | |
| rdms:bestpractices [2025/12/08 11:27] – [Transferring Huge Data Sets] added some more text. jelte | rdms:bestpractices [2025/12/09 10:41] (current) – [Transferring Huge Data Sets] slight rephrasing and Grammarly check giulio |
|---|
| </code> | </code> |
| |
| ===== Transferring Huge Data Sets ===== | ===== Transferring Large Data Sets ===== |
| For the transfer of very big data sets, especially those containing files in the realm of several 100GBs and more, we recommend to use the [[.:access:linux:icommands|iCommands]] CLI tool. For these kind of transfers, we recommend to use the ''iput'' (upload) and ''iget'' (download) commands with the following parameters: | For the transfer of very big data sets, especially those containing files in the realm of several 100GBs and more, we recommend using the [[.:access:linux:icommands|iCommands]] CLI tool. We specifically recommend to use the ''iput'' (upload) and ''iget'' (download) commands with the following parameters: |
| |
| <code> | <code> |
| |
| # Upload: Big folder | # Upload: Big folder |
| $ iput -r -T -X /path/to/restartfile --lfrestart /path/to/lfrestartfile --retries 3 /path/to/big/local/folder /rug/home/destination_collection/ | $ iput -r -T -X /path/to/restartfile --lfrestart /path/to/lfRestartFile --retries 3 /path/to/big/local/folder /rug/home/destination_collection/ |
| |
| # Download: Single big file | # Download: Single big file |
| |
| # Download: Big folder | # Download: Big folder |
| $ iget -r -T -X /path/to/restartfile --lfrestart /path/to/lfrestartfile --retries 3 /path/to/big/local/folder /rug/home/destination_collection/ | $ iget -r -T -X /path/to/restartfile --lfrestart /path/to/lfRestartFile --retries 3 /path/to/big/local/folder /rug/home/destination_collection/ |
| </code> | </code> |
| |
| The additional used parameters for the `iput` command have the following function: | The additional parameters used for the `iput` command have the following function: |
| * ''-T'': Renew the socket connection after 10 minutes. This can be useful for big data transfers to prevent events like the firewall canceling the connection. | * ''-T'': Renew the socket connection after 10 minutes. This can be useful for big data transfers to prevent events like the firewall canceling the connection. |
| * ''-X /path/to/restartfile'': When this parameter is used, the command writes restart information to the specified restart file. This file contains info how many of the files where already uploaded and what was the last uploaded file. It is especially useful for transferring folders with multiple files. | * ''-X /path/to/restartfile'': When this parameter is used, the command writes restart information to the specified restart file. This file contains information on how many of the files were already uploaded and what the last uploaded file was. It is especially useful for transferring folders with multiple files. |
| * ''%%--lfrestart%% /path/to/lfRestartFile'': When this parameter is used, the command writes different restart information for large files to ''/path/to/lfRestartFile''. If the transfer fails, this allows to continue the transfer with from the point where it failed for the large file. | * ''%%--lfrestart%% /path/to/lfRestartFile'': When this parameter is used, the command writes different restart information for large files to ''/path/to/lfRestartFile''. If the transfer fails, this allows you to continue the transfer from the point where it failed for the large file. |
| * ''%%--retries%% <int>'': This function can be used in combination with the restart files. It specifies the number of automated retries of the transfer. | * ''%%--retries%% <int>'': This function can be used in combination with the restart files. It specifies the number of automated retries of the transfer. |
| |
| **Note:** For the transfer of huge amount of data, especially single big files, we recommend **not to use** the ''-K'' flag to automated calculated, store, and compare the checksums during transfer. This can take sometimes a very long time and also result in timeouts. Instead, either wait for the automated checksum calculation to finish or force checksum calculation after transfer by using the ''ichksum'' command. | **Note:** For the transfer of a large amount of data, especially single big files, we recommend **not to use** the ''-K'' flag. The flag leads to calculating, storing, and comparing the checksums of the file(s) during transfer. This can sometimes take a very long time and also result in timeouts. We recommend that you instead either wait for the automated checksum calculation to finish or force checksum calculation after the transfer by using the ''ichksum'' command. |
| |
| ===== Bundling of Data Sets ===== | ===== Bundling of Data Sets ===== |