Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
rdms:bestpractices [2026/05/21 12:44] – burcurdms:bestpractices [2026/05/21 12:48] (current) – burcu
Line 1: Line 1:
 {{indexmenu_n>9}} {{indexmenu_n>9}}
 ====== Best Practices ====== ====== Best Practices ======
-This section presents a selection of best practices for using the RDMS. Adhering to these best practices will ensure the most optimal user experience with the RDMS. +This section presents a selection of best practices for using the RDMS. Adhering to these best practices will ensure the optimal user experience with the RDMS. 
  
 The section will be gradually updated with new usage examples and tips. The section will be gradually updated with new usage examples and tips.
Line 85: Line 85:
 ===== Bundling of Data Sets ===== ===== Bundling of Data Sets =====
  
-To improve the performance of the RDMS, it is recommended to store data sets in a structured format like ''*.tar'', ''*.tar.gz'', ''*.zip'', or similar (see below for more info about data compression) instead of individual files/folder. This significantly improves transfer rates as the system engages in multi-threaded transfers after reaching a minimal file size threshold (32 MB). Transferring multiple smaller files furthermore results in big overhead, diminishing performance. +To improve the performance of the RDMS, it is recommended to store data sets in a structured format like ''*.tar'', ''*.tar.gz'', ''*.zip'', or similar (see below for more info about data compression) instead of individual files/folders. This significantly improves transfer rates as the system engages in multi-threaded transfers after reaching a minimal file size threshold (32 MB). Transferring multiple smaller files furthermore results in a big overhead, diminishing performance. 
  
 Best practices to handle such cases are: Best practices to handle such cases are:
Line 103: Line 103:
 ==== Choosing a Data Compression Format ==== ==== Choosing a Data Compression Format ====
  
-While the bundling of data without extra compression (''*.tar'') is already very helpful to increase the performance of data transfers, additional compression is often useful as this can reduce the data size tremendously. There are different possibilities of compression, for example:+While the bundling of data without extra compression (''*.tar'') is already very helpful to increase the performance of data transfers, additional compression is often useful, as this can reduce the data size tremendously. There are different possibilities of compression, for example:
   * ''*.tar.gz''   * ''*.tar.gz''
   * ''*.tar.bz2''   * ''*.tar.bz2''
Line 114: Line 114:
  
 **Notes:**  **Notes:** 
-  * Not all compression types can be extracted via ''ibun'' on the RDMS side if needed. From the above listed formats, ''*.7z'' does not work. In this cases, the file needs to be downloaded first before being able to extract. +  * Not all compression types can be extracted via ''ibun'' on the RDMS side if needed. From the above-listed formats, ''*.7z'' does not work. In these cases, the file needs to be downloaded first before being able to be extracted. 
-  * In general, for archived data sets, it is also recommended to not extract them on the RDMS, but rather keep them in their bundled (and compressed) format for long-term storage. +  * In general, for archived data sets, it is also recommended not to extract them on the RDMS, but rather keep them in their bundled (and compressed) format for long-term storage. 
-  * In certain cases, it makes sense to not bundle the whole data set into one package, but rather in suitable sub-packages. For example if those constitute of defined subsets of the data where it makes sense to bundle. +  * In certain cases, it makes sense to not bundle the whole data set into one package, but rather in suitable sub-packages. For example, if those constitute defined subsets of the data, it makes sense to bundle. 
-  * Also note that for bundled and compressed formats, it is not easy to directly see the content of the archives (exception: content of ''*.tar'' which can be previewed in the [[rdms:webapp:databrowser|data browser of the web interface]]). For cases where the bundled, and potentially compressed, data set is still of a big size, it is recommended to **create a list of files/folders in the archive locally before bundling and then upload this with the bundled data set.** In this cases, the text file, which is much smaller than data set, can be downloaded first and it can be used to check if the respective data set contains the searched for data. How these lists of files/folders are created depends on your system. Linux users can, for example, use the ''find'' or ''tree'' commands for that while Windows users can achieve similar results via the ''dir'' command (Windows command prompt) or  ''Get-ChildItem'' (Windows Powershell). +  * Also note that for bundled and compressed formats, it is not easy to directly see the content of the archives (exception: content of ''*.tar'' which can be previewed in the [[rdms:webapp:databrowser|data browser of the web interface]]). For cases where the bundled, and potentially compressed, data set is still of a big size, it is recommended to **create a list of files/folders in the archive locally before bundling and then upload this with the bundled data set.** In these cases, the text file, which is much smaller than the data set, can be downloaded first, and it can be used to check if the respective data set contains the searched-for data. How these lists of files/folders are created depends on your system. Linux users can, for example, use the ''find'' or ''tree'' commands for that, while Windows users can achieve similar results via the ''dir'' command (Windows command prompt) or  ''Get-ChildItem'' (Windows PowerShell). 
  
 Please contact [[rdms-support@rug.nl|rdms-support@rug.nl]] if you are not sure how to bundle/compress your data sets for long-term storage.  Please contact [[rdms-support@rug.nl|rdms-support@rug.nl]] if you are not sure how to bundle/compress your data sets for long-term storage. 
 ===== Locked Files (HIERARCHY_ERROR) ===== ===== Locked Files (HIERARCHY_ERROR) =====
  
-In rare cases, data may arrive in an incomplete form in the RDMS. This usually happens if a data transfer abruptly interrupted, for example due to connection problems, without proper finalization. +In rare cases, data may arrive in an incomplete form in the RDMS. This usually happens if a data transfer is abruptly interrupted, for example, due to connection problems, without proper finalization. 
  
-Restarting the data transfer may solve this issue. However, it is possible that the already transferred data remains in a locked state, causing problems when the transfer is restarted as those files cannot be overwritten directly. +Restarting the data transfer may solve this issue. However, it is possible that the already transferred data remains in a locked state, causing problems when the transfer is restarted, as those files cannot be overwritten directly. 
  
 If you experience these issues, it is recommended to contact [[rdms-support@rug.nl|RDMS-Support]]. If you experience these issues, it is recommended to contact [[rdms-support@rug.nl|RDMS-Support]].
  
-Users of the command-line tool [[rdms:access:linux:icommands|iCommands]] have furthermore the possibility to detect such locked files directly using an appropriate CLI command. +Users of the command-line tool [[rdms:access:linux:icommands|iCommands]] have, furthermore, the possibility to detect such locked files directly using an appropriate CLI command. 
  
 In general, these issues manifest in ''HIERARCHY_ERRORs'' when a data transfer to the RDMS (e.g. via ''iput'' or ''irsync'') is attempted via CLI.  In general, these issues manifest in ''HIERARCHY_ERRORs'' when a data transfer to the RDMS (e.g. via ''iput'' or ''irsync'') is attempted via CLI. 
  
-To check all files at a RDMS location ''/rug/home/path/to/folder'' including all its subfolders, and to detect just those files that are marked as locked, the following command can be executed:+To check all files at an RDMS location ''/rug/home/path/to/folder'' including all its subfolders, and to detect just those files that are marked as locked, the following command can be executed:
  
 <code> <code>
Line 146: Line 146:
 ==== Removal of Locked Files ==== ==== Removal of Locked Files ====
  
-While the locked files cannot be directly removed, they can still be moved first to another location in your home/team location, for example as a separate folder for locked files. Afterwards, the data transfer can be restarted. +While the locked files cannot be directly removed, they can still be moved first to another location in your home/team location, for example, as a separate folder for locked files. Afterwards, the data transfer can be restarted. 
  
 Best practices to handle locked files and resolve the ''HIERARCH_ERROR'' are: Best practices to handle locked files and resolve the ''HIERARCH_ERROR'' are:
Line 153: Line 153:
   - Use the 'iquest' command to identify locked files and move them to the newly created location. CLI users can utilize 'imv' for this purpose.   - Use the 'iquest' command to identify locked files and move them to the newly created location. CLI users can utilize 'imv' for this purpose.
   - Restart the data transfer. The ''HIERARCHY_ERROR'' should be resolved.   - Restart the data transfer. The ''HIERARCHY_ERROR'' should be resolved.
-  - If you accumulated multiple locked files in your folder which you cannot delete, please contact [[rdms-support@rug.nl|RDMS-Support]] and we will help you remove these. +  - If you accumulated multiple locked files in your folder that you cannot delete, please contact [[rdms-support@rug.nl|RDMS-Support]], and we will help you remove these. 
  
 **Note:** It is recommended not to contact RDMS support for every locked file, but instead first try to resolve it as described above. However, if numerous locked files are detected, you can directly contact RDMS support. **Note:** It is recommended not to contact RDMS support for every locked file, but instead first try to resolve it as described above. However, if numerous locked files are detected, you can directly contact RDMS support.