This section presents a selection of best practices for using the RDMS. Adhering to these best practices will ensure the most optimal user experience with the RDMS.
The section will be gradually updated with new usage examples and tips.
If you believe that important information should be added to this section, please contact RDMS support with your request!
For the optimal usage of the RDMS, it is highly recommended to follow these best practices for naming your files/folders:
$%^*&#=!
) in your file/folder names..
) in your file/folder names. '
or “
) in your file/folder names. _
) or hyphen (-
) instead of white spaces in file/folder names. Example of a folder structure with correct naming:
$ itree project_name project_name analytical_data machine_01 20231223_analysis.ext 20240111_analysis.ext 20240325_analysis.ext machine_02 20230222_analysis.ext 20230710_analysis.ext 20240109_analysis.ext machine_03 20231020_analysis.ext 20231120_analysis.ext 20231212_analysis.ext manuscripts publication_v01.odt
Example of a folder structure with incorrect naming:
$ itree "Project with XXX and YYY" Project with XXX and YYY analytical_data analytical devices @ building 1 experiment 100% scan rate.ext experiment 74% scan rate.ext experiment 80% scan rate.ext analytical devices @ building 2 Experiment 01 by user.name@rug.nl.ext Experiment 02 by user.name@rug.nl.ext Experiment 03 by user.name@rug.nl.ext analytical devices @ building 3 $1-100%.ext Versuch #1.ext Versuch_Öldiffusion_erste_Möglichkeit.ext manuscripts publication.final version.odt
To improve the performance of the RDMS, it is recommended to store data sets in a structured format like *.tar
, *.tar.gz
, *.zip
, or similar (see below for more info about data compression) instead of individual files/folder. This significantly improves transfer rates as the system engages in multi-threaded transfers after reaching a minimal file size threshold (32 MB). Transferring multiple smaller files furthermore results in big overhead, diminishing performance.
Best practices to handle such cases are:
For extraction on the RDMS, CLI users can use the ibun -x
command as also described in the iCommands for (Meta)data Management section of this wiki.
For RDMS web interface users, the “Uncompress tar” function, accessible via right-click on a *.tar
file, enables extraction. Currently, this function supports only *.tar
formats.
Note: The ibun
command does not support symlinks. It is therefore recommended to dereference symlinks upon local creation of the archives. For the tar
command, this can be achieved via the additional -h
flag.
While the bundling of data without extra compression (*.tar
) is already very helpful to increase the performance of data transfers, additional compression is often useful as this can reduce the data size tremendously. There are different possibilities of compression, for example:
*.tar.gz
*.tar.bz2
*.tar.xz
*.tar.zst
*.zip
.7z
From our experiences, *.tar.zst
which uses Zstandard compression delivers a very good compromise between achieved compression and compression time.
Notes:
ibun
on the RDMS side if needed. From the above listed formats, *.7z
does not work. In this cases, the file needs to be downloaded first before being able to extract.*.tar
which can be previewed in the data browser of the web interface). For cases where the bundled, and potentially compressed, data set is still of a big size, it is recommended to create a list of files/folders in the archive locally before bundling and then upload this with the bundled data set. In this cases, the text file, which is much smaller than data set, can be downloaded first and it can be used to check if the respective data set contains the searched for data. How these lists of files/folders are created depends on your system. Linux users can, for example, use the find
or tree
commands for that while Windows users can achieve similar results via the dir
command (Windows command prompt) or Get-ChildItem
(Windows Powershell). Please contact rdms-support@rug.nl if you are not sure how to bundle/compress your data sets for long-term storage.
In rare cases, data may arrive in an incomplete form in the RDMS. This usually happens if a data transfer abruptly interrupted, for example due to connection problems, without proper finalization.
Restarting the data transfer may solve this issue. However, it is possible that the already transferred data remains in a locked state, causing problems when the transfer is restarted as those files cannot be overwritten directly.
If you experience these issues, it is recommended to contact RDMS-Support.
Users of the command-line tool iCommands have furthermore the possibility to detect such locked files directly using an appropriate CLI command.
In general, these issues manifest in HIERARCHY_ERRORs
when a data transfer to the RDMS (e.g. via iput
or irsync
) is attempted via CLI.
To check all files at a RDMS location /rug/home/path/to/folder
including all its subfolders, and to detect just those files that are marked as locked, the following command can be executed:
$ iquest "status: %s, name: %s/%s" "SELECT DATA_REPL_STATUS, COLL_NAME, DATA_NAME WHERE COLL_NAME LIKE '/rug/home/path/to/folder%' AND DATA_REPL_STATUS <> '1'"
This command will check the specified location for files which have a replica status of 2 (“read-locked”) or 3 (“write-locked”), and then output it in the format:
status: <2/3>. name: <path_to_folder>/<name_of_file>
While the locked files cannot be directly removed, they can still be moved first to another location in your home/team location, for example as a separate folder for locked files. Afterwards, the data transfer can be restarted.
Best practices to handle locked files and resolve the HIERARCH_ERROR
are:
HIERARCHY_ERROR
should be resolved.Note: It is recommended not to contact RDMS support for every locked file, but instead first try to resolve it as described above. However, if numerous locked files are detected, you can directly contact RDMS support.