Data Transfer and File Size

Depending of the data size and amount of files, we advise different possibilities for files transfer in/out of the RDMS.

Single files under 32 MB can be directly up-/ and downloaded via the RDMS web interface.

To download a file via the web interface, just right-click on the file in question, then select Download file from the context menu.

Note: If the Download file button is not visible from the context menu, the file size exceeds the limit of 32 MB. In this case, use one of the other options that we advise for data transfer (see below).

To upload a file directly inside the web interface, you first navigate to the destination folder. Afterwards, select the and select Upload File from the menu.

The data transfer of larger files is not possible via the RDMS web interface, instead you have to use another client. Multiple options exist including WebDAV and command-line tools. With these, any size file can be transferred, but it is recommended to transfer the largest files at the command-line in Linux (see below).

WebDAV is a common file transfer protocol. For Windows, MacOS and Linux there are multiple WebDAV clients you can use.

We currently recommend and provide support for:

• Windows: Cyberduck and WinSCP
• MacOS: Finder
• For Linux: WebDAV on Linux

For very big data (sets) in the region of several hundred GB or higher, the iRODS-native iCommands are the best solution.

Here are three different ways to use them:

• Via a Linux OS: See the manual on installing and using iCommands on Linux
• Via Hábrók: The Hábrók HPC cluster has a fast connection to the RDMS and the iCommands are already pre-installed on the cluster. It is therefore also possible, to first transfer the data to Hábrók via SSH file transfer. The peregrine network is faster than the regular network, so chances for errors are smaller. Of course you do need an account for Peregrine, but then you can upload to peregrine:/scratch/<pnummer> and later on upload it to the RDMS.

• Under Windows: WSL2. This is a virtualization of the Linux kernel in the Windows Operating System. https://docs.microsoft.com/en-us/windows/wsl/wsl2-install. After installing WSL2 you need to install iCommands.

With iCommands you can unpack a zip or tar file with these small files. Uploading the files while in a zip or tar archive will tremendously reduce the operation time.