Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision 		Previous revision
rdms:data:filesize [2023/05/08 15:22]
jelte Changed some text 		rdms:data:filesize [2024/04/04 12:01] (current)
jelte spell correction
Line 3: Line 3:
 Depending of the data size and amount of files, we advise different possibilities for files transfer in/out of the RDMS. Depending of the data size and amount of files, we advise different possibilities for files transfer in/out of the RDMS.
  
-==== Under 32 MB ====+==== Data (Sets) under 50 MB ====
  
-Single files under 32 MB can be directly up-/ and downloaded via the [[https://research.web.rug.nl/rdmswebapp|RDMS web interface]].+Single files under 50 MB can be directly up-/ and downloaded via the [[https://research.web.rug.nl/rdmswebapp|RDMS web interface]].
  
 === Download Files in the Web Interface === === Download Files in the Web Interface ===
Line 13: Line 13:
 {{ :rdms:data:download_interface.png?600 |}} {{ :rdms:data:download_interface.png?600 |}}
  
-**Note:** If the ''Download file'' button is not visible from the context menu, the file size exceeds the limit of 32 MB. In this case, use one of the other options that we advise for data transfer (see below).+**Note:** If the ''Download file'' button is not visible from the context menu, the file size exceeds the limit of 50 MB. In this case, use one of the other options that we advise for data transfer (see below).
  
 === Upload Files in the Web Interface === === Upload Files in the Web Interface ===
Line 21: Line 21:
 {{ :rdms:data:upload_interface.png?600 |}} {{ :rdms:data:upload_interface.png?600 |}}
  
-==== Bigger than 32 MB ====+==== Data (Sets) over 50 MB ====
  
-The data transfer of larger files is not possible via the RDMS web interface, instead you have to use another client. Multiple options exist including WebDAV and command-line tools. With these, any size file can be transferred, but it is recommended to transfer the largest files at the command-line in Linux (see below).+The data transfer of larger files is not possible via the RDMS web interface. There are various clients and command-line tools such as ''iCommands'' to transfer data to RDMS, add metada or perform more advanced tasks. 
 + 
 +=== iCommands === 
 + 
 +''iCommands'' is an Unix utility that gives users a command-line interface. Currently it is the most efficient way of transferring data, creating metadata and perform more advanced tasks. In order to use the ''iCommands'' you need to work in a command-line tool such as a console, shell or a terminal. For further information please check the [[https://wiki.hpc.rug.nl/rdms/access/linux/createprofile|iCommands]] section of this manual.
  
 === WebDAV === === WebDAV ===
Line 32: Line 36:
  
   * Windows: [[rdms:access:windows:cyberduck|Cyberduck]] and [[rdms:access:windows:winscp|WinSCP]]   * Windows: [[rdms:access:windows:cyberduck|Cyberduck]] and [[rdms:access:windows:winscp|WinSCP]]
-  * MacOS: [[rdms:access:mac|Finder]]+  * MacOS: [[rdms:access:mac|Finder]] and [[rdms:access:windows:cyberduck|Cyberduck]]
   * For Linux: [[rdms:access:linux:webdav|WebDAV on Linux]]   * For Linux: [[rdms:access:linux:webdav|WebDAV on Linux]]
  
-==== Huge Data Sets ====+==== Large Data Sets ====
  
-For very big data (sets) in the region of several hundred GB or higher, the iRODS-native ''iCommands'' are the best solution.  +For large data (sets) in the order of several hundreds of GB or higher, the iRODS-native ''iCommands'' are the best solution.  
  
 Here are three different ways to use them: Here are three different ways to use them:
-  * Via a Linux OS: See the manual on installing and using [[rdms:access:linux|iCommands on Linux]] 
-  * Via Hábrók: The Hábrók HPC cluster has a fast connection to the RDMS and the ''iCommands'' are already pre-installed on the cluster. It is therefore also possible, to first transfer the data to Hábrók via [[https://wiki.hpc.rug.nl/habrok/data_management/transferring_data|SSH file transfer]].  The [[https://wiki.hpc.rug.nl/peregrine/connecting_to_the_system/start|peregrine network]] is faster than the regular network, so chances for errors are smaller.  Of course you do need an [[https://wiki.hpc.rug.nl/peregrine/introduction/policies#getting_access|account for Peregrine]], but then you can upload to peregrine:/scratch/<pnummer> and later on upload it to the RDMS. 
  
-  * Under Windows: [[https://docs.microsoft.com/en-us/windows/wsl/wsl2-about|WSL2]].  This is a virtualization of the Linux kernel in the Windows Operating System. https://docs.microsoft.com/en-us/windows/wsl/wsl2-install.  After installing WSL2 you need to [[rdms:access:linux:createprofile|install iCommands]]. +  * Via Hábrók cluster: The Hábrók HPC cluster has a [[https://wiki.hpc.rug.nl/habrok/introduction/cluster_description#network|fast network connection]] to the RDMS and the ''iCommands'' are already pre-installed on the cluster. It is therefore also possible, to first transfer the data to Hábrók via [[https://wiki.hpc.rug.nl/habrok/data_management/transferring_data|SSH file transfer]]. Of course you do need an [[https://wiki.hpc.rug.nl/peregrine/introduction/policies#getting_access|account for Hábrók]], but then you can upload to [[https://wiki.hpc.rug.nl/habrok/data_management/storage_areas?s[]=scratch|one of the storage locations]] that are available on the cluster. For the intermediate storage on the cluster for transfer to the RDMS, we recommend that you use the ''/scratch'' directory on Hábrók. 
 + 
 +  * Via a Linux OS: See the manual on installing and using [[rdms:access:linux|iCommands on Linux]].\\ If you do not have a Linux OS available, you can also use the [[https://iris.service.rug.nl/tas/public/ssp/content/detail/service?unid=8b36d7efb51048fa82ce480ff0206177|Linux workplaces (LWP)]] provided by the university. Within the LWP, the `iCommands` are pre-installed. There also exists a [[https://vlwp.rug.nl/nxwebplayer|virtual LWP version]] that can be access via the browser. For more details on the usage of the LWP, you can also refer to the [[https://lwpwiki.webhosting.rug.nl/index.php/Main_Page|LWP wiki]].  
 + 
 +  * Under Windows: Microsoft offers the possibility to install the 'Windows Subsystem for Linux', or short [[https://docs.microsoft.com/en-us/windows/wsl/wsl2-about|WSL2]].  This is a virtualization of the Linux kernel in the Windows OS. After successful [[https://docs.microsoft.com/en-us/windows/wsl/wsl2-install|installation]] of WSL2, you can [[rdms:access:linux:icommands|install iCommands]] within the virtual Linux environment and use it for data transfer.\\ **Note:** On university-managed Windows installation, so called 'UWP', the WSL cannot be installed and used.   
 + 
 +==== Large Number of Files ==== 
  
-==== A large set of small files ==== +Independent of the total size of the data, it is recommended that data sets that contain large number of single files (thousands of files and more), are uploaded as an archive, for example as a zip or a tar archives, and then extracted on the RDMS if needed. Uploading the files while in a zip/tar archive will tremendously reduce the operation time.
  
-With [[rdms:access:linux:createprofile|iCommands]] you can unpack a zip or tar file with these small files. Uploading the files while in a zip or tar archive will tremendously reduce the operation time.+The extraction of archived files on the RDMS is currently only possible with the ''iCommands'' which provides the `ibun` command to extract several archive formats that are stored in the RDMS. Please refer to the [[https://docs.irods.org/4.3.0/icommands/user/#ibun|official documentation]] of the ''ibun'' command to see example of its usage.