iCommands
The complete functionality to the RDMS comes with iCommands on the command-line interface (CLI).
iCommandsis the official CLI client by iRODS, the open-source software upon which the RDMS is built.iCommandsconsist of a whole set of different commands that allow to interact with the RDMS from the CLI. Next to commands relevant to general users (navigation, data transfer, etc.), it has commands specifically for metadata, and commands for management of groups.iCommandsare already installed on the LWP and Habrók HPC cluster. In these environments, you just need to:- Create a
~\.irodsdirectory in your home directory. - Create an
irods_environment.jsonfile in that directory.
- For other Linux distributions, explanation to install
iCommandsare also explained in this section.
Configure iCommands
For iCommands to work properly, a configuration file is needed. If the file does not exists on your system, the configuration file can be setup in the following way:
1. Create the .irods directory in your Linux home directory (~ is short for /home/<username>/ under Linux):
$ mkdir ~/.irods
2. Create the configuration file:
Create the file ~/.irods/irods_environment.json using your preferred text editor. The file must contain plain ASCII characters. Add the following lines to the file, modifying as instructed below:
Notes:
- The iRODS username
<emailadress>should be replaced by your UG email address. - Your username is case-sensitive, make sure to use all lowercase letters!
To see if everything was properly configures, you can now start the session by invoking the iinit command which will initialize your connection to the RDMS:
$ iinit Enter your current PAM password:
If you do not see an error after specifying your password, you are set to start using iCommands. See the section below for more information on using iCommands including some examples.
Note:
The iinit command will not just initialize your connection to the RDMS, but will also store your password in an obfuscated format under ~/.irods/.irodsA. Therefore, you do not have to re-type your password each time a subsequent command is executed during the same session.
Installing iCommands
Packages for iCommands are currently provided for CentOS 7, EL 8/9 (e.g., Rocky Linux or AlmaLinux), Ubuntu 18/20/22 and Debian 11/12 (https://irods.org/download/). iCommands can be installed on other Linux distributions too, but special configurations will be needed. For such cases, we recommend the use of a virtual machine that runs one of the officially supported Linux distributions.
Installation on Ubuntu and Debian
Install the public key and add the iRODS repository
$ wget -qO - https://packages.irods.org/irods-signing-key.asc | sudo apt-key add - $ echo "deb [arch=amd64] https://packages.irods.org/apt/ $(lsb_release -sc) main" | sudo tee /etc/apt/sources.list.d/renci-irods.list $ sudo apt-get update
Install iCommands
$ sudo apt install irods-icommands
Installation on EL-based Distributions
Before you can install iCommands on a RedHat-based Linux distributions (CentOS, Rocky Linux, and AlmaLinux), be sure that the necessery dependencies can be satisfied.
Install the Extra Packages for Enterprise Linux (EPEL) repository
$ sudo yum install epel-release
Install external dependencies
$ sudo yum install python-psutil python-jsonschema
Install the public key and add the iRODS repository
$ sudo rpm --import https://packages.irods.org/irods-signing-key.asc $ wget -qO - https://packages.irods.org/renci-irods.yum.repo | sudo tee /etc/yum.repos.d/renci-irods.yum.repo
Install iCommands
$ sudo yum install irods-icommands
Using iCommands
In this section, we will describe the usage of different available iCommands in more detail. Examples will be given for the different iCommands showing their possible usage. Please keep in mind that the iCommands provide extensive functionality and not all will be described in this wiki.
Furthermore, different iCommands can be included in (computing) scripts to automate tasks if needed which will not covered in this section as it is specific for the respective use case.
If you have additional question about functionality of iCommands in general or specific subcommands, you can contact rdms-support@rug.nl.
Next to the information provided on this wiki, a plethora of external guides on iCommands and its usage exists. Some important to mention are:
Getting Help for specific iCommands
As with linux commands, there is built in help for all iCommands which can be shown by using the addition -h flag with the respective command. For example, help for the ils command is displayed like:
$ ils -h
which will display the help of the respective command.
Furthermore, you can run ihelp to get an overview of all available iCommands.
Common iCommands Usage Examples
For the explanation of commonly used iCommands, we will split the iCommands into three different sections:
- iCommands for Navigation will describe the iCommands that are used to navigate the RDMS from the CLI, but also contain information about starting/ending of the connection.
- iCommands for (Metda)data Management will describe how to transfer data in/out of the RDMS, explain how to set permissions for other users, how to work with metadata from the CLI, including a more detailed description on how to query the database for more extensive searches.
- iCommands for Group Management will focus on the respective commands that can be used by 'groupadmins' to manage RDMS groups from the CLI. As this section is not relevant for all users, it will be discussed separately.
iCommands for Navigation
We already saw the iinit command in action which initializes a new RDMS session and also stored the obfuscated password for re-use with following iCommands.
To again close a session from the CLI, the iexit command can be used.
If a session is initialized, we can use the ipwd subcommand to display our current working directory within the RDMS:
To reveal the content of a directory, we can use the ils <path>. When executing ils without specifying an additional RDMS path, it will show us the content of our current working directory.
When used without additional flags, ils will just list files and folder (folders = collections in iRODS terms, marked by 'C-' in front of data name), but will not provide any additional information about file sizes or permission information. If we want to see these information, we can use the additional -A flag to get information about file permissions and inheritance, and the -l or -L flag for (very) long output which will display additional information, for example file sizes. As usual, flags can be combined for a specific command.
The ils command can be also used with the additional -r to perform a recursive operation, in which case it will not just display the output of the current RDMS directory, but instead also the content of its lower-level folders.
Alternatively, the itree command can be executed to get a similar overview of the folder/file structure at a specific RDMS location. Moreover, itree also displays a summary of files (data objects) and folder (collections) that were found at the specified location. Running itree without specifying a path will again display results for the current working directory.
If we want to navigate to another location inside of the RDMS, we can use the icd command. For navigation with icd the destination path can be either specified relatively to the current working directory or given as an absolute path. Furthermore, executing icd without any additional path, will always navigate to your RDMS home location.
iCommands for (Metda)data Management
Upload to the RDMS
To transfer data to the the RDMS, the iput command can be used. It is of the general syntax iput [flags] <source> <dest> where <source> points to files/folders on your local environment while <dest> specifies the destination on the RDMS. The destination entry can be also left empty in which case the data will be transferred to the current RDMS working directory. Specified paths can be relative or absolute.
If you want to transfer a whole folder structure, the additional -r flag has to be used.
If you want to see the transfer of the data transfer displayed, you can use the additional -P flag.
Notes:
- It is also recommended to use the
-Kwhich will calculate and compare the checksum of the files that are transferred. - For long-lasting transfers, it is furthermore recommended to use the
-Tflag which will renew the socket connection after 10 minutes which can eliminate possible errors resulting from time-outs due to firewall settings. - An additional option exists to specify a restart file using the
-Xflag. For uploading directories, this can be used to restart the transfer if needed. This can be combined with the–retries [N]flag to automatically restart the transfer [N] times if needed. - For improved transfer performance, especially for data sets containing a lot of small, individual files, it is recommend to upload the files in a bundled format (e.g., tar archive) and then bundle within the RDMS if desired (see the section about (un)bundling below).
- If you want to overwrite a file that is already present in the system, use the
-f(force) flag to overwrite, but be cautious when overwriting data.
Download from the RDMS
For downloads of data from the RDMS, the iget command can be used. Its general syntax, iget [flags] <source> <dest>, is identical to iput, except that <source> is this time the RDMS side while <dest> specifies the destination on your local file system.
Also, the same flags can be used and the above mentioned notes for the usage of iput also apply for iget.
Synchronization with the RDMS
Another method for the transfer of data is the usage of the irsync command which synchronizes a source directory with a target directory. The distinction is made by placing a i: in front of the source/destination paths.
Notes:
- The
irsynccommand also allows to specify multiple sources that can be synced to one destination. - To see detailed information about the progress of the synchronization, the additional
-v(verbose) flag can be used. - The
-Kflag can be used to calculate and compare checksum for the transferred files on the source and destination side. This is recommended. - The
irsynccommand does not have a specific force flag to overwrite files. Instead,irsyncworks in a way that it detects new/changed files on the specified source and uploads them to the target destination. Already existing files that were not changed are not touch/transferred byirsyncwhile changed files get overwritten on the destination!
With this notes in mind, a typical irsync command with recommend flags would look like:
(Un)bundling
The ibun command allows to bundle directories or extract a bundled directories within the RDMS. Without specifying additional flags, tar bundles will be created when bundling a directory, but it is also possible to specify other types (available are: tar, gzip, bzip2 and zip).
The additional -c flag is used to indicate bundling. Additionally, you have to also specify which RDMS directory you want to bundle as well as a name for the created bundle
The additional -x flag is used to indicate extraction. Additionally, you have to also specify which bundle you want to extract as well as specify a RDMS directory to which the content of the bundle is extracted.
Notes:
- The bundling and extraction operations do not display a progress. For large folders/archives, it might take a longer time to finish while your terminal seems to be stuck.
- If you want to overwrite, you can again use the additional
-fflag.
Assigning Data Permission
For the assignment of access permissions, the ichmod command is used. Currently, it is possible to assign the following four permission levels: 'Null', 'Read', 'Write', 'Own'.
The general syntax to assign permission is ichmod [-r] <access_level> <user-/groupname> <path> where <access_level> is one of the mentioned four permission levels, <user/-groupname> the name of the user or group that should get the new permissions, and <path> is the RDMS path for the files/folders that you want to assign the new permissions.
If you plan to assign the permission recursively on a folders, meaning also to assign the new permissions for all its subfolders and contained files, you can run the command with the additional -r flag.
It is also possible to enable/disable permission inheritance of a folder using the ichmod command.
Please refer to the Wiki section about permissions and inheritance to get more information about the different permission levels as well as the meaning of inheritance.
Notes:
If you are unsure about the currently set permissions on a file/folder or you are unsure if inheritance is off/on, run the ils -A <path> to display these information.
Working with Metadata
The iCommands can be also used to add metadata to a file/folder, list already existing metadata, modify existing metadata, and also query (search) for objects that contain certain metadata.
Within the RDMS, metadata is always added as so called 'attribute-value-units' triples, AVUs. The 'attribute' specifies the metadata key (e.g. 'Author') while the 'value' specifies the corresponding value for this key (e.g. 'John Doe'). The 'unit' entry is optional, but can be useful in cases where the entries could be otherwise ambiguous (e.g. A: 'Distance:, V: '100', U: 'km'). This can be left blank if no unit is desired.
The command that is used to work with metadata from the CLI, is the imeta command, which has additional subcommands depending on the desired operation :
- List current metadata:
imeta [-d, -C] <path> - Add new metadata:
imeta add [-d, -C] <path> - Modify metadata:
imeta mod [-d, -C] <path> <old_attribute> <old_value> <old_unit> n:<new_attribute> v:<new_value> u:<new_unit> - Query for metadata:
imeta qu <attribute> <Operator> <value>
Notes:
- The
imetacommand requires that you also specify the additional-Cor-dflag. - Use
-Cif you want to work with folder (collection). For exampleimeta ls -C <foldername>. - Use
-dif you want to work with a file (data object). For exampleimeta ls -d <filename>. - There are additional help pages available for the different
imetasubcommands which can be displayed by executingimeta help <subcommand>. For example:imeta help quto display more information of the query (search) command.
Search for Data
The iCommands provide different options to search for files/folders:
- If you know the name of the file/folder, you can simply use the
ilocatecommand which will display you the full RDMS path if files/folders by this name were found in the system. - If you want to do simple search by using metadata, you can use the
imeta [-C, -d] ququery. With this, you can also combine multiple metadata entries (see example below) for a more granular query/search. - The
iquestcommand can be used for even more powerful queries/searches within the database. This is a very powerful search that can be (for example) also used to count the amount of files in a certain RDMS location. It uses SQL-like terminology for the queries and also allows to use wildcards for the queries. The help for this command (executeiquest -hcontains a more detailed description about the syntax as well as some examples.
iCommands for Group Management
To manage RDMS Groups, the igroupadmin command is used. This command can be used to list group members (igroupadmin lg <groupname>), add/remove users from groups (igroupadmin <atg/rfg> <groupname> <emailadress>), or even to create a new group (igroupadmin mkgroup <groupname>).
iCommands Cheat Sheet
The following table contains an overview of the most useful iCommands with a short description.
| iCommand | Meaning | |
|---|---|---|
ihelp <command> | Print help of <command> or displays all available iCommands if no command is specified | |
iinit | Initialize connection to the RDMS and also save the password in scrambled format for use with subsequent iCommands | |
iexit | Close connection to the RDMS and also remove the scrambled password file | |
ipwd | Displays your current RDMS working directory | |
icd | Change the current working directory | |
ils [-A, -L, -r] | Listing files and folders-A: Also display access and inheritance information-L: Displays more information (long format)-r: List recursively (also display content of sub folders) | |
itree [-c, -f, -s] | Display folder structure as a tree view-c: Just display folders (collections), no files-f: Display full path of listed items-s: Also display file sizes in the tree view | |
imkdir [-p] | Create a new folder in the RDMS-p: Also create parent folders if they do not exist | |
irm [-r, -f] | Move files/folders to trash-r: Recursive (delete whole folder structures)-f: Do not move to trash, but delete permanently | |
irmtrash | Empty your trash | |
imv | Move files/folders within the RDMS | |
icp [-K, -r, -f] | Copy files/folders within the RDMS-K: Calculate and verify checksum during copying -r: Recursive (copy whole folder structures)-f: Force, overwrites if files/folders already exist | |
ilocate | Locate files in the RDMS | |
ichmod [-r] | Change access permission and enable/disable permission inheritance-r: Recursive (change for whole folder structures) | |
imeta add [-d, -C] | Add metadata-d: Used for files (data objects)-C: Used for folders (collections) | |
imeta ls [-d, -C] | List metadata-d: Used for files (data objects)-C: Used for folders (collections) | |
iget [-K, -r, -P, -T, -X] | Download data from the RDMS-K: Calculate and verify checksum-r: Download whole folder structure (recursive)-P: Display progress during transfer-T: Renew socket connection (long transfers)-X: Specify restart file (long transfers) | |
iput [-K, -r, -P, -T, -X] | Upload data to the RDMS-K: Calculate and verify checksum-r: Upload whole folder structure (recursive)-P: Display progress during transfer-T: Renew socket connection (long transfers)-X: Specify restart file (long transfers) | |
irsync [-K] | Synchronize data between RDMS and local | |
ibun [-x, -c, -D] | (Un)bundle data in the RDMS | |
iuserinfo <user> | Display information (name, groups, etc) of a user, displays information about yourself if no user is specified | |
iquest | Execute complex queries using metadata | |
igroupadmin | Manage RDMS Groups (just for group admins) |
Further Tips and Tricks
This section contains some additional tips and tricks that make working with iCommands more effective.
Enable Bash Auto-Completion
Most Linux distribution come by default with the so called Bourne Again Shell (Bash) which has an internal auto-completion function via “tab key”. This can be used to, for example, auto-complete certain paths when using the standard Linux commands (e.g., ls, cd).
This auto-completion does not work by default for the iCommands which means that for example using ils with the “tab key” to auto-complete an RDMS path will not work. Luckily, the iRODS community provides a helpful irods_completion.bash shell script via its iRODS contrib Github page that can be used to enable this functionality.
Temporarily enable Bash Auto-Completion
As a first step, you need to download the irods_completion.bash shell script from the mentioned Github repository. The Linux commands cURL or wget can be used for this directly from the terminal.
In this example, we will assume that the auto-completion script will be downloaded and saved in your Linux home directory (abbreviated as ~).
To download the shell script to your home location, open a terminal window and then execute:
# Using wget $ wget https://raw.githubusercontent.com/irods/contrib/main/irods_completion.bash -P ~/ # Using cURL $ curl https://raw.githubusercontent.com/irods/contrib/main/irods_completion.bash > ~/irods_completion.bash
After successfully downloading the script, you can enable it temporarily (for your current shell session) by executing:
$ source ~/irods_completion.bash
Afterwards, the “tab key” can be used for auto-completion on the RDMS side. If you hit the “tab key” two times in a row, your terminal will display all available paths in the current location that fits your auto-completion criteria.
Permanently enable Bash Auto-Completion
It is also possible to automatically “source” the auto_completion.bash script every time a new Bash shell is started.
To do so, you have to first download the shell script as described above. Afterwards, we will edit the ~/.bashrc file which is automatically loaded when a Bash session is started so that it automatically loads/sources the auto-completion script.
While you can use your favorite text editor for this, we will use nano in this example. In your Linux terminal, execute:
$ nano ~/.bashrc
Now add source ~/irods_completion.bash to the end of the opened file, and then save (nano: Ctrl + o) the file and exit your text editor (nano: Ctrl + x).
From now on the auto-completion script will be loaded by default when you start a new shell session. If you also want to enable the auto-completion for your current session, you have to once execute:
$ source ~/.bashrc
Notes
- The auto-completion script that is described in this section is specific for the Bash shell. If you are unsure if you are using the correct shell, you can execute
echo $SHELLin your terminal which will show you the default shell in your system. - If you want to disable the automated loading of the auto-completion script, you can edit the
~/.bashrcfile again. Either delete the part that containssource ~/irods_completion.bashor comment it out by adding a “#” in front of the line.