====== iCommands ====== The complete functionality to the RDMS comes with ''iCommands'' on the command-line interface (CLI). * ''iCommands'' is the official CLI client by iRODS, the open-source software upon which the RDMS is built. * ''iCommands'' consist of a whole set of different commands that allow to interact with the RDMS from the CLI. Next to [[https://docs.irods.org/4.3.0/icommands/user/|commands relevant to general users]] (navigation, data transfer, etc.), it has [[https://docs.irods.org/4.3.0/icommands/metadata/|commands specifically for metadata]], and [[https://docs.irods.org/4.3.0/icommands/groupadmin/|commands for management of groups]]. * ''iCommands'' are already installed on the LWP and Habrók HPC cluster. In these environments, you just need to: - Create a ''~\.irods'' directory in your home directory. - Create an ''irods_environment.json'' file in that directory. * For other Linux distributions, explanation to install ''iCommands'' are also explained in this section. ===== Configure iCommands ===== For ''iCommands'' to work properly, a configuration file is needed. If the file does not exists on your system, the configuration file can be setup in the following way: 1. Create the ''.irods'' directory in your Linux home directory (''~'' is short for ''/home//'' under Linux): $ mkdir ~/.irods 2. Create the configuration file: Create the file ''~/.irods/irods_environment.json'' using your preferred text editor. The file must contain plain ASCII characters. Add the following lines to the file, modifying as instructed below: { "irods_authentication_scheme": "PAM", "irods_client_server_negotiation": "request_server_negotiation", "irods_client_server_policy": "CS_NEG_REQUIRE", "irods_default_resource": "rootResc", "irods_encryption_algorithm": "AES-256-CBC", "irods_encryption_key_size": 32, "irods_encryption_num_hash_rounds": 16, "irods_encryption_salt_size": 8, "irods_host": "store.data.rug.nl", "irods_port": 1247, "irods_ssl_verify_server": "cert", "irods_user_name": "", "irods_zone_name": "rug" } \\ **Notes:** * The iRODS username '''' should be replaced by your UG email address. * Your username is case-sensitive, make sure to use all lowercase letters! To see if everything was properly configures, you can now start the session by invoking the ''iinit'' command which will initialize your connection to the RDMS: $ iinit Enter your current PAM password: If you do not see an error after specifying your password, you are set to start using ''iCommands''. See the section below for more information on using ''iCommands'' including some examples. **Note:**\\ The ''iinit'' command will not just initialize your connection to the RDMS, but will also store your password in an obfuscated format under ''~/.irods/.irodsA''. Therefore, you do not have to re-type your password each time a subsequent command is executed during the same session. ===== Installing iCommands ===== Packages for iCommands are currently provided for **CentOS 7**, **EL 8/9 (e.g., Rocky Linux or AlmaLinux)**, **Ubuntu 18/20/22** and **Debian 11/12** (https://irods.org/download/). iCommands can be installed on other Linux distributions too, but special configurations will be needed. For such cases, we recommend the use of a virtual machine that runs one of the officially supported Linux distributions. ==== Installation on Ubuntu and Debian ==== **Install the public key and add the iRODS repository** $ wget -qO - https://packages.irods.org/irods-signing-key.asc | sudo apt-key add - $ echo "deb [arch=amd64] https://packages.irods.org/apt/ $(lsb_release -sc) main" | sudo tee /etc/apt/sources.list.d/renci-irods.list $ sudo apt-get update **Install iCommands** $ sudo apt install irods-icommands **Additional Notes for Users of Ubuntu 20+ and Debian 11+** On Debian-based operating systems, the ''apt-key add'' feature was deprecated, and you might see the following message in the first step: $ wget -qO - https://packages.irods.org/irods-signing-key.asc | sudo apt-key add - Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). OK As of now, the warning can be still ignored and the steps can be performed as described, but in the longer term it is highly advisable to add the repository signing key in another manner as described, for example, [[https://www.linuxuprising.com/2021/01/apt-key-is-deprecated-how-to-add.html|here]]. As of now, iRODS has not yet decided on the best way to handle this in the case of the iRODS package repository, and this information will be updated accordingly in the future. ==== Installation on EL-based Distributions ==== Before you can install iCommands on a RedHat-based Linux distributions (CentOS, Rocky Linux, and AlmaLinux), be sure that the necessery dependencies can be satisfied. **Install the Extra Packages for Enterprise Linux (EPEL) repository** $ sudo yum install epel-release **Install external dependencies** $ sudo yum install python-psutil python-jsonschema **Install the public key and add the iRODS repository** $ sudo rpm --import https://packages.irods.org/irods-signing-key.asc $ wget -qO - https://packages.irods.org/renci-irods.yum.repo | sudo tee /etc/yum.repos.d/renci-irods.yum.repo **Install iCommands** $ sudo yum install irods-icommands ===== Using iCommands ===== In this section, we will describe the usage of different available ''iCommands'' in more detail. Examples will be given for the different ''iCommands'' showing their possible usage. Please keep in mind that the ''iCommands'' provide extensive functionality and not all will be described in this wiki. Furthermore, different ''iCommands'' can be included in (computing) scripts to automate tasks if needed which will not covered in this section as it is specific for the respective use case. If you have additional question about functionality of ''iCommands'' in general or specific subcommands, you can contact . Next to the information provided on this wiki, a plethora of external guides on ''iCommands'' and its usage exists. Some important to mention are: * [[https://docs.vscentrum.be/en/latest/data/iCommands.html|iCommands Guide by the Vlaams Supercomputing Center]] * [[https://github.com/kuleuven/iRODS-User-Training/blob/main/05_iCommands_Handson_User-Training.md|KU Leuven iCommands User Training]] * [[https://servicedesk.surf.nl/wiki/display/WIKI/How-to+guides+for+iRODS@SURF|Guide iRODS@SURF]] ==== Getting Help for specific iCommands ==== As with linux commands, there is built in help for all ''iCommands'' which can be shown by using the addition ''-h'' flag with the respective command. For example, help for the ''ils'' command is displayed like: $ ils -h which will display the help of the respective command. $ ils -h Usage: ils [-ArlLv] dataObj|collection ... Usage: ils --bundle [-r] dataObj|collection ... Display data objects and collections stored in iRODS. The following is typical output for ils with no arguments: $ ils /tempZone/public: big foo The only information displayed are the names of data objects in the current working collection. The long format (given by -l) will display some more information: $ ils -l /tempZone/public: rods 0 demoResc 41943040 2021-04-29.17:57 & big rods 2 comp;arch 40236581 2021-04-19.03:00 X big alice 2 repl1;child1 283 2021-04-29.17:54 ? foo alice 3 repl1;child2 283 2021-04-29.17:56 ? foo alice 4 repl1;child3 283 2021-04-29.17:57 ? foo This displays replica information and system metadata for the replicas of the data objects in the current working collection. This information includes the owner of the data object and the replica's number, resource hierarchy, size, last modified time, and status. The very long format (given by -L) will display even more information: $ ils -L /tempZone/public: rods 0 demoResc 41943040 2021-04-29.17:57 & big sha2:gKNyEYjkAhiwiyZ3a8U72ugeR4T/9x1xRQoZcxnLoRM= generic /var/lib/irods/Vault/public/big rods 2 comp;arch 40236581 2021-04-19.03:00 X big sha2:KNyEYjkhiwyZ3a8U72ugeR4T/9x1xRQoZcxnLoRMigg= generic /var/lib/irods/archVault/public/big alice 2 repl1;child1 283 2021-04-29.17:54 ? foo generic /var/lib/irods/child1vault/public/foo alice 3 repl1;child2 283 2021-04-29.17:56 ? foo generic /var/lib/irods/child2vault/public/foo alice 4 repl1;child3 283 2021-04-29.17:57 ? foo generic /var/lib/irods/child3vault/public/foo This displays even more replica information and system metadata such as each replica's checksum, data type, and physical location in the storage resource. The different replica statuses are represented by the following characters which can be seen between the last modified time and the data object name. 0. X: stale - The replica is no longer known to be good. Usually indicates that a more recently written-to replica exists. This does not necessarily mean that the data is incorrect but merely that it may not reflect the "truth" of this data object. 1. &: good - The replica is good. Usually the latest replica to be written. When a replica is good, all bytes and system metadata (size and checksum, if present) are understood to have been recorded correctly. 2. ?: intermediate - The replica is actively being written to. The state of the replica cannot be determined because the client writing to it still has the replica opened. Replicas which are intermediate cannot be opened for read or write, nor can they be unlinked or renamed. 3. ?: write-locked - One of this replica's sibling replicas is actively being written to but is itself at rest. Replicas which are write-locked cannot be opened for read or write nor can they be unlinked or renamed. These replica statuses correspond with the data_is_dirty column in r_data_main. Options are: -A ACL (access control list) and inheritance format -d List collections themselves, not their contents -l long format -L very long format -r recursive - show subcollections -t ticket - use a read (or write) ticket to access collection information -v verbose -V Very verbose -h this help --bundle - list the subfiles in the bundle file (usually stored in the /myZone/bundle collection) created by iphybun command. iRODS Version 4.3.0 ils \\ Furthermore, you can run ''ihelp'' to get an overview of all available ''iCommands''. ==== Common iCommands Usage Examples ==== For the explanation of commonly used ''iCommands'', we will split the ''iCommands'' into three different sections: - **iCommands for Navigation** will describe the iCommands that are used to navigate the RDMS from the CLI, but also contain information about starting/ending of the connection. - **iCommands for (Metda)data Management** will describe how to transfer data in/out of the RDMS, explain how to set permissions for other users, how to work with metadata from the CLI, including a more detailed description on how to query the database for more extensive searches. - **iCommands for Group Management** will focus on the respective commands that can be used by 'groupadmins' to manage RDMS groups from the CLI. As this section is not relevant for all users, it will be discussed separately. === iCommands for Navigation === We already saw the ''iinit'' command in action which initializes a new RDMS session and also stored the obfuscated password for re-use with following ''iCommands''. To again close a session from the CLI, the ''iexit'' command can be used. If a session is initialized, we can use the ''ipwd'' subcommand to display our current working directory within the RDMS: $ ipwd /rug/home/rdms-testers@rug.nl \\ To reveal the content of a directory, we can use the ''ils ''. When executing ''ils'' without specifying an additional RDMS path, it will show us the content of our current working directory. $ ils /rug/home/rdms-testers@rug.nl: example_file C- /rug/home/rdms-testers@rug.nl/some_example_folder C- /rug/home/rdms-testers@rug.nl/some_other_folder \\ When used without additional flags, ''ils'' will just list files and folder (folders = collections in iRODS terms, marked by 'C-' in front of data name), but will not provide any additional information about file sizes or permission information. If we want to see these information, we can use the additional ''-A'' flag to get information about file permissions and inheritance, and the ''-l'' or ''-L'' flag for (very) long output which will display additional information, for example file sizes. As usual, flags can be combined for a specific command. $ ils -A /rug/home/rdms-testers@rug.nl: ACL - rdms-testers@rug.nl#rug:own Inheritance - Disabled example_file ACL - rdms-testers@rug.nl#rug:own example_file2 ACL - rdms-testers@rug.nl#rug:own C- /rug/home/rdms-testers@rug.nl/some_example_folder C- /rug/home/rdms-testers@rug.nl/some_other_folder $ ils -l /rug/home/rdms-testers@rug.nl: rdms-testers 0 rootResc;rootRandy;ptC;replC;randy20;pt223;mnt_irods223 0 2023-07-21.13:05 & example_file rdms-testers 1 rootResc;rootRandy;ptC;replC;randy11;pt124;mnt_irods124 0 2023-07-21.13:05 & example_file rdms-testers 0 rootResc;rootRandy;ptC;replC;randy11;pt103;mnt_irods103 44 2023-07-21.13:08 & example_file2 rdms-testers 1 rootResc;rootRandy;ptC;replC;randy20;pt223;mnt_irods223 44 2023-07-21.13:08 & example_file2 C- /rug/home/rdms-testers@rug.nl/some_example_folder C- /rug/home/rdms-testers@rug.nl/some_other_folder $ ils -lA /rug/home/rdms-testers@rug.nl: ACL - rdms-testers@rug.nl#rug:own Inheritance - Disabled rdms-testers 0 rootResc;rootRandy;ptC;replC;randy20;pt223;mnt_irods223 0 2023-07-21.13:05 & example_file ACL - rdms-testers@rug.nl#rug:own rdms-testers 1 rootResc;rootRandy;ptC;replC;randy11;pt124;mnt_irods124 0 2023-07-21.13:05 & example_file ACL - rdms-testers@rug.nl#rug:own rdms-testers 0 rootResc;rootRandy;ptC;replC;randy11;pt103;mnt_irods103 44 2023-07-21.13:08 & example_file2 ACL - rdms-testers@rug.nl#rug:own rdms-testers 1 rootResc;rootRandy;ptC;replC;randy20;pt223;mnt_irods223 44 2023-07-21.13:08 & example_file2 ACL - rdms-testers@rug.nl#rug:own C- /rug/home/rdms-testers@rug.nl/some_example_folder C- /rug/home/rdms-testers@rug.nl/some_other_folder \\ The ''ils'' command can be also used with the additional ''-r'' to perform a //recursive// operation, in which case it will not just display the output of the current RDMS directory, but instead also the content of its lower-level folders. $ ils -r /rug/home/rdms-testers@rug.nl: example_file example_file2 C- /rug/home/rdms-testers@rug.nl/some_example_folder /rug/home/rdms-testers@rug.nl/some_example_folder: C- /rug/home/rdms-testers@rug.nl/some_other_folder /rug/home/rdms-testers@rug.nl/some_other_folder: C- /rug/home/rdms-testers@rug.nl/some_other_folder/a_subfolder1 /rug/home/rdms-testers@rug.nl/some_other_folder/a_subfolder1: more_example_files_1 more_example_files_10 more_example_files_2 more_example_files_3 more_example_files_4 more_example_files_5 more_example_files_6 more_example_files_7 more_example_files_8 more_example_files_9 \\ Alternatively, the ''itree'' command can be executed to get a similar overview of the folder/file structure at a specific RDMS location. Moreover, ''itree'' also displays a summary of files (data objects) and folder (collections) that were found at the specified location. Running ''itree'' without specifying a path will again display results for the current working directory. $ itree /rug/home/rdms-testers@rug.nl/some_other_folder/a_subfolder1 a_subfolder1 more_example_files_1 more_example_files_10 more_example_files_2 more_example_files_3 more_example_files_4 more_example_files_5 more_example_files_6 more_example_files_7 more_example_files_8 more_example_files_9 Found 0 collections and 10 data objects \\ If we want to navigate to another location inside of the RDMS, we can use the ''icd'' command. For navigation with ''icd'' the destination path can be either specified relatively to the current working directory or given as an absolute path. Furthermore, executing ''icd'' without any additional path, will always navigate to your RDMS home location. # Example using icd and relative file path $ ils /rug/home/rdms-testers@rug.nl: example_file example_file2 C- /rug/home/rdms-testers@rug.nl/some_example_folder C- /rug/home/rdms-testers@rug.nl/some_other_folder $ icd some_example_folder $ ipwd /rug/home/rdms-testers@rug.nl/some_example_folder # Example using icd and absolute file path $ icd /rug/home/rdms-testers@rug.nl/some_other_folder/a_subfolder1 $ ipwd /rug/home/rdms-testers@rug.nl/some_other_folder/a_subfolder1 # Using icd without path, always navigates to RDMS Home directory $ icd $ ipwd /rug/home/rdms-testers@rug.nl === iCommands for (Metda)data Management === **Upload to the RDMS**\\ To transfer data to the the RDMS, the ''iput'' command can be used. It is of the general syntax ''iput [flags] '' where '''' points to files/folders on your local environment while '''' specifies the destination on the RDMS. The destination entry can be also left empty in which case the data will be transferred to the current RDMS working directory. Specified paths can be relative or absolute. # Transfer a file to the RDMS $ iput test.txt /rug/home/rdms-testers@rug.nl/some_example_folder $ ils /rug/home/rdms-testers@rug.nl/some_example_folder /rug/home/rdms-testers@rug.nl/some_example_folder: test.txt \\ If you want to transfer a whole folder structure, the additional ''-r'' flag has to be used. $ iput -r Testdir_L1 Running recursive pre-scan... pre-scan complete... transferring data... $ itree Testdir_L1 Testdir_L1 Testdir_L2a Testfile_1 Testfile_2 Testfile_3 Testfile_4 Testfile_5 Testdir_L2b Testfile_1 Testfile_2 Testfile_3 Testfile_4 Testfile_5 Testdir_L2c Testfile_1 Testfile_2 Testfile_3 Testfile_4 Testfile_5 Testdir_L2d Testfile_1 Testfile_2 Testfile_3 Testfile_4 Testfile_5 testfile Found 4 collections and 21 data objects \\ If you want to see the transfer of the data transfer displayed, you can use the additional ''-P'' flag. $ iput -P 1GB_Testfile_1 0/1 - 0.00% of files done 0.000/953.674 MB - 0.00% of file sizes done Processing 1GB_Testfile_1 - 953.674 MB 2023-07-21.14:21:16 1GB_Testfile_1 - 360.000/953.674 MB - 37.75% done 2023-07-21.14:21:23 1GB_Testfile_1 - 796.837/953.674 MB - 83.55% done 2023-07-21.14:21:26 1GB_Testfile_1 - 953.674/953.674 MB - 100.00% done 2023-07-21.14:21:26 \\ **Notes:** * It is also recommended to use the ''-K'' which will calculate and compare the checksum of the files that are transferred. * For long-lasting transfers, it is furthermore recommended to use the ''-T'' flag which will renew the socket connection after 10 minutes which can eliminate possible errors resulting from time-outs due to firewall settings. * An additional option exists to specify a restart file using the ''-X'' flag. For uploading directories, this can be used to restart the transfer if needed. This can be combined with the ''--retries [N]'' flag to automatically restart the transfer [N] times if needed. * For improved transfer performance, especially for data sets containing a lot of small, individual files, it is recommend to upload the files in a bundled format (e.g., tar archive) and then bundle within the RDMS if desired (see the section about (un)bundling below). * If you want to overwrite a file that is already present in the system, use the ''-f'' (force) flag to overwrite, but be cautious when overwriting data. **Download from the RDMS**\\ For downloads of data from the RDMS, the ''iget'' command can be used. Its general syntax, ''iget [flags] '', is identical to ''iput'', except that '''' is this time the RDMS side while '''' specifies the destination on your local file system. Also, the same flags can be used and the above mentioned notes for the usage of ''iput'' also apply for ''iget''. **Synchronization with the RDMS**\\ Another method for the transfer of data is the usage of the ''irsync'' command which synchronizes a source directory with a target directory. The distinction is made by placing a ''i:'' in front of the source/destination paths. # Synchronize the content of local folder to a folder on the RDMS: $ irsync -r i: Synchronize the content of remote folder on the RDMS with a local folder: $ irsync -r i: Synchronize the content of two remote RDMS folders: $ irsync -r i: i: \\ **Notes:** * The ''irsync'' command also allows to specify multiple sources that can be synced to one destination. * To see detailed information about the progress of the synchronization, the additional ''-v'' (verbose) flag can be used. * The ''-K'' flag can be used to calculate and compare checksum for the transferred files on the source and destination side. This is recommended. * The ''irsync'' command does not have a specific force flag to overwrite files. Instead, ''irsync'' works in a way that it detects new/changed files on the specified source and uploads them to the target destination. Already existing files that were not changed are not touch/transferred by ''irsync'' while changed files get overwritten on the destination! With this notes in mind, a typical ''irsync'' command with recommend flags would look like: $ irsync -rvK git_projects/nmr_meta_parser/ i:/rug/home/rdms-testers@rug.nl/nmr_meta_parser/ Running recursive pre-scan... pre-scan complete... transferring data... C- /rug/home/rdms-testers@rug.nl/nmr_meta_parser: nmr_parser.py 0.001 MB | 2.293 sec | 0 thr | 0.000 MB/s nmr_meta_main.py 0.001 MB | 1.270 sec | 0 thr | 0.001 MB/s nmr_meta_binder.py 0.007 MB | 2.514 sec | 0 thr | 0.003 MB/s C- /rug/home/rdms-testers@rug.nl/nmr_meta_parser/parsers: experiment_parser.py 0.001 MB | 1.650 sec | 0 thr | 0.000 MB/s C- /rug/home/rdms-testers@rug.nl/nmr_meta_parser/parsers/__pycache__: bruker.cpython-310.pyc 0.004 MB | 1.205 sec | 0 thr | 0.003 MB/s experiment_parser.cpython 0.001 MB | 1.613 sec | 0 thr | 0.001 MB/s varian.cpython-310.pyc 0.004 MB | 0.417 sec | 0 thr | 0.009 MB/s varian.py 0.005 MB | 0.241 sec | 0 thr | 0.020 MB/s bruker.py 0.004 MB | 0.242 sec | 0 thr | 0.017 MB/s __init__ 0.000 MB | 0.201 sec | 0 thr | 0.000 MB/s C- /rug/home/rdms-testers@rug.nl/nmr_meta_parser/__pycache__: parse_utils.cpython-310.p 0.002 MB | 0.321 sec | 0 thr | 0.006 MB/s nmr_meta_cl_parser.cpytho 0.003 MB | 0.341 sec | 0 thr | 0.008 MB/s nmr_meta_binder.cpython-3 0.006 MB | 0.232 sec | 0 thr | 0.027 MB/s nmr_parser.cpython-310.py 0.001 MB | 0.378 sec | 0 thr | 0.002 MB/s requirements.txt 0.000 MB | 0.483 sec | 0 thr | 0.000 MB/s .gitignore 0.000 MB | 0.254 sec | 0 thr | 0.000 MB/s README.md 0.004 MB | 0.440 sec | 0 thr | 0.010 MB/s parse_utils.py 0.001 MB | 1.770 sec | 0 thr | 0.001 MB/s C- /rug/home/rdms-testers@rug.nl/nmr_meta_parser/.git: C- /rug/home/rdms-testers@rug.nl/nmr_meta_parser/.git/objects: C- /rug/home/rdms-testers@rug.nl/nmr_meta_parser/.git/objects/5c: d67aca348a0e4758231637733 0.000 MB | 0.522 sec | 0 thr | 0.001 MB/s C- /rug/home/rdms-testers@rug.nl/nmr_meta_parser/.git/objects/a6: 244a57cf70a69c6731243888c 0.000 MB | 0.523 sec | 0 thr | 0.001 MB/s C- /rug/home/rdms-testers@rug.nl/nmr_meta_parser/.git/objects/c4: 69d8c5b6502b25ea021967a40 0.000 MB | 0.623 sec | 0 thr | 0.000 MB/s C- /rug/home/rdms-testers@rug.nl/nmr_meta_parser/.git/objects/01: caaf0720045c0265dd835cca0 0.001 MB | 0.809 sec | 0 thr | 0.002 MB/s C- /rug/home/rdms-testers@rug.nl/nmr_meta_parser/.git/objects/8a: df0d2bfd9d1f3da44afb66775 0.000 MB | 0.836 sec | 0 thr | 0.000 MB/s C- /rug/home/rdms-testers@rug.nl/nmr_meta_parser/.git/objects/17: 65841eab7cea2807e8be2915d 0.001 MB | 0.403 sec | 0 thr | 0.001 MB/s C- /rug/home/rdms-testers@rug.nl/nmr_meta_parser/.git/objects/4c: 061cd396da8e40b03325fa520 0.000 MB | 0.364 sec | 0 thr | 0.000 MB/s C- /rug/home/rdms-testers@rug.nl/nmr_meta_parser/.git/objects/info: C- /rug/home/rdms-testers@rug.nl/nmr_meta_parser/.git/objects/d7: 337483d3a05065e3fffe4cf53 0.000 MB | 0.284 sec | 0 thr | 0.001 MB/s C- /rug/home/rdms-testers@rug.nl/nmr_meta_parser/.git/objects/9f: 759a7d969b2e19f619cbc7c14 0.000 MB | 0.367 sec | 0 thr | 0.000 MB/s C- /rug/home/rdms-testers@rug.nl/nmr_meta_parser/.git/objects/80: f8218ef959908ebc47b02a24d 0.001 MB | 0.257 sec | 0 thr | 0.005 MB/s C- /rug/home/rdms-testers@rug.nl/nmr_meta_parser/.git/objects/97: bd6ee70910937174233828393 0.001 MB | 0.272 sec | 0 thr | 0.002 MB/s C- /rug/home/rdms-testers@rug.nl/nmr_meta_parser/.git/objects/15: 43b4fcb9a03324daca1a5d87d 0.000 MB | 0.273 sec | 0 thr | 0.001 MB/s C- /rug/home/rdms-testers@rug.nl/nmr_meta_parser/.git/objects/4b: 83c7605922dd90e79cfc1e145 0.000 MB | 0.334 sec | 0 thr | 0.001 MB/s C- /rug/home/rdms-testers@rug.nl/nmr_meta_parser/.git/objects/pack: pack-08f6731e945a0424f737 0.079 MB | 0.313 sec | 0 thr | 0.251 MB/s pack-08f6731e945a0424f737 0.011 MB | 0.249 sec | 0 thr | 0.042 MB/s C- /rug/home/rdms-testers@rug.nl/nmr_meta_parser/.git/branches: config 0.000 MB | 0.209 sec | 0 thr | 0.001 MB/s FETCH_HEAD 0.000 MB | 0.202 sec | 0 thr | 0.000 MB/s C- /rug/home/rdms-testers@rug.nl/nmr_meta_parser/.git/logs: C- /rug/home/rdms-testers@rug.nl/nmr_meta_parser/.git/logs/refs: C- /rug/home/rdms-testers@rug.nl/nmr_meta_parser/.git/logs/refs/remotes: C- /rug/home/rdms-testers@rug.nl/nmr_meta_parser/.git/logs/refs/remotes/origin: develop 0.000 MB | 0.446 sec | 0 thr | 0.000 MB/s main 0.000 MB | 0.287 sec | 0 thr | 0.001 MB/s HEAD 0.000 MB | 0.268 sec | 0 thr | 0.001 MB/s C- /rug/home/rdms-testers@rug.nl/nmr_meta_parser/.git/logs/refs/heads: main 0.000 MB | 0.477 sec | 0 thr | 0.001 MB/s HEAD 0.000 MB | 0.302 sec | 0 thr | 0.001 MB/s description 0.000 MB | 0.209 sec | 0 thr | 0.000 MB/s C- /rug/home/rdms-testers@rug.nl/nmr_meta_parser/.git/refs: C- /rug/home/rdms-testers@rug.nl/nmr_meta_parser/.git/refs/remotes: C- /rug/home/rdms-testers@rug.nl/nmr_meta_parser/.git/refs/remotes/origin: develop 0.000 MB | 0.240 sec | 0 thr | 0.000 MB/s main 0.000 MB | 0.244 sec | 0 thr | 0.000 MB/s HEAD 0.000 MB | 0.248 sec | 0 thr | 0.000 MB/s C- /rug/home/rdms-testers@rug.nl/nmr_meta_parser/.git/refs/tags: C- /rug/home/rdms-testers@rug.nl/nmr_meta_parser/.git/refs/heads: main 0.000 MB | 0.258 sec | 0 thr | 0.000 MB/s packed-refs 0.000 MB | 0.220 sec | 0 thr | 0.001 MB/s C- /rug/home/rdms-testers@rug.nl/nmr_meta_parser/.git/info: exclude 0.000 MB | 0.246 sec | 0 thr | 0.001 MB/s ORIG_HEAD 0.000 MB | 0.225 sec | 0 thr | 0.000 MB/s HEAD 0.000 MB | 0.529 sec | 0 thr | 0.000 MB/s index 0.001 MB | 0.227 sec | 0 thr | 0.006 MB/s C- /rug/home/rdms-testers@rug.nl/nmr_meta_parser/.git/hooks: pre-commit.sample 0.002 MB | 0.232 sec | 0 thr | 0.007 MB/s post-update.sample 0.000 MB | 0.542 sec | 0 thr | 0.000 MB/s push-to-checkout.sample 0.003 MB | 0.257 sec | 0 thr | 0.010 MB/s applypatch-msg.sample 0.000 MB | 0.233 sec | 0 thr | 0.002 MB/s pre-rebase.sample 0.005 MB | 0.246 sec | 0 thr | 0.019 MB/s pre-receive.sample 0.001 MB | 0.248 sec | 0 thr | 0.002 MB/s pre-push.sample 0.001 MB | 0.486 sec | 0 thr | 0.003 MB/s prepare-commit-msg.sample 0.001 MB | 0.261 sec | 0 thr | 0.005 MB/s update.sample 0.003 MB | 0.257 sec | 0 thr | 0.014 MB/s pre-applypatch.sample 0.000 MB | 0.260 sec | 0 thr | 0.002 MB/s pre-merge-commit.sample 0.000 MB | 0.243 sec | 0 thr | 0.002 MB/s fsmonitor-watchman.sample 0.004 MB | 0.238 sec | 0 thr | 0.019 MB/s commit-msg.sample 0.001 MB | 0.231 sec | 0 thr | 0.004 MB/s C- /rug/home/rdms-testers@rug.nl/nmr_meta_parser/utils: __init__.py 0.000 MB | 0.243 sec | 0 thr | 0.000 MB/s utils.py 0.012 MB | 0.217 sec | 0 thr | 0.055 MB/s DummyIrodsConnector.py 0.007 MB | 0.242 sec | 0 thr | 0.029 MB/s C- /rug/home/rdms-testers@rug.nl/nmr_meta_parser/utils/__pycache__: utils.cpython-310.pyc 0.013 MB | 0.539 sec | 0 thr | 0.024 MB/s DummyIrodsConnector.cpyth 0.009 MB | 0.255 sec | 0 thr | 0.034 MB/s IrodsConnector.cpython-31 0.040 MB | 0.292 sec | 0 thr | 0.137 MB/s __init__.cpython-310.pyc 0.000 MB | 0.251 sec | 0 thr | 0.001 MB/s IrodsConnector.py 0.051 MB | 0.261 sec | 0 thr | 0.195 MB/s nmr_meta_cl_parser.py 0.003 MB | 0.232 sec | 0 thr | 0.014 MB/s \\ **(Un)bundling**\\ The ''ibun'' command allows to bundle directories or extract a bundled directories within the RDMS. Without specifying additional flags, tar bundles will be created when bundling a directory, but it is also possible to specify other types (available are: tar, gzip, bzip2 and zip). The additional ''-c'' flag is used to indicate bundling. Additionally, you have to also specify which RDMS directory you want to bundle as well as a name for the created bundle # Bundle directory to a tar archive $ ibun -c bundled_folder.tar /rug/home/rdms-testers@rug.nl/some_other_folder # Bundle directory to a gzip archive $ ibun -cDgzip bundled_folder.gz /rug/home/rdms-testers@rug.nl/some_other_folder # Bundle directory to a bzip2 archive $ ibun -cDbzip2 bundled_folder.bz2 /rug/home/rdms-testers@rug.nl/some_other_folder # Bundle directory to a zip archive $ ibun -cDzip bundled_folder.zip /rug/home/rdms-testers@rug.nl/some_other_folder \\ The additional ''-x'' flag is used to indicate extraction. Additionally, you have to also specify which bundle you want to extract as well as specify a RDMS directory to which the content of the bundle is extracted. # Extract tar archive $ ibun -x bundled_folder.tar unbundled_tar # Extract gzip archive $ ibun -x bundled_folder.gz unbundled_gzip # Extract bzip2 archive $ ibun -x bundled_folder.bz2 unbundled_bzip2 # Extract zip archive $ ibun -x bundled_folder.zip unbundled_zip \\ **Notes:** * The bundling and extraction operations do not display a progress. For large folders/archives, it might take a longer time to finish while your terminal seems to be stuck. * If you want to overwrite, you can again use the additional ''-f'' flag. **Assigning Data Permission**\\ For the assignment of access permissions, the ''ichmod'' command is used. Currently, it is possible to assign the following four permission levels: 'Null', 'Read', 'Write', 'Own'. The general syntax to assign permission is ''ichmod [-r] '' where '''' is one of the mentioned four permission levels, '''' the name of the user or group that should get the new permissions, and '''' is the RDMS path for the files/folders that you want to assign the new permissions. If you plan to assign the permission recursively on a folders, meaning also to assign the new permissions for all its subfolders and contained files, you can run the command with the additional ''-r'' flag. It is also possible to enable/disable permission inheritance of a folder using the ''ichmod'' command. Please refer to the [[rdms:data:permissions|Wiki section about permissions and inheritance]] to get more information about the different permission levels as well as the meaning of inheritance. # Assign new permissions (here: read) for a user $ ichmod write rdms-testers@rug.nl /rug/home/rdms-testers@rug.nl/example_file # Assign new permissions (here: own) for a group on a folder and it's content (see the added '-r' flag!) $ ichmod own rdms-testers@rug.nl /rug/home/rdms-testers@rug.nl/some_example_folder # Revoking permissions = Assign 'null' permission $ ichmod null rdms-testers@rug.nl /rug/home/rdms-testers@rug.nl/example_file # Activate Inheritance on a folder $ ichmod inherit /rug/home/rdms-testers@rug.nl/some_example_folder # Deactivate Inheritance on a folder $ ichmod noninherit /rug/home/rdms-testers@rug.nl/some_example_folder \\ **Notes:** If you are unsure about the currently set permissions on a file/folder or you are unsure if inheritance is off/on, run the ''ils -A '' to display these information. $ ils -A /rug/home/rdms-testers@rug.nl/some_example_folder /rug/home/rdms-testers@rug.nl/some_example_folder: ACL - rdms-testers@rug.nl#rug:own Inheritance - Disabled test.txt ACL - rdms-testers@rug.nl#rug:own # In this example, we can see that the folder in question has inheritance disabled, and just one user has 'own' permission assigned to the main folder as well as the one document contained in it. \\ **Working with Metadata**\\ The ''iCommands'' can be also used to add metadata to a file/folder, list already existing metadata, modify existing metadata, and also query (search) for objects that contain certain metadata. Within the RDMS, metadata is always added as so called 'attribute-value-units' triples, AVUs. The 'attribute' specifies the metadata key (e.g. 'Author') while the 'value' specifies the corresponding value for this key (e.g. 'John Doe'). The 'unit' entry is optional, but can be useful in cases where the entries could be otherwise ambiguous (e.g. A: 'Distance:, V: '100', U: 'km'). This can be left blank if no unit is desired. The command that is used to work with metadata from the CLI, is the ''imeta'' command, which has additional subcommands depending on the desired operation : * List current metadata: ''imeta [-d, -C] '' * Add new metadata: ''imeta add [-d, -C] '' * Modify metadata: ''imeta mod [-d, -C] n: v: u:'' * Query for metadata: ''imeta qu '' # List current metadata on a folder $ imeta ls -C /rug/home/rdms-testers@rug.nl/some_example_folder AVUs defined for collection /rug/home/rdms-testers@rug.nl/some_example_folder: None # Add new metadata (AVU) to a folder $ imeta add -C /rug/home/rdms-testers@rug.nl/some_example_folder Author Wiki_User # Modify metadata of a folder $ imeta mod -C /rug/home/rdms-testers@rug.nl/some_example_folder Author Wiki_User n:Author v:Example_User # Query (search) for metadata $ imeta qu -C Author = Example_User collection: /rug/home/rdms-testers@rug.nl/some_example_folder \\ **Notes:** * The ''imeta'' command requires that you also specify the additional ''-C'' or ''-d'' flag. * Use ''-C'' if you want to work with folder (collection). For example ''imeta ls -C ''. * Use ''-d'' if you want to work with a file (data object). For example ''imeta ls -d ''. * There are additional help pages available for the different ''imeta'' subcommands which can be displayed by executing ''imeta help ''. For example: ''imeta help qu'' to display more information of the query (search) command. **Search for Data**\\ The iCommands provide different options to search for files/folders: - If you know the name of the file/folder, you can simply use the ''ilocate'' command which will display you the full RDMS path if files/folders by this name were found in the system. - If you want to do simple search by using metadata, you can use the ''imeta [-C, -d] qu'' query. With this, you can also combine multiple metadata entries (see example below) for a more granular query/search. - The ''iquest'' command can be used for even more powerful queries/searches within the database. This is a very powerful search that can be (for example) also used to count the amount of files in a certain RDMS location. It uses SQL-like terminology for the queries and also allows to use wildcards for the queries. The help for this command (execute ''iquest -h'' contains a more detailed description about the syntax as well as some examples. # Locate files/folders by name $ ilocate example_file /rug/home/rdms-testers@rug.nl/example_file /rug/home/rdms-testers@rug.nl/some_other_folder/example_file # Query for one or multiple metadata entries $ imeta qu -d Author = Wiki_User collection: /rug/home/rdms-testers@rug.nl dataObj: example_file ---- collection: /rug/home/rdms-testers@rug.nl dataObj: example_file2 $ imeta qu -d Author = Wiki_User and Purpose = Example collection: /rug/home/rdms-testers@rug.nl dataObj: example_file $ imeta qu -d Author = Wiki_User and Purpose = Something_Else collection: /rug/home/rdms-testers@rug.nl dataObj: example_file2 # Use iquest to get the total size of the user's home folder as well as all contained folders $ iquest "SELECT SUM(DATA_SIZE) WHERE COLL_NAME LIKE '/rug/home/rdms-testers@rug.nl%'" DATA_SIZE = 4001838012 \\ === iCommands for Group Management === To manage RDMS Groups, the ''igroupadmin'' command is used. This command can be used to list group members (''igroupadmin lg ''), add/remove users from groups (''igroupadmin ''), or even to create a new group (''igroupadmin mkgroup ''). # Create a new group $ igroupadmin mkgroup an_example_group # List members of group $ igroupadmin lg an_example_group Members of group an_example_group: No rows found # Add member to group $ igroupadmin atg an_example_group rdms-testers@rug.nl $ igroupadmin lg an_example_group Members of group an_example_group: rdms-testers@rug.nl#testrug # Remove member from group $ igroupadmin rfg an_example_group rdms-testers@rug.nl Members of group an_example_group: No rows found \\ ==== iCommands Cheat Sheet ==== The following table contains an overview of the most useful ''iCommands'' with a short description. ^ iCommand ^ Meaning ^ | ''ihelp '' | Print help of '''' or displays all available ''iCommands'' if no command is specified | | ''iinit'' | Initialize connection to the RDMS and also save the password in scrambled format for use with subsequent ''iCommands'' | | ''iexit'' | Close connection to the RDMS and also remove the scrambled password file | | ''ipwd'' | Displays your current RDMS working directory | | ''icd'' | Change the current working directory | | ''ils [-A, -L, -r]'' | Listing files and folders\\ ''-A'': Also display access and inheritance information\\ ''-L'': Displays more information (long format)\\ ''-r'': List recursively (also display content of sub folders) | | ''itree [-c, -f, -s]'' | Display folder structure as a tree view\\ ''-c'': Just display folders (collections), no files\\ ''-f'': Display full path of listed items\\ ''-s'': Also display file sizes in the tree view | | ''imkdir [-p]'' | Create a new folder in the RDMS\\ ''-p'': Also create parent folders if they do not exist | | ''irm [-r, -f]'' | Move files/folders to trash\\ ''-r'': Recursive (delete whole folder structures)\\ ''-f'': Do not move to trash, but delete permanently | | ''irmtrash'' | Empty your trash | | ''imv'' | Move files/folders within the RDMS | | ''icp [-K, -r, -f]'' | Copy files/folders within the RDMS\\ ''-K'': Calculate and verify checksum during copying \\ ''-r'': Recursive (copy whole folder structures)\\ ''-f'': Force, overwrites if files/folders already exist | | ''ilocate'' | Locate files in the RDMS | | ''ichmod [-r]'' | Change access permission and enable/disable permission inheritance\\ ''-r'': Recursive (change for whole folder structures) | | ''imeta add [-d, -C]'' | Add metadata\\ ''-d'': Used for files (data objects)\\ ''-C'': Used for folders (collections) | | ''imeta ls [-d, -C]'' | List metadata\\ ''-d'': Used for files (data objects)\\ ''-C'': Used for folders (collections) | | ''iget [-K, -r, -P, -T, -X]'' | Download data from the RDMS\\ ''-K'': Calculate and verify checksum\\ ''-r'': Download whole folder structure (recursive)\\ ''-P'': Display progress during transfer\\ ''-T'': Renew socket connection (long transfers)\\ ''-X'': Specify restart file (long transfers) | | ''iput [-K, -r, -P, -T, -X]'' | Upload data to the RDMS\\ ''-K'': Calculate and verify checksum\\ ''-r'': Upload whole folder structure (recursive)\\ ''-P'': Display progress during transfer\\ ''-T'': Renew socket connection (long transfers)\\ ''-X'': Specify restart file (long transfers) | | | ''irsync [-K]'' | Synchronize data between RDMS and local | | ''ibun [-x, -c, -D]'' | (Un)bundle data in the RDMS | | ''iuserinfo '' | Display information (name, groups, etc) of a user, displays information about yourself if no user is specified | | ''iquest'' | Execute complex queries using metadata | | ''igroupadmin'' | Manage RDMS Groups (just for group admins) | ==== Further Tips and Tricks ==== This section contains some additional tips and tricks that make working with ''iCommands'' more effective. === Enable Bash Auto-Completion === Most Linux distribution come by default with the so called [[https://en.wikipedia.org/wiki/Bash_(Unix_shell)|Bourne Again Shell (Bash)]] which has an internal auto-completion function via "tab key". This can be used to, for example, auto-complete certain paths when using the standard Linux commands (e.g., ''ls'', ''cd''). This auto-completion does not work by default for the ''iCommands'' which means that for example using ''ils'' with the "tab key" to auto-complete an RDMS path will not work. Luckily, the iRODS community provides a helpful ''irods_completion.bash'' shell script via its [[https://github.com/irods/contrib/tree/main|iRODS contrib Github]] page that can be used to enable this functionality. **Temporarily enable Bash Auto-Completion** As a first step, you need to download the ''irods_completion.bash'' shell script from the mentioned Github repository. The Linux commands ''cURL'' or ''wget'' can be used for this directly from the terminal. In this example, we will assume that the auto-completion script will be downloaded and saved in your Linux home directory (abbreviated as ''~''). To download the shell script to your home location, open a terminal window and then execute: # Using wget $ wget https://raw.githubusercontent.com/irods/contrib/main/irods_completion.bash -P ~/ # Using cURL $ curl https://raw.githubusercontent.com/irods/contrib/main/irods_completion.bash > ~/irods_completion.bash After successfully downloading the script, you can enable it temporarily (for your current shell session) by executing: $ source ~/irods_completion.bash Afterwards, the "tab key" can be used for auto-completion on the RDMS side. If you hit the "tab key" two times in a row, your terminal will display all available paths in the current location that fits your auto-completion criteria. **Permanently enable Bash Auto-Completion** It is also possible to automatically "source" the ''auto_completion.bash'' script every time a new Bash shell is started. To do so, you have to first download the shell script as described above. Afterwards, we will edit the ''~/.bashrc'' file which is automatically loaded when a Bash session is started so that it automatically loads/sources the auto-completion script. While you can use your favorite text editor for this, we will use ''nano'' in this example. In your Linux terminal, execute: $ nano ~/.bashrc Now add ''source ~/irods_completion.bash'' to the end of the opened file, and then save (''nano'': Ctrl + o) the file and exit your text editor (''nano'': Ctrl + x). From now on the auto-completion script will be loaded by default when you start a new shell session. If you also want to enable the auto-completion for your current session, you have to once execute: $ source ~/.bashrc **Notes** * The auto-completion script that is described in this section is specific for the Bash shell. If you are unsure if you are using the correct shell, you can execute ''echo $SHELL'' in your terminal which will show you the default shell in your system. * If you want to disable the automated loading of the auto-completion script, you can edit the ''~/.bashrc'' file again. Either delete the part that contains ''source ~/irods_completion.bash'' or comment it out by adding a "#" in front of the line.