====== Metadata Management via CLI ====== The ''iCommands'' allow to execute most metadata tasks also via command-line interface (CLI). This section will explain how to: * Read/List metadata * Create metadata * Modify metadata * Set new metadata value * Remove metadata * Copy metadata * Query metadata While the [[rdms:access:linux:icommands|iCommands Wiki section]] contains the information to get you started using the ''iCommands'' and also lists the commands that can be used for metadata management, the current section will further built on top of this by providing more detailed examples. **Notes:**\\ * If you want to work with [[metadata templates]], you have to use the [[https://research.web.rug.nl/rdmswebapp/rdms.login.page?next=/rdmswebapp/|web portal]] for the creation and filling of the templates. * Within the RDMS, metadata is always added as so called 'attribute-value-units' triples, AVUs. The 'attribute' specifies the metadata key (e.g. 'Author') while the 'value' specifies the corresponding value for this key (e.g. 'John Doe'). The 'unit' entry is optional, but can be useful in cases where the entries could be otherwise ambiguous (e.g. A: 'Distance', V: '100', U: 'km'). This can be left blank if no unit is desired. * When working with the ''imeta'' command, you **have to specify** if you are working on a file (data object) or folder (collection). This can be achieved by specifying the additional ''-d'' (for files) or ''-C'' (for folders) flags when using the command (see below for examples). * For all of the ''imeta'' command, you can use single or double quotes around an entry if it includes blanks, for example ''imeta add -d Author 'Name Surname'''. ===== iCommands for Metadata Management ===== The ''imeta'' command is the relevant sub-command from the ''iCommands'' packages that is used to work with metadata from the CLI. You can execute the following command from the CLI to get an explanation of the ''imeta'' command as well as an overview of its possible usage: $ imeta help $ imeta help Usage: imeta [-vVMhz] [command] -v verbose -V Very verbose -M Admin mode -z Zonename work with the specified Zone -h This help Commands are: add -d|C|R|u Name AttName AttValue [AttUnits] (Add new AVU triple) addw -d Name AttName AttValue [AttUnits] (Add new AVU triple using Wildcards in Name) rm -d|C|R|u Name AttName AttValue [AttUnits] (Remove AVU) rmw -d|C|R|u Name AttName AttValue [AttUnits] (Remove AVU, use Wildcards) rmi -d|C|R|u Name MetadataID (Remove AVU by MetadataID) mod -d|C|R|u Name AttName AttValue [AttUnits] [n:Name] [v:Value] [u:Units] (modify AVU; new name (n:), value(v:), and/or units(u:) set -d|C|R|u Name AttName newValue [newUnits] (Assign a single value) ls -[l]d|C|R|u Name [AttName] (List existing AVUs for item Name) lsw -[l]d|C|R|u Name [AttName] (List existing AVUs, use Wildcards) qu -d|C|R|u AttName Operator AttVal [...] (Query objects with matching AVUs) cp -d|C|R|u -d|C|R|u Name1 Name2 (Copy AVUs from item Name1 to Name2) upper (Toggle between upper case mode for queries (qu)) Metadata attribute-value-units triples (AVUs) consist of an Attribute-Name, Attribute-Value, and an optional Attribute-Units. They can be added via the 'add' command (and in other ways), and then queried to find matching objects. For each command, -d, -C, -R, or -u is used to specify which type of object to work with: dataobjs (iRODS files), collections, resources, or users. A user must be a rodsadmin to set or modify AVUs on resources and users. Fields represented with upper case, such as Name, are entered values. For example, 'Name' is the name of a dataobject, collection, resource, or user. Note: rmw is deprecated. For rmw and lsw, the % and _ wildcard characters (as defined for SQL) can be used for matching attribute values. Note: addw is deprecated. For addw, the % and _ wildcard characters (as defined for SQL) can be used for matching object names. This is currently implemented only for data-objects (-d). A blank execute line invokes the interactive mode, where imeta prompts and executes commands until 'quit' or 'q' is entered. Like other unix utilities, a series of commands can be piped into it: 'cat file1 | imeta' (maintaining one connection for all commands). Single or double quotes can be used to enter items with blanks. Entered usernames are of the form username[#zone]. If #zone is not provided, the zone from your irods_environment.json is assumed. The appropriate zone (local or remote) is determined from the path names or via -z Zonename (for 'qu' and when working with resources). If you are an iRODS administrator, you can include the -M option to run in administrator mode and add, remove, or set metadata on any collection, data object, user, or resource. Try 'help command' for more help on a specific command. 'help qu' will explain additional options on the query. iRODS Version 4.3.0 imeta \\ Furthermore, you can execute ''imeta help '' to get a more detailed help displayed for the respective subcommand of ''imeta''. For example ''imeta help add'' would display information about the ''imeta add'' subcommand which can be used to add metadata from the CLI (see below for examples). $ imeta help add add -d|C|R|u Name AttName AttValue [AttUnits] (Add new AVU triple) Add an AVU to a dataobj (-d), collection(-C), resource(-R), or user(-u) Example: add -d file1 distance 12 miles ==== Read/List Metadata ==== To list the metadata that is currently added to data that resides in the RDMS, the following command is used: $ imeta ls [-d, -C] [] where '''' specifies the relative or absolute path to the data to which the metadata should be added, and the optional ''[]'' can be used to just list metadata that has a certain attribute added to it. Alternatively, it is also possible to use wildcards in the additional ''[]'' entry, in which case the following command is used: $ imeta lsw [-d, -C] [] **Examples**: Let's assume the following example file/folder hierarchy: $ ils /rug/home/rdms-testers@rug.nl: example_file_1 example_file_2 example_file_3 example_file_4 example_file_5 example_file_6 example_file_7 example_file_8 example_file_9 C- /rug/home/rdms-testers@rug.nl/example_folder_1 C- /rug/home/rdms-testers@rug.nl/example_folder_2 In this initial state, the following conditions apply: - ''example_file_8'' has a metadata of the type ''A:Author'' and ''V:Tester_1'' - ''example_file_9'' has a metadata of the type ''A:Author'' and ''V:Tester_2'' - All other files/folders currently do not have any metadata added We can now list the metadata for the different files by executing the ''imeta ls'' command. For example, to check the metadata of one if the example files which currently does not hold any metadata, we would execute: # Example using the relative path $ imeta ls -d example_file_1 AVUs defined for dataObj /rug/home/rdms-testers@rug.nl/example_file_1: None # Example using the absolute path $ imeta ls -d /rug/home/rdms-testers@rug.nl/example_file_2 AVUs defined for dataObj /rug/home/rdms-testers@rug.nl/example_file_2: None Or similarly for one of the example folders: # Example using the relative path $ imeta ls -C /rug/home/rdms-testers@rug.nl/example_folder_1 AVUs defined for collection /rug/home/rdms-testers@rug.nl/example_folder_1: None # Example using the absolute path $ imeta ls -C /rug/home/rdms-testers@rug.nl/example_folder_2 AVUs defined for collection /rug/home/rdms-testers@rug.nl/example_folder_2: None Note the different use of the ''-d'' and ''-C'' flags when files or folders are checked. As expected, there is no metadata shown for any of the files/folders. In this case, the system displays a ''None'' message. If the same is done for the two files that have metadata associated with them, we get the currently available metadata instead of the ''None'' message: $ imeta ls -d example_file_8 AVUs defined for dataObj /rug/home/rdms-testers@rug.nl/example_file_8: attribute: Author value: Tester_1 units: $ imeta ls -d example_file_9 AVUs defined for dataObj /rug/home/rdms-testers@rug.nl/example_file_9: attribute: Author value: Tester_2 units: ==== Adding Metadata ==== To add metadata to a file/folder, the following command is used: $ imeta add [-d, -C] [] where '''' is again the relative or absolute path to the data to which the metadata should be added. **Examples:** We will use the ''imeta add'' command to add a new AVU to the files ''example_file_8'' and ''example_file_9'': # Add metadata with distance information to both files $ imeta add -d example_file_8 Distance 140 km $ imeta add -d example_file_9 Distance 150 km Checking both files again with ''imeta ls'' shows that the new AVU triple was successfully added: # Check if the metadata was successfully added $ imeta ls -d example_file_8 AVUs defined for dataObj /rug/home/rdms-testers@rug.nl/example_file_8: attribute: Author value: Tester_1 units: ---- attribute: Distance value: 140 units: km $ imeta ls -d example_file_9 AVUs defined for dataObj /rug/home/rdms-testers@rug.nl/example_file_9: attribute: Author value: Tester_2 units: ---- attribute: Distance value: 150 units: km Of course, the same command can be also used to add new metadata to a folder instead of a file by using '-C': # Add metadata to a folder using 'imeta add', then check with 'imeta ls -C' to see that it was correctly added. $ imeta add -C example_folder_1 Author Tester_1 $ imeta ls -C example_folder_1 AVUs defined for collection /rug/home/rdms-testers@rug.nl/example_folder_1: attribute: Author value: Tester_1 units: $ imeta add -C example_folder_1 Author Tester_2 $ imeta ls -C example_folder_1 AVUs defined for collection /rug/home/rdms-testers@rug.nl/example_folder_1: attribute: Author value: Tester_1 units: ---- attribute: Author value: Tester_2 units: ==== Modifying Metadata ==== To modify metadata to a file/folder, the following commands are used: $ imeta mod [-d, -C] [] [n:] [v:] [u:] where the values ''n:'', ''v:'' and ''u:'' are used to specify the new metadata entries that should be changed. Again, '''' specifies the relative or absolute path to the data to which the metadata should be added. **Examples**: In our example, we could use ''imeta mod'' to adjust the name of one of the authors that we added to the ''example_folder_1'' like: # Display currently set metadata before 'imeta mod'. $ imeta ls -C example_folder_1 AVUs defined for collection /rug/home/rdms-testers@rug.nl/example_folder_1: attribute: Author value: Tester_1 units: ---- attribute: Author value: Tester_2 units: # Use 'imeta mod' to change the name (value) of the metadata information about the author. $ imeta mod -C example_folder_1 Author Tester_1 v:Tester_3 $ imeta ls -C example_folder_1 AVUs defined for collection /rug/home/rdms-testers@rug.nl/example_folder_1: attribute: Author value: Tester_2 units: ---- attribute: Author value: Tester_3 units: Furthermore, we could also not just change/adjust the value, author name in this case, but change the complete AVU with one command: # Display currently set metadata before 'imeta mod'. $ imeta ls -C example_folder_1 AVUs defined for collection /rug/home/rdms-testers@rug.nl/example_folder_1: attribute: Author value: Tester_2 units: ---- attribute: Author value: Tester_3 units: # Use 'imeta mod' to change the complete metadata information to another AVU (here: from 'Author: Tester_3' to 'Distance: 12 miles'). $ imeta mod -C example_folder_1 Author Tester_3 n:Distance v:12 u:miles $ imeta ls -C example_folder_1 AVUs defined for collection /rug/home/rdms-testers@rug.nl/example_folder_1: attribute: Author value: Tester_2 units: ---- attribute: Distance value: 12 units: miles ==== Set Metadata ==== Alternatively to ''imeta mod'', it is also possible to use ''imeta set'' which sets the value(s) that are currently set for AVU triple to one (!) new value. The general syntax for this sub-command is: $ imeta set [-d, -C] [] where '''' again specifies the relative or absolute path to the data to which the metadata should be set. **Examples**: Assuming that ''example_file_7'' was setup in a way that it has two metadata information with the same key 'Author', but two different values, the ''imeta set'' command can be used to set it to one metadata entry (**note** the use of the quotes around the new entry which contains a blank line): # Display currently set metadata before 'imeta mod'. $ imeta ls -d example_file_7 AVUs defined for dataObj /rug/home/rdms-testers@rug.nl/example_file_7: attribute: Author value: Tester_4 units: ---- attribute: Author value: Tester_5 units: # Use 'imeta set' to set the metadata about the authors to one new value. $ imeta set -d example_file_7 Author 'First Second' $ imeta ls -d example_file_7 AVUs defined for dataObj /rug/home/rdms-testers@rug.nl/example_file_7: attribute: Author value: First Second units: If instead, the data does not contain any metadata when ''imeta set'' is executed, it acts similar to ''imeta add'' and just adds the new AVU to the data: # Use 'imeta set' to set completely new metadata AVU. $ imeta ls -d example_file_6 AVUs defined for dataObj /rug/home/rdms-testers@rug.nl/example_file_6: None $ imeta set -d example_file_6 Author 'First Second' $ imeta ls -d example_file_6 AVUs defined for dataObj /rug/home/rdms-testers@rug.nl/example_file_6: attribute: Author value: First Second units: ==== Removing Metadata ==== To remove metadata to a file/folder, the following commands is used: $ imeta rm [-d, -C] [] where the values '''', '''' and ''[]'' are used to specify the new metadata entries that should be changed. Again, '''' specifies the relative or absolute path to the data to which the metadata should be added. **Examples**: In this case, we could use ''imeta rm'' to for example remove the ''Distance: 12 miles'' metadata information that we just added to ''example_folder_1'': # First let's see which metadata is set for the data in question. $ imeta ls -C example_folder_1 AVUs defined for collection /rug/home/rdms-testers@rug.nl/example_folder_1: attribute: Author value: Tester_2 units: ---- attribute: Distance value: 12 units: miles # Remove a metadata entry using 'imeta rm' and then check if it was removed. $ imeta rm -C example_folder_1 Distance 12 miles $ imeta ls -C example_folder_1 AVUs defined for collection /rug/home/rdms-testers@rug.nl/example_folder_1: attribute: Author value: Tester_2 units: ==== Copy Metadata ==== To copy metadata to a file/folder from another, the following commands is used: $ imeta cp [-d, -C] where '''' specifies the relative or absolute path to the data **from** which the metadata should be copied, and'''' specifies the relative or absolute path to the data **to** which the metadata should be copied. **Note**:\\ This functionality is currently blocked on our system! ==== Query Metadata ==== To query for metadata, the following command is used: $ imeta qu [-d, -C] [...] where '''' and ''Value'' are again the AVUs for which the query should be performed, and '''' is the respective relation between those. Possible operators are: * ''<'': Used to query for metadata where the value associated with an attribute **is smaller** than a certain value. * ''>'': Used to query for metadata where the value associated with an attribute **is bigger** than a certain value. * ''%%<=%%'': Used to query for metadata where the value associated with an attribute **is smaller or equal** than a certain value. * ''>='': Used to query for metadata where the value associated with an attribute **is bigger or equal** than a certain value. * ''='': Used to query for metadata where the value associated with an attribute **is equal** to a certain value. * ''!='': Used to query for metadata where the value associated with an attribute **is not equal** to a certain value. * ''<>'': Same as ''!='' * ''like'': Used to query for metadata where the value associated with an attribute **is like** a certain value and which allows using wildcards for the value. * ''not like'': Used to query for metadata where the value associated with an attribute **is not like** a certain value and which allows using wildcards for the value. **Examples**: In the presented case, we could use the ''imeta qu'' command to query for all folders or files where the ''Author: Tester_2'' using the ''='' operator: $ imeta qu -d Author = Tester_2 collection: /rug/home/rdms-testers@rug.nl dataObj: example_file_9 We could also use the ''like'' operator to extend our query not just to the cases where ''Author: Tester_2'', but also ''Author: Tester_1'' by using using wildcards at the correct places: # A general query with '%' wildcard. Note that for example_file_8, we defined 'Author: Tester_1' while for example_file_9 we defined 'Author: Tester_2'. Both are still found. $ imeta qu -d Author 'like' Tester% collection: /rug/home/rdms-testers@rug.nl dataObj: example_file_8 ---- collection: /rug/home/rdms-testers@rug.nl dataObj: example_file_9 # Difference between '%' and '_' wildcards using the following two example files. $ imeta ls -d example_file_7 AVUs defined for dataObj /rug/home/rdms-testers@rug.nl/example_file_7: attribute: Author value: First Second units: $ imeta ls -d example_file_6 AVUs defined for dataObj /rug/home/rdms-testers@rug.nl/example_file_6: attribute: Author value: First Second units: # Trial using the '%' which replace a whole string or multiple strings. $ imeta qu -d Author 'like' 'First %' collection: /rug/home/rdms-testers@rug.nl dataObj: example_file_6 ---- collection: /rug/home/rdms-testers@rug.nl dataObj: example_file_7 # In contrast, '_' does just replace a single digit. $ imeta qu -d Author 'like' 'First _' No rows found $ imeta qu -d Author 'like' 'First Secon_' collection: /rug/home/rdms-testers@rug.nl dataObj: example_file_6 ---- collection: /rug/home/rdms-testers@rug.nl dataObj: example_file_7 $ imeta qu -d Author 'like' 'First S_cond' collection: /rug/home/rdms-testers@rug.nl dataObj: example_file_6 ---- collection: /rug/home/rdms-testers@rug.nl dataObj: example_file_7 **Notes about using wildcards**: * Two wildcards can be used in the queries:''%'' and ''_''. * While ''%'' will replace any term, ''_'' will just act as wildcard for one letter (see the second query example above that shows the difference). * The placeholder can be used not just at the beginning/end of the term for which is queried, but instead at any place. It is furthermore possible to use numerical comparison operators like ''>'', ''<'' or ''>='' ''%%<=%%'' to just query results where a value is 'bigger', 'smaller', 'bigger or equal' or 'smaller or equal' to the specified query value. # Show results which have 'Distance: 140' as metadata. $ imeta qu -d Distance = 140 collection: /rug/home/rdms-testers@rug.nl dataObj: example_file_8 # Show results which have 'Distance: 150' as metadata. $ imeta qu -d Distance = 150 collection: /rug/home/rdms-testers@rug.nl dataObj: example_file_9 # Show results which have a metadata key with an associated value of less than 150. $ imeta qu -d Distance '<' 150 collection: /rug/home/rdms-testers@rug.nl dataObj: example_file_5 ---- collection: /rug/home/rdms-testers@rug.nl dataObj: example_file_8 # Show results which have a metadata key with an associated value of more or equal to 150. $ imeta qu -d Distance '>=' 150 collection: /rug/home/rdms-testers@rug.nl dataObj: example_file_9 # Show results which have a metadata key with an associated value of more as 150. $ imeta qu -d Distance '>' 150 No rows found **Notes about using numerical comparison**: * The numerical comparison **only works for decimal notation** (e.g. 10 > 9 or 2.1 < 2.4), but it does not work in cases of other notations (e.g. 1e3 > 1e2 or 2e-4 < 1e-3). Also, queries can be used to query/filter for multiple, combined metadata entries # First see which metadata is added to the files. $ imeta ls -d example_file_8 AVUs defined for dataObj /rug/home/rdms-testers@rug.nl/example_file_8: attribute: Author value: Tester_1 units: ---- attribute: Distance value: 140 units: km $ imeta ls -d example_file_9 AVUs defined for dataObj /rug/home/rdms-testers@rug.nl/example_file_9: attribute: Author value: Tester_2 units: ---- attribute: Distance value: 150 units: km # Use a single and combined query to see the difference in the results. $ imeta qu -d Distance '>=' 140 collection: /rug/home/rdms-testers@rug.nl dataObj: example_file_8 ---- collection: /rug/home/rdms-testers@rug.nl dataObj: example_file_9 $ imeta qu -d Distance '>=' 140 and Author = Tester_2 collection: /rug/home/rdms-testers@rug.nl dataObj: example_file_9 ==== Using the Interactive Mode ==== It is also possible to use the ''imeta'' command in an interactive mode. To enter the interactive mode, just type ''imeta'' without any additional subcommand. You can see that you are in the interactive mode of ''imeta'' when the ''imeta>'' is displayed in your terminal. Once in the interactive mod, subsequent commands can be added after each other. If you want to leave the interactive mode again, type ''q'' or ''quit'' and confirm. **Examples:** # First check that the metadata that is currently set for the example file. $ imeta ls -d example_file_5 AVUs defined for dataObj /rug/home/rdms-testers@rug.nl/example_file_5: None # Enter the interactive mod and subsequently execute different imeta tasks before exiting the interactive mode. $imeta imeta>add -d example_file_5 Type 'Test File' imeta>add -d example_file_5 Author "John Doe" imeta>add -d example_file_5 Distance 10e-5 km imeta>quit # Check again to see if all tasks were executed. $ $ imeta ls -d example_file_5 AVUs defined for dataObj /rug/home/rdms-testers@rug.nl/example_file_5: attribute: Author value: John Doe units: ---- attribute: Distance value: 10e-5 units: km ---- attribute: Type value: Test File units: ==== Further Tips ==== This sections contains some further important and helpful tips for the working with metadata from the CLI. === Copying Files and Metadata === When copying a folder/file that contains metadata, the **metadata is not copied** to the copy of the file/folder! # Check metadata for the original $ imeta ls -d example_file_9 AVUs defined for dataObj /rug/home/rdms-testers@rug.nl/example_file_9: attribute: Author value: Tester_2 units: ---- attribute: Distance value: 150 units: km # Make a copy of the file. $ icp example_file_9 example_file_9_copy # Check metadata for the copy. $ imeta ls -d example_file_9_copy AVUs defined for dataObj /rug/home/rdms-testers@rug.nl/example_file_9_copy: None === Advanced Usages === Of course, as with other ''iCommands'', it is possible to include the ''imeta'' sub-command in scripts etc. to execute more advanced/extended tasks. The interactive ''imeta'' mode can be also used to pipe information into it and then automatically execute the desired tasks. For example, one could combine and ''iquest'' query to first select files that fulfill the query condition and then use this information to set metadata on all those files with one terminal statement: # This example will first query for all files in the collection at '' that have a name of the type 'TEST%' (e.g. 'TEST_Sample', 'TEST_Reaction', etc.). # It will then use this query result and use them for the 'set -d / ' statement. # This will then be piped into 'imeta'. As a result the AVUs specified in the 'set' statement will be added to all files that met the query condition. iquest 'set -d %s/%s ' "select COLL_NAME, DATA_NAME where COLL_NAME = '' and DATA_NAME like 'TEST%'" | imeta