Metadata Management via CLI

The iCommands allow to execute most metadata tasks also via command-line interface (CLI). This section will explain how to:

• Read/List metadata
• Create metadata
• Modify metadata
• Set new metadata value
• Remove metadata
• Copy metadata
• Query metadata

While the iCommands Wiki section contains the information to get you started using the iCommands and also lists the commands that can be used for metadata management, the current section will further built on top of this by providing more detailed examples.

Notes:

• If you want to work with metadata templates, you have to use the web portal for the creation and filling of the templates.
• Within the RDMS, metadata is always added as so called 'attribute-value-units' triples, AVUs. The 'attribute' specifies the metadata key (e.g. 'Author') while the 'value' specifies the corresponding value for this key (e.g. 'John Doe'). The 'unit' entry is optional, but can be useful in cases where the entries could be otherwise ambiguous (e.g. A: 'Distance', V: '100', U: 'km'). This can be left blank if no unit is desired.
• When working with the imeta command, you have to specify if you are working on a file (data object) or folder (collection). This can be achieved by specifying the additional -d (for files) or -C (for folders) flags when using the command (see below for examples).
• For all of the imeta command, you can use single or double quotes around an entry if it includes blanks, for example imeta add -d <path> Author 'Name Surname'.

The imeta command is the relevant sub-command from the iCommands packages that is used to work with metadata from the CLI.

You can execute the following command from the CLI to get an explanation of the imeta command as well as an overview of its possible usage:

$ imeta help

$ imeta help
Usage: imeta [-vVMhz] [command]
 -v verbose
 -V Very verbose
 -M Admin mode
 -z Zonename  work with the specified Zone
 -h This help
 
Commands are:
 add  -d|C|R|u Name AttName AttValue [AttUnits] (Add new AVU triple)
 addw -d Name AttName AttValue [AttUnits] (Add new AVU triple
                                           using Wildcards in Name)
 rm   -d|C|R|u Name AttName AttValue [AttUnits] (Remove AVU)
 rmw  -d|C|R|u Name AttName AttValue [AttUnits] (Remove AVU, use Wildcards)
 rmi  -d|C|R|u Name MetadataID (Remove AVU by MetadataID)
 mod  -d|C|R|u Name AttName AttValue [AttUnits] [n:Name] [v:Value] [u:Units]
      (modify AVU; new name (n:), value(v:), and/or units(u:)
 set  -d|C|R|u Name AttName newValue [newUnits] (Assign a single value)
 ls   -[l]d|C|R|u Name [AttName] (List existing AVUs for item Name)
 lsw  -[l]d|C|R|u Name [AttName] (List existing AVUs, use Wildcards)
 qu   -d|C|R|u AttName Operator AttVal [...] (Query objects with matching AVUs)
 cp   -d|C|R|u -d|C|R|u Name1 Name2 (Copy AVUs from item Name1 to Name2)
 upper (Toggle between upper case mode for queries (qu))
 
Metadata attribute-value-units triples (AVUs) consist of an Attribute-Name,
Attribute-Value, and an optional Attribute-Units.  They can be added
via the 'add' command (and in other ways), and
then queried to find matching objects.
 
For each command, -d, -C, -R, or -u is used to specify which type of
object to work with: dataobjs (iRODS files), collections, resources,
or users.
 
A user must be a rodsadmin to set or modify AVUs on resources and users.
 
Fields represented with upper case, such as Name, are entered values.  For
example, 'Name' is the name of a dataobject, collection, resource,
or user.
 
Note: rmw is deprecated.
For rmw and lsw, the % and _ wildcard characters (as defined for SQL) can
be used for matching attribute values.
 
Note: addw is deprecated.
For addw, the % and _ wildcard characters (as defined for SQL) can
be used for matching object names.  This is currently implemented only
for data-objects (-d).
 
A blank execute line invokes the interactive mode, where imeta
prompts and executes commands until 'quit' or 'q' is entered.
Like other unix utilities, a series of commands can be piped into it:
'cat file1 | imeta' (maintaining one connection for all commands).
 
Single or double quotes can be used to enter items with blanks.
 
Entered usernames are of the form username[#zone].  If #zone is not
provided, the zone from your irods_environment.json is assumed.
 
The appropriate zone (local or remote) is determined from the path names
or via -z Zonename (for 'qu' and when working with resources).
 
If you are an iRODS administrator, you can include the -M option to run
in administrator mode and add, remove, or set metadata on any collection,
data object, user, or resource.
 
Try 'help command' for more help on a specific command.
'help qu' will explain additional options on the query.

iRODS Version 4.3.0                imeta

Furthermore, you can execute imeta help <subcommand> to get a more detailed help displayed for the respective subcommand of imeta.

For example imeta help add would display information about the imeta add subcommand which can be used to add metadata from the CLI (see below for examples).

$ imeta help add
 add -d|C|R|u Name AttName AttValue [AttUnits]  (Add new AVU triple)
Add an AVU to a dataobj (-d), collection(-C), resource(-R), 
or user(-u)
Example: add -d file1 distance 12 miles

To list the metadata that is currently added to data that resides in the RDMS, the following command is used:

$ imeta ls [-d, -C] <path> [<Attribute>]

where <path> specifies the relative or absolute path to the data to which the metadata should be added, and the optional [<Attribute>] can be used to just list metadata that has a certain attribute added to it.

Alternatively, it is also possible to use wildcards in the additional [<Attribute>] entry, in which case the following command is used:

$ imeta lsw [-d, -C] <path> [<Attribute>]

Examples:

Let's assume the following example file/folder hierarchy:

$ ils 
/rug/home/rdms-testers@rug.nl:
  example_file_1
  example_file_2
  example_file_3
  example_file_4
  example_file_5
  example_file_6
  example_file_7
  example_file_8
  example_file_9
  C- /rug/home/rdms-testers@rug.nl/example_folder_1
  C- /rug/home/rdms-testers@rug.nl/example_folder_2

In this initial state, the following conditions apply:

1. example_file_8 has a metadata of the type A:Author and V:Tester_1
2. example_file_9 has a metadata of the type A:Author and V:Tester_2
3. All other files/folders currently do not have any metadata added

We can now list the metadata for the different files by executing the imeta ls command.

For example, to check the metadata of one if the example files which currently does not hold any metadata, we would execute:

# Example using the relative path
$ imeta ls -d example_file_1
AVUs defined for dataObj /rug/home/rdms-testers@rug.nl/example_file_1:
None

# Example using the absolute path
$ imeta ls -d /rug/home/rdms-testers@rug.nl/example_file_2
AVUs defined for dataObj /rug/home/rdms-testers@rug.nl/example_file_2:
None

Or similarly for one of the example folders:

# Example using the relative path
$ imeta ls -C /rug/home/rdms-testers@rug.nl/example_folder_1
AVUs defined for collection /rug/home/rdms-testers@rug.nl/example_folder_1:
None


# Example using the absolute path
$ imeta ls -C /rug/home/rdms-testers@rug.nl/example_folder_2
AVUs defined for collection /rug/home/rdms-testers@rug.nl/example_folder_2:
None

Note the different use of the -d and -C flags when files or folders are checked.

As expected, there is no metadata shown for any of the files/folders. In this case, the system displays a None message.

If the same is done for the two files that have metadata associated with them, we get the currently available metadata instead of the None message:

$ imeta ls -d example_file_8
AVUs defined for dataObj /rug/home/rdms-testers@rug.nl/example_file_8:
attribute: Author
value: Tester_1
units: 

$ imeta ls -d example_file_9
AVUs defined for dataObj /rug/home/rdms-testers@rug.nl/example_file_9:
attribute: Author
value: Tester_2
units: 

To add metadata to a file/folder, the following command is used:

$ imeta add [-d, -C] <path> <Attribute> <Value> [<Unit>]

where <path> is again the relative or absolute path to the data to which the metadata should be added.

Examples:

We will use the imeta add command to add a new AVU to the files example_file_8 and example_file_9:

# Add metadata with distance information to both files

$ imeta add -d example_file_8 Distance 140 km

$ imeta add -d example_file_9 Distance 150 km

Checking both files again with imeta ls shows that the new AVU triple was successfully added:

# Check if the metadata was successfully added

$ imeta ls -d example_file_8
AVUs defined for dataObj /rug/home/rdms-testers@rug.nl/example_file_8:
attribute: Author
value: Tester_1
units: 
----
attribute: Distance
value: 140
units: km

$ imeta ls -d example_file_9
AVUs defined for dataObj /rug/home/rdms-testers@rug.nl/example_file_9:
attribute: Author
value: Tester_2
units: 
----
attribute: Distance
value: 150
units: km

Of course, the same command can be also used to add new metadata to a folder instead of a file by using '-C':

# Add metadata to a folder using 'imeta add', then check with 'imeta ls -C' to see that it was correctly added. 

$ imeta add -C example_folder_1 Author Tester_1

$ imeta ls -C example_folder_1
AVUs defined for collection /rug/home/rdms-testers@rug.nl/example_folder_1:
attribute: Author
value: Tester_1
units: 

$ imeta add -C example_folder_1 Author Tester_2

$ imeta ls -C example_folder_1
AVUs defined for collection /rug/home/rdms-testers@rug.nl/example_folder_1:
attribute: Author
value: Tester_1
units: 
----
attribute: Author
value: Tester_2
units: 

To modify metadata to a file/folder, the following commands are used:

$ imeta mod [-d, -C] <path> <Attribute> <Value> [<Unit>] [n:<Attribute>] [v:<Value>] [u:<Units>]

where the values n:<Attribute>, v:<Value> and u:<Units> are used to specify the new metadata entries that should be changed. Again, <path> specifies the relative or absolute path to the data to which the metadata should be added.

Examples:

In our example, we could use imeta mod to adjust the name of one of the authors that we added to the example_folder_1 like:

# Display currently set metadata before 'imeta mod'.

$ imeta ls -C example_folder_1
AVUs defined for collection /rug/home/rdms-testers@rug.nl/example_folder_1:
attribute: Author
value: Tester_1
units: 
----
attribute: Author
value: Tester_2
units: 

# Use 'imeta mod' to change the name (value) of the metadata information about the author.

$ imeta mod -C example_folder_1 Author Tester_1 v:Tester_3

$ imeta ls -C example_folder_1
AVUs defined for collection /rug/home/rdms-testers@rug.nl/example_folder_1:
attribute: Author
value: Tester_2
units: 
----
attribute: Author
value: Tester_3
units: 

Furthermore, we could also not just change/adjust the value, author name in this case, but change the complete AVU with one command:

# Display currently set metadata before 'imeta mod'.

$ imeta ls -C example_folder_1
AVUs defined for collection /rug/home/rdms-testers@rug.nl/example_folder_1:
attribute: Author
value: Tester_2
units: 
----
attribute: Author
value: Tester_3
units: 

# Use 'imeta mod' to change the complete metadata information to another AVU (here: from 'Author: Tester_3' to 'Distance: 12 miles').

$ imeta mod -C example_folder_1 Author Tester_3 n:Distance v:12 u:miles

$ imeta ls -C example_folder_1
AVUs defined for collection /rug/home/rdms-testers@rug.nl/example_folder_1:
attribute: Author
value: Tester_2
units: 
----
attribute: Distance
value: 12
units: miles

Alternatively to imeta mod, it is also possible to use imeta set which sets the value(s) that are currently set for AVU triple to one (!) new value. The general syntax for this sub-command is:

$ imeta set [-d, -C] <path> <Attribute> <new Value> [<new Unit>]

where <path> again specifies the relative or absolute path to the data to which the metadata should be set.

Examples:

Assuming that example_file_7 was setup in a way that it has two metadata information with the same key 'Author', but two different values, the imeta set command can be used to set it to one metadata entry (note the use of the quotes around the new entry which contains a blank line):

# Display currently set metadata before 'imeta mod'.

$ imeta ls -d example_file_7
AVUs defined for dataObj /rug/home/rdms-testers@rug.nl/example_file_7:
attribute: Author
value: Tester_4
units: 
----
attribute: Author
value: Tester_5
units: 

# Use 'imeta set' to set the metadata about the authors to one new value.

$ imeta set -d example_file_7 Author 'First Second'

$ imeta ls -d example_file_7
AVUs defined for dataObj /rug/home/rdms-testers@rug.nl/example_file_7:
attribute: Author
value: First Second
units:

If instead, the data does not contain any metadata when imeta set is executed, it acts similar to imeta add and just adds the new AVU to the data:

# Use 'imeta set' to set completely new metadata AVU. 

$ imeta ls -d example_file_6
AVUs defined for dataObj /rug/home/rdms-testers@rug.nl/example_file_6:
None

$ imeta set -d example_file_6 Author 'First Second'

$ imeta ls -d example_file_6
AVUs defined for dataObj /rug/home/rdms-testers@rug.nl/example_file_6:
attribute: Author
value: First Second
units: 

To remove metadata to a file/folder, the following commands is used:

$ imeta rm [-d, -C] <path> <Attribute> <Value> [<Unit>]

where the values <Attribute>, <Value> and [<Unit>] are used to specify the new metadata entries that should be changed. Again, <path> specifies the relative or absolute path to the data to which the metadata should be added.

Examples:

In this case, we could use imeta rm to for example remove the Distance: 12 miles metadata information that we just added to example_folder_1:

# First let's see which metadata is set for the data in question.

$ imeta ls -C example_folder_1
AVUs defined for collection /rug/home/rdms-testers@rug.nl/example_folder_1:
attribute: Author
value: Tester_2
units: 
----
attribute: Distance
value: 12
units: miles

# Remove a metadata entry using 'imeta rm' and then check if it was removed. 

$ imeta rm -C example_folder_1 Distance 12 miles

$ imeta ls -C example_folder_1
AVUs defined for collection /rug/home/rdms-testers@rug.nl/example_folder_1:
attribute: Author
value: Tester_2
units: 

To copy metadata to a file/folder from another, the following commands is used:

$ imeta cp [-d, -C] <path_soruce> <path_dest>

where <path_source> specifies the relative or absolute path to the data from which the metadata should be copied, and<path_source> specifies the relative or absolute path to the data to which the metadata should be copied.

Note:
This functionality is currently blocked on our system!

To query for metadata, the following command is used:

$ imeta qu [-d, -C] <Attribute> <Operator> <Value> [...]

where <Attribute> and Value are again the AVUs for which the query should be performed, and <Operator> is the respective relation between those.

Possible operators are:

• <: Used to query for metadata where the value associated with an attribute is smaller than a certain value.
• >: Used to query for metadata where the value associated with an attribute is bigger than a certain value.
• <=: Used to query for metadata where the value associated with an attribute is smaller or equal than a certain value.
• >=: Used to query for metadata where the value associated with an attribute is bigger or equal than a certain value.
• =: Used to query for metadata where the value associated with an attribute is equal to a certain value.
• !=: Used to query for metadata where the value associated with an attribute is not equal to a certain value.
• <>: Same as !=
• like: Used to query for metadata where the value associated with an attribute is like a certain value and which allows using wildcards for the value.
• not like: Used to query for metadata where the value associated with an attribute is not like a certain value and which allows using wildcards for the value.

Examples:

In the presented case, we could use the imeta qu command to query for all folders or files where the Author: Tester_2 using the = operator:

$ imeta qu -d Author = Tester_2
collection: /rug/home/rdms-testers@rug.nl
dataObj: example_file_9

We could also use the like operator to extend our query not just to the cases where Author: Tester_2, but also Author: Tester_1 by using using wildcards at the correct places:

# A general query with '%' wildcard. Note that for example_file_8, we defined 'Author: Tester_1' while for example_file_9 we defined 'Author: Tester_2'. Both are still found. 
 
$ imeta qu -d Author 'like' Tester%
collection: /rug/home/rdms-testers@rug.nl
dataObj: example_file_8
----
collection: /rug/home/rdms-testers@rug.nl
dataObj: example_file_9


# Difference between '%' and '_' wildcards using the following two example files.

$ imeta ls -d example_file_7
AVUs defined for dataObj /rug/home/rdms-testers@rug.nl/example_file_7:
attribute: Author
value: First Second
units: 

$ imeta ls -d example_file_6
AVUs defined for dataObj /rug/home/rdms-testers@rug.nl/example_file_6:
attribute: Author
value: First Second
units: 

# Trial using the '%' which replace a whole string or multiple strings.

$ imeta qu -d Author 'like' 'First %'
collection: /rug/home/rdms-testers@rug.nl
dataObj: example_file_6
----
collection: /rug/home/rdms-testers@rug.nl
dataObj: example_file_7


# In contrast, '_' does just replace a single digit.

$ imeta qu -d Author 'like' 'First _'
No rows found

$ imeta qu -d Author 'like' 'First Secon_'
collection: /rug/home/rdms-testers@rug.nl
dataObj: example_file_6
----
collection: /rug/home/rdms-testers@rug.nl
dataObj: example_file_7

$ imeta qu -d Author 'like' 'First S_cond'
collection: /rug/home/rdms-testers@rug.nl
dataObj: example_file_6
----
collection: /rug/home/rdms-testers@rug.nl
dataObj: example_file_7

Notes about using wildcards:

• Two wildcards can be used in the queries:% and _.
• While % will replace any term, _ will just act as wildcard for one letter (see the second query example above that shows the difference).
• The placeholder can be used not just at the beginning/end of the term for which is queried, but instead at any place.

It is furthermore possible to use numerical comparison operators like >, < or >= <= to just query results where a value is 'bigger', 'smaller', 'bigger or equal' or 'smaller or equal' to the specified query value.

# Show results which have 'Distance: 140' as metadata.

$ imeta qu -d Distance = 140
collection: /rug/home/rdms-testers@rug.nl
dataObj: example_file_8


# Show results which have 'Distance: 150' as metadata.

$ imeta qu -d Distance = 150
collection: /rug/home/rdms-testers@rug.nl
dataObj: example_file_9


# Show results which have a metadata key with an associated value of less than 150.

$ imeta qu -d Distance '<' 150
collection: /rug/home/rdms-testers@rug.nl
dataObj: example_file_5
----
collection: /rug/home/rdms-testers@rug.nl
dataObj: example_file_8


# Show results which have a metadata key with an associated value of more or equal to 150.

$ imeta qu -d Distance '>=' 150
collection: /rug/home/rdms-testers@rug.nl
dataObj: example_file_9


# Show results which have a metadata key with an associated value of more as 150.

$ imeta qu -d Distance '>' 150
No rows found

Notes about using numerical comparison:

• The numerical comparison only works for decimal notation (e.g. 10 > 9 or 2.1 < 2.4), but it does not work in cases of other notations (e.g. 1e3 > 1e2 or 2e-4 < 1e-3).

Also, queries can be used to query/filter for multiple, combined metadata entries

# First see which metadata is added to the files.

$ imeta ls -d example_file_8
AVUs defined for dataObj /rug/home/rdms-testers@rug.nl/example_file_8:
attribute: Author
value: Tester_1
units: 
----
attribute: Distance
value: 140
units: km


$ imeta ls -d example_file_9
AVUs defined for dataObj /rug/home/rdms-testers@rug.nl/example_file_9:
attribute: Author
value: Tester_2
units: 
----
attribute: Distance
value: 150
units: km


# Use a single and combined query to see the difference in the results. 

$ imeta qu -d Distance '>=' 140
collection: /rug/home/rdms-testers@rug.nl
dataObj: example_file_8
----
collection: /rug/home/rdms-testers@rug.nl
dataObj: example_file_9

$ imeta qu -d Distance '>=' 140 and Author = Tester_2
collection: /rug/home/rdms-testers@rug.nl
dataObj: example_file_9

It is also possible to use the imeta command in an interactive mode. To enter the interactive mode, just type imeta without any additional subcommand.

You can see that you are in the interactive mode of imeta when the imeta> is displayed in your terminal.

Once in the interactive mod, subsequent commands can be added after each other. If you want to leave the interactive mode again, type q or quit and confirm.

Examples:

# First check that the metadata that is currently set for the example file.

$ imeta ls -d example_file_5
AVUs defined for dataObj /rug/home/rdms-testers@rug.nl/example_file_5:
None


# Enter the interactive mod and subsequently execute different imeta tasks before exiting the interactive mode.

$imeta
imeta>add -d example_file_5 Type 'Test File'
imeta>add -d example_file_5 Author "John Doe"
imeta>add -d example_file_5 Distance 10e-5 km
imeta>quit


# Check again to see if all tasks were executed.

$ $ imeta ls -d example_file_5
AVUs defined for dataObj /rug/home/rdms-testers@rug.nl/example_file_5:
attribute: Author
value: John Doe
units: 
----
attribute: Distance
value: 10e-5
units: km
----
attribute: Type
value: Test File
units: 

This sections contains some further important and helpful tips for the working with metadata from the CLI.

When copying a folder/file that contains metadata, the metadata is not copied to the copy of the file/folder!

# Check metadata for the original

$ imeta ls -d example_file_9
AVUs defined for dataObj /rug/home/rdms-testers@rug.nl/example_file_9:
attribute: Author
value: Tester_2
units: 
----
attribute: Distance
value: 150
units: km


# Make a copy of the file.

$ icp  example_file_9 example_file_9_copy


# Check metadata for the copy.

$ imeta ls -d example_file_9_copy
AVUs defined for dataObj /rug/home/rdms-testers@rug.nl/example_file_9_copy:
None

Of course, as with other iCommands, it is possible to include the imeta sub-command in scripts etc. to execute more advanced/extended tasks.

The interactive imeta mode can be also used to pipe information into it and then automatically execute the desired tasks.

For example, one could combine and iquest query to first select files that fulfill the query condition and then use this information to set metadata on all those files with one terminal statement:

# This example will first query for all files in the collection at '<path>' that have a name of the type 'TEST%' (e.g. 'TEST_Sample', 'TEST_Reaction', etc.).
# It will then use this query result and use them for the 'set -d <path>/<filename> <attribute> <value> <unit>' statement.
# This will then be piped into 'imeta'. As a result the AVUs specified in the 'set' statement will be added to all files that met the query condition.

iquest 'set -d %s/%s <attribute> <value> <unit>' "select COLL_NAME, DATA_NAME where COLL_NAME = '<path>' and DATA_NAME like 'TEST%'" | imeta