{{indexmenu_n>2}}
====== Archiving Workflow ======
{{ :rdms:workflows:rdms_archiving_workflow_wiki.svg |}}
The Archiving workflow in the RDMS allows the owner of an [[rdms:solution:projects|RDMS Project]] to archive the data contained in the project folder by following a step-by-step process in the web interface.
An archive in the RDMS is a bundled dataset, called a **data package**, that contains both the data and its related metadata, and has been frozen by making it **read-only** in the system. Each archive is automatically labelled with a creation date to indicate when the data was frozen.
Once archived, the dataset (data package) can be pushed to the Publication workflow (currently still in development), which will enable the metadata to be published externally in line with the Open Science framework.
During the archiving process, there are three distinct roles, each of which becomes active at different stages.
**Owner/Admin:**
This role is responsible for assigning the data manager and metadata manager roles as well as starting the archiving process. By default, the creator of the RDMS project is its admin, but the role can also be assigned to other users (see below for info about assigning roles). Best practice is to assign this role to the project supervisor.
**Data Manager:**
This role is responsible for verifying that the data sent to the archive is complete and uncorrupted, and giving the final approval of the archive. Best practice is to assign this role to the person(s) who are most familiar with the data.
**Metadata Manager:**
This role is responsible for verifying and completing the metadata information related to the archive. Best practice is to assign this role to the person(s) who know the origin and scope of the data.
A single user can have any number of these roles assigned to them, and/or multiple users can have the same or different roles and work at different stages of the archiving process. The important part is that each role is assigned to at least one user; otherwise, the workflow cannot be completed.
===== Steps of the Archiving Workflow =====
This section explains the Archiving workflow starting from an existing RDMS example project and walks you through the requirements to start the workflow, the different steps, and the roles active at each step. It will also elaborate more on the content of the created data package.
==== Prerequisites ====
=== Existing Project ===
To start an Archiving workflow, the first prerequisite is that the RDMS Project you want to archive **must already exist**. In addition, the project must **contain data**. Attempting to archive an empty project will result in an error after the first step of the workflow.
=== Using the Web Interface ===
The archiving workflow requires using the [[rdms:webapp:|RDMS web interface]]. It is not possible to execute the workflow via CLI, e.g., iCommands.
=== Correct User Privileges ===
If you want to start an archiving workflow as a project admin, you must have the appropriate elevated permissions to initiate the workflow and assign user roles(Data Manager and Metadata Manager). If you lack these permissions, please contact [[rdms-support@rug.nl|rdms-support@rug.nl]]. The easiest way to check if you have the correct permissions is to check if you can assign roles to users in the project management tab.
For the other roles involved in the workflow, Metadata and Data Manager, no special permissions are required. However, they must have at least read/write permission in the project. Without these permissions, the workflow will not allow them to modify or approve the data or its metadata.
=== Assigning Roles ===
If you have the required permissions, we recommend assigning the appropriate workflow roles for the RDMS Project **before** starting an Archiving workflow. As the project owner, you can assign these roles from the Project Management tab.
{{ :rdms:workflows:rdms_workflow_adding_roles.png?direct&800 |}}
By clicking on the pencil symbol next to the name of an existing project member, their project [[rdms:data:permissions|permissions]] as well as project roles can be adjusted (see below for best practices).
{{ :rdms:workflows:rdms_workflow_adding_roles_2.png?direct&600 |}}
After the roles are assigned, the archiving workflow can either start with the initialization of a new workflow by the project admin or continue from where it left off before the required roles were assigned.
**Notes:**
* To assign a user as Project Admin, select the **own** permission. Please note that the user needs elevated privileges; having **own** permission alone is not sufficient. If this is needed, please contact [[rdms-support@rug.nl|rdms-supprt@rug.nl]].
* To assign a user role, the user needs to have at least 'read' permission in the project.
* The section about [[rdms:workflows:archiving#best_practices|best practices]] provides useful guidance on how these roles could be assigned effectively.
==== Step 1: Initialize a new Archiving Workflow ====
**Active role**: Project Admin\\
**Prerequisites**: Project to be archived exists, Project folder contains data.\\
This first step is the initialization of the archiving workflow. The project admin can start it from two places in the RDMS, either via the [[rdms:webapp:datamanagement|Data Management tab]] or the [[rdms:workflows:start|Workflows tab]].
----
To initialize the workflow via the Data Management tab, open the respective project in the tab, then open the menu in the top-right corner (cogwheel symbol) and select "Archive data", as shown below.
{{ :rdms:workflows:rdms_workflow_init_2.png?direct&600 |}}
In the new pop-up window that will open, select the project data that you wish to archive. When you are done selecting the data you wish to archive, click the arrow at the bottom of the window to move to the next step.
{{ :rdms:workflows:rdms_workflow_init_1.png?direct&600 |}}
You will be redirected to another window that shows an overview of the selected data. The window also allows specifying a version tag for the created archiving workflow. The default tag is of the format ''archive'' with the timestamp being the [[https://en.wikipedia.org/wiki/Unix_time|Unix time]] when the archiving workflow was initialized, but it can be customized by the user.
{{ :rdms:workflows:rdms_workflow_init_4.png?direct&600 |}}
After clicking "Archive data", you have completed this part of the workflow. Look to "Step 2" on this page for the next part.
----
If you want to initialize the workflow directly in the workflows tab, open the tab in the sidebar menu and select "Archiving". Using the cogwheel button in the top-right corner, you can click on the "Archive new data" option to initialize a new archiving workflow. This will open a new pop-up window where you can specify the data that you wish to archive from a selected project. As described above, you will be able to assign a version tag to the data you wish to archive (see screenshots above).
{{ :rdms:workflows:rdms_workflow_init_5.png?direct&600 |}}
----
**Notes:**
* The version tag can't be adjusted by the user afterwards. Please take this into account when adjusting this value.
* If you adjust the archive name, it is still recommended to keep a timestamp ([[rdms:bestpractices|usual naming recommendations]] apply).
* While the project admin can already select data during initialization, more data can be added or removed in the next step. This can be done by the data manager.
**Special considerations**:
When you start the archiving process, you will be prompted to select the folders or files you want to archive. In this step, you can decide if you want to archive the entire project folder or just a part of it. Most of the time, you will select the entire folder. However, there are cases where part of the archive needs to be deleted before the customary retention period (10 years), due to privacy regulations.\\
In such cases, we advise you to create **two archives**: **one containing normal data** that should be stored for 10 years, the **other containing the sensitive data** that needs to be deleted earlier. A good practice would be to label both archives in a way that makes it clear that they are interlinked and which one contains the sensitive data. This is best done in the project folder before the archiving starts. Please contact [[rdms-support@rug.nl|rdms-support@rug.nl]] if your data fits what we just described and you are unsure how to use the archiving workflow in such cases.
==== Step 2: Check Data and copy to Project Archive ====
**Active role**: Data Manager\\
**Prerequisites**: Step 1 has finished, and the project folder is not empty.\\
In this step, the data manager checks if the data sent to be archived is complete and uncorrupted. If the data manager confirms that all is good, the data is copied to a separate folder in the project's archive, located at ''/rug/home/DataArchive/Projects//''.
----
As a data manager, you can do this step via the workflows tab in the web interface, where the available archive drafts are listed in the archiving workflow page. After the project admin initializes the workflow, you can find the newly created archive draft in the first column, labelled "Prepare data". The drafts are organized into cards, and you can identify the respective archive draft by the version tag that was assigned by the project admin in the previous step. At the top of each card, you can find a button with three vertical dots, which you can use to execute different tasks on the selected workflow. See the screenshot below for the location of the button and the options available to you.
{{ :rdms:workflows:rdms_workflow_dataprep_1.png?direct&600 |}}
If you select the "Append data" option, you will be able to add data to the archive. Selecting this option will also open a new window, where you will be guided through adding data. Use this option if the project admin did not add all the data necessary to the archive at the previous step.\\
After all data was added, you can click on the "Prepare data" option to get a view of the currently selected data in a new window. In this window, you can verify that the data that needs to be archived is correct and complete, but you can also remove data again. Additionally, you can select an option that will allow you to add [[rdms:metadata:|RDMS metadata]] to the archive. What we mean here is that you will be adding metadata that was added to files and folders included in the archive, not that you are adding metadata **about** the archive. This will happen in a later step.
{{ :rdms:workflows:rdms_workflow_dataprep_2.png?direct&600 |}}
\\
\\
Finally, once you are ready, click on the "Copy data to archive" option to move the archive draft to the next step. A window will open, where you can verify if the data was sent to Archive once again. If you decide to approve the data in this window, then the archiving workflow will start copying your data from the project space to the project's data archive.
{{ :rdms:workflows:rdms_workflow_dataprep_3.png?direct&600 |}}
During the copying of the data, the archiving workflow is blocked. You can see that the workflow is still busy by checking if the yellow frame around a card is still blinking. After the copying is finished and the yellow frame disappears, you can continue the workflow with the following step, the creation of the data package. If the workflow runs into an error, it will display a red frame around the card once it stops.
{{ :rdms:workflows:rdms_workflow_dataprep_4.png?direct&600 |}}
==== Step 3: Creation of the Data Package ====
**Active role**: Data Manager\\
**Prerequisites**: Step 2 has finished.
In this step, the previously unbundled data that was moved to the project's archive space in the RDMS is bundled into a data package. This data package is a tar file containing the selected data, as well as RDMS file and folder metadata if the option to export it was selected in the previous step.
----
This step should be a short one if the data is in order. It is mostly implemented to make sure that if there is more than one data manager, they can check what the other data managers did. If you did implement our suggestion of having one data manager with final say, then this step is where they get to do a last check. In this step, you have the option to "Preview data", to "Create data package", and to "Prepare data" (see screenshot below).
{{ :rdms:workflows:rdms_workflow_dp_2.png?direct&600 |}}
If you select "Preview data", a window will show you the data contained in the archive draft. At this stage, the data package has not been created yet, and data can still be freely added to the archive. If you wish to add more data, select "Prepare data" to move the card back a step. Add the necessary data to the archive draft, then move the card back to the "Data package" column.
{{ :rdms:workflows:rdms_workflow_dp_1.png?direct&600 |}}
If you select the "Create data package" option in the menu, the RDMS system will automatically bundle the previously unbundled data into a tar archive. Afterwards, the next step (adding metadata to the data package) can follow.
{{ :rdms:workflows:rdms_workflow_dp_4.png?direct&600 |}}
**Note**: As mentioned above, you cannot directly add more data via the workflows page in this step. If you or another data manager sees that wrong data was selected or that data is missing, then you have to select the "Prepare data" option, which will move the workflow back to the previous step. There, you can adjust the content of the archive draft before moving it back to the current step.
==== Step 4: Add/Approve Metadata ====
**Active role**: Metadata Manager and Data Manager\\
**Prerequisites**: Step 3 is finished, (a DOI for the data set exists).
In this step, metadata about the archive can be added to the data package after the data package has been successfully created. Also, **if it already exists**, a [[https://en.wikipedia.org/wiki/Digital_object_identifier|DOI]] of a related publication can be added to the data package.
----
While the RDMS in general allows the user to add metadata with or without a metadata template, the archiving workflow only allows adding metadata via templates. This is done to help standardize the metadata for archived projects and, therefore, make them better findable. Templates can be created by users and also shared with others. If there is no suitable metadata template present, you will therefore have to create one, as described in the [[rdms:metadata:metadatatemplates|Metadata Template section of the wiki]]. Nevertheless, please remember that you are adding metadata about the archive during this step, not about the single files and folders within it. As such, you might not need too much complexity when it comes to the metadata template you want to use.\\
\\
As in previous steps, the three-dot menu holds all the actions you can perform at this stage. They are, in order, "Add DOI", "Add metadata template", "Approve metadata", and "Data package". If you are a data manager, you can move the archive draft back to the previous step. We do not expect you to have to do it, but last-minute changes to a data set could still happen. This is why you still have the option to edit the data.
{{ :rdms:workflows:rdms_workflow_meta_3.png?direct&600 |}}
If you select "Add metadata template", you will see a new window open. At the very top of the window, you can choose which template you want to fill in. Then you can select or type in the different metadata entries the template requires you to add. This step allows for both metadata as well as data managers to add metadata.
{{ :rdms:workflows:rdms_workflow_meta_2.png?direct&600 |}}
If you have already generated a DOI for the dataset, then you can use the "Add DOI" menu button to insert the existing DOI into the metadata of the archive. You can also add a DOI linked to a related publication in this stage of the archiving workflow. The RDMS will check the DOI and will verify its validity. Please note that the correct format of the DOI to be specified is ''prefix/suffix'', **not** URL.
{{ :rdms:workflows:rdms_workflow_meta_1.png?direct&600 |}}
The last option in the menu we have not yet addressed is "Approve metadata". This action is **available only to the metadata manager**. If you or other metadata managers have checked that the metadata has been filled in properly, then you can press the button "Approve metadata" to move the archive draft to the final stage of the archiving workflow. Note that a DOI link was automatically added as a metadata entry if a DOI was specified.
{{ :rdms:workflows:rdms_workflow_meta_4.png?direct&600 |}}
==== Step 5: Move Data Package to Archive ====
**Active role**: Data Manager\\
**Prerequisites**: Step 4 is finished, data and metadata are complete.
In this step, a last confirmation is needed by the data manager to finish the workflow if everything has been set up properly. This is the last step of the workflow and a **point of no return**.
----
At this stage of the archiving workflow, you have two options left as the data manager. You can either push the archive draft back to the add/confirm metadata stage by pressing "Metadata", because some things are still missing, or press the "Archive" button to create the final archive and finish the workflow.
{{ :rdms:workflows:rdms_archiving_appove_1.png?direct&800 |}}
Pressing "Archive" will open a new window one last time, presenting all the info about the archived data package, its final destination, as well as all metadata that will be added to the archive. If everything looks good, you can use the "Archive" button to finalize the workflow.
{{ :rdms:workflows:rdms_archiving_appove_2.png?direct&800 |}}
When the operation is finished, you can find the data package in its final destination, as shown in the screenshot below. The archive will contain all the data added during the workflow, as well as all the metadata. For an explanation of the structure of the archive, please look to the next section.
{{ :rdms:workflows:rdms_archiving_approve_3.png?direct&800 |}}
===== The Data Package and its Content =====
At the end of the archiving workflow, you will have created a data package. In the RDMS, we use this term to identify a data set with a specific structure that resulted from the archiving workflow. In this section, we will have a more detailed look at the data package and explain its internal structure.
In general, the following applies:
* The created data package is always in a structured ''*.tar'' format, which is a standard format for bundling data that can be opened with different tools.
* Inside the tar, there are different subfolders for the selected and archived data, as well as the information about the metadata on files and folders included in the archive, saved in ''*.json'' format. This second folder with the metadata info is only created if you selected to include metadata during step 2 of the archiving workflow. Otherwise, you will only see the folder containing the data.
In our example case, we selected metadata to be included and one folder containing the data. Thus, our archive has the following structure in the end:
# This is the general structure of the created data package after being extracted.
archive1740649820/ # This is the name (version tag) of the archive that we specified during the workflow
├── 2025_2_27_10_51_11_889000000 # Subfolder that contains the selected (meta)data
│ └── Some_project_data # This is the folder from which we started the workflow. Below is its content (not completely
│ └── LA-187-1 # shown)
└── RUGRDMS_METADATA # As we selected in the example to have metadata included, we get this folder as well
└── 1Some_project_data.metadata.json # This is the available metadata for the "Some_project_data" folder in .json format.
If we have a look at the JSON file with the metadata, we see that it contains info about the metadata related to the selected data, not the one related to the archive. The following is a snippet of that file that shows how this info is exported and included in the data package.
[
{
"l_header": "# DO NOT EDIT. Automatically generated for archiving.",
"l_className": "rugirodsrest.RugIRODSRestArchiveMetaToStore",
"l_toplevel_path": "/devrugZone/home/Projects/Example_Project_1/Some_project_data",
"l_objectType": "NORMAL",
"l_objectFullPath": "/devrugZone/home/Projects/Example_Project_1/Some_project_data",
"l_symlink_destination": "",
"l_metaDataList": [
{
"metadataDomain": "COLLECTION",
"domainObjectId": "619037",
"domainObjectUniqueName": "/devrugZone/home/Projects/Example_Project_1/Some_project_data",
"avuId": 620497,
"size": 0,
"createdAt": "Feb 26, 2025 3:28:02 PM",
"modifiedAt": "Feb 26, 2025 4:27:06 PM",
"avuAttribute": "Origin",
"avuValue": "RDMS",
"avuUnit": "",
"count": 1,
"lastResult": true,
"totalRecords": 0
},
{
"metadataDomain": "COLLECTION",
"domainObjectId": "619037",
"domainObjectUniqueName": "/devrugZone/home/Projects/Example_Project_1/Some_project_data",
"avuId": 290732,
"size": 0,
"createdAt": "Feb 26, 2025 3:28:02 PM",
"modifiedAt": "Feb 26, 2025 4:27:06 PM",
"avuAttribute": "Type",
"avuValue": "Testing",
"avuUnit": "",
"count": 2,
"lastResult": true,
"totalRecords": 0
}
]
},
[...]
===== Best Practices =====
This section provides recommendations on how to assign roles within a project to distribute the tasks of the Archiving workflow efficiently among project participants. It also highlights best practices specific to the project archiving process.
In general, roles can be assigned as follows:
* **Project Admin**: This role should be assigned to the project lead. The Project Admin manages the project (permissions, roles, etc.) and is also the only one who can start the workflow. Other than that, this role does not need to take additional steps in the workflow.
* **Data Manager**: The Data Manager verifies that all relevant data has been included in the archive. This role is best assigned to the person most familiar with the project’s data.
* In a simple research project, this might be the main researcher who generated the data.
* In projects where multiple people are familiar with different parts of the data, we recommend assigning the Data Manager role to each of them, so they can verify the integrity of their respective parts.
* Since multiple Data Managers can be assigned, we also recommend agreeing in advance who has the final say in case of disagreements, as only one approval is required to move to the next step.
* The Data Manager may also add metadata in the next stage of the workflow, but cannot approve it.
* **Metadata Manager**: The Metadata Manager is responsible for verifying that the metadata associated with the archive is accurate and complete.
* Ideally, this role should be assigned to someone with knowledge of the data who was not directly involved in earlier steps of the workflow.
* If such a person is not available, a Data Manager may also serve as Metadata Manager.
* If there were multiple Data Managers in the earlier steps, we suggest assigning the Metadata Manager role to one who did not have final authority over the dataset.
* If your project or research group has dedicated data management staff, they are well-suited for this role.
As noted above, multiple roles can be assigned to the same user. If a user is both a data and metadata manager, then the whole workflow, except for the initialization, can be done by that single user. This is also a valid possibility, but we suggest you make use of the "checks and balances" that the archiving workflow introduces by assigning roles to different users, where possible.