Data transfer with Google Drive
There is a preinstalled software module on Hábrók named rclone with which you can download and upload files and directories to and from your (RUG) Google Drive.
Please be mindful of your data management and data archival requirements.
Google Drive is a very helpful tool to save and access files but it is not recommended as an appropriate storage solution for research data archival. Consider using tools hosted at the university like the RDMS. For questions regarding data management, you can contact the Digital Competence Center (dcc@rug.nl) or the data steward in your faculty.
Loading the application
Loading rclone is similar to loading other modules in Hábrók. You can do so by running module load rclone/1.66.0. From that point on, you can use the application. However, it still needs to be configured to your own Google Drive, for which there are instructions in the next section.
Setting up Google Drive's API
There are extended instructions in the initial setup page of the official documentation, but here we detail the necessary steps to connect the application to your RUG Google Drive.
Make sure you are logged in with your university account if you wish to access your RUG Google Drive and enable the Google Drive
API on your Google account. You can do so by accessing the
Google Developer console.
Create a new project. To do so, type "Create a project" in the search bar at the top. Create this project in the 'rug.nl' organization to use your University Google Drive. Choose a descriptive name for your project, such as 'gdrive'.
It will take a few seconds for your project to be created and the project dashboard should open automatically. If it doesn't, open it manually.
On the project dashboard, open the side bar menu on the top left and navigate to 'APIs & Services' → 'Enable APIs & services'.
On the 'APIs & Services' page, select '+ Enable APIs and Services' at the top. This will take you to the '
API Library' page. On this page, search for 'Google Drive
API' and enable it.
The previous step should open the 'Google Drive
API' page. On it, click the 'Credentials' tab on the left,
not the 'Create credentials' link at the top of the page.
At top of the page you should see a message telling you that it is necessary to configure a 'Consent Screen'. Do so by selecting 'Configure Consent Screen'. Then give your application a name, such as 'rclone', afterwards in 'Audience' select 'External' , and add your own email to the 'User Support Email' field. Lastly, tick the box that you agree to the 'Google
API Services: User Data Policy' and press Create.
On the side bar on the left-hand side of the screen, select 'Data Access' and on the following page choose 'Add or remove scopes'. On the pop-up that will appear, under 'Manually add scopes' add: https://www.googleapis.com/auth/docs,https://www.googleapis.com/auth/drive,https://www.googleapis.com/auth/drive.metadata.readonly, after which you should see three boxes ticked. Press 'Update', then press 'Save ' on the main 'Data Access' page.
On the side bar on the left, go to 'Audience' and add your email under to the list of 'Test users'.
After adding the scopes, press the 3 horizontal bars at the top left of the screen to open the side bar. Then, navigate to 'APIs & Services → Credentials. On the Credentials select '+ Create Credentials' at the top. Choose 'OAuth client ID'.
On the OAuth Client ID section, choose 'Desktop App' as the application type and name it something like 'gdrive'.
You should now see a Client ID. Make sure you download the resulting JSON file. The Google Drive
API is now complete!
After going through these steps you don't need to use the Remote Desktop to use rclone and can use it from the login and interactive nodes (or in compute jobs).
Loading and configuring the application
We will need a browser for the next section, so it is easier to do so via a Remote Desktop Session through the Hábrók Web Portal. Please log in with your university account at https://portal.hb.hpc.rug.nl.
Then, click on 'Hábrók Remote Destkop' which will open a screen to queue a remote desktop job. Choose the regular partition, 2 cores, 1 hour and 8 GB and press launch. After a few moments you will see your job in the queue, soon after a blue Launch Hábrók Remote Desktop button will appear. Press it to enter the desktop environment.
You will then see a Linux Desktop Environment with a few icons at the top. The small black square will open a terminal. Press it to open the terminal and load rclone by doing:
module load rclone/1.66.0
rclone config
You will see a prompt asking to create a new remote. Do so by responding with n.
Give it a name, such as gdrive.
Choose Google Drive from the list you obtain by responding with 17 to the prompt.
When prompted for your client ID, paste this value, which you can find in the JSON file you downloaded in the previous section. Do the same when prompted for the client secret. Note! To paste text to the environment you need to copy it first to the clipboard. Open the clipboard by clicking the small white arrow on the left of your screen and then pressing the clipboard icon. Any text you paste in the resulting text box can then be copied inside the desktop environment.
Choose 'Full access all files, excluding Application Data Folder' by responding with 1.
When asked for a service account, leave the option blank.
When prompted to use a web browser to authenticate, respond with Y and wait for a firefox window to open.
On the web page you see, log in to your RUG Google Account as usual, and grant the application permission to access your Google Drive.
The configuration is now complete! You can close the browser and refuse to configure a Shared Drive and to access the advanced setup.
Using Rclone
Once the initial setup is complete, transferring files to and from Google Drive is simple. rclone offers many options, which you can find here.
A list of useful basic options for Google Drive, assuming you named your remote as 'gdrive':
# List the directories in your Google Drive
rclone lsd gdrive:
# Copy a file called my_file.txt from your $HOME directory to
# a directory called backup on your Google drive
rclone copy $HOME/my_file.txt gdrive:backup
# Copy the my_file.txt in your Google Drive's directory named backup
# to a directory on Hábrók in $HOME/habrok_backup
rclone copy gdrive:backup/my_file.txt $HOME/habrok_backup/
# Make the directory $HOME/results identical to the folder
# results on your Google Drive.
# !!!!! This will remove and modify files to make directories identical !!!!!
rclone sync $HOME/results gdrive:results