Differences

This shows you the differences between two versions of the page.

--- habrok:data_management:google_drive [2024/08/23 13:24] – removed pedro
+++ habrok:data_management:google_drive [2024/09/27 16:05] (current) – [Using Google Drive Upload] pedro
@@ Line 1: / Line 1: @@
+====== Data transfer with Google Drive ======
+There is a preinstalled software module on Hábrók named [[https://rclone.org/|rclone]] with which you can download and upload files and directories to and from your (RUG) Google Drive.
+**Please be mindful of your data management and data archival requirements**. Google Drive is a very helpful tool to save and access files but it is not recommended as an appropriate storage solution for research data archival. Consider using tools hosted at the university like the [[[[rdms:start|RDMS]]. For questions regarding data management, you can contact the [[https://www.rug.nl/digital-competence-centre/contact/contact-the-dcc|Digital Competence Center]] ([[dcc@rug.nl|dcc@rug.nl]]) or the data steward in your faculty.
+===== Loading the application =====
+Loading ''rclone'' is similar to loading other modules in Hábrók. You can do so by running ''module load rclone/1.66.0''. From that point on, you can use the application. However, it still needs to be configured to your own Google Drive, for which there are instructions in the next section.
+===== Setting up Google Drive's API =====
+There are extended instructions in the initial setup page of the official [[https://rclone.org/drive/#making-your-own-client-id|documentation]], but here we detail the necessary steps to connect the application to your RUG Google Drive.
+  - Make sure you are logged in with your university account if you wish to access your RUG Google Drive and enable the Google Drive API on your Google account. You can do so by accessing the [[https://console.developers.google.com/|Google Developer console]].
+  - Create a new project. To do so, type "Create a project" in the search bar at the top. Create this project in the 'rug.nl' organization to use your University Google Drive. Choose a descriptive name for your project, such as 'gdrive'.
+  - It will take a few seconds for your project to be created and the project dashboard should open automatically. If it doesn't, open it manually.
+  - On the project dashboard, open the side bar menu on the top left and navigate to 'APIs & Services' -> 'Enable APIs & services'.
+  - On the 'APIs & Services' page, select Enable 'APIs and Services' at the top. This will take you to the 'API Library' page. On this page, search for 'Google Drive API' and enable it.
+  - The previous step should open the 'Google Drive API' page. On it, click the 'Credentials' tab on the left, no the 'Create credentials' link at the top of the page.
+  - At of the page you should see a message telling you that it is necessary to configure a 'Consent Screen'. Do so by selecting 'Configure Consent Screen'. Then, select 'External' and select 'Create'. Give your application a name, such as 'rclone' and add your own email to the 'User Support Email' and 'Developer Contact Email' fields. Save the form and proceed to adding scopes on the next page.
+  - Select 'Add or remove scopes' and under 'Manually add scopes' add: ''%%https://www.googleapis.com/auth/docs,https://www.googleapis.com/auth/drive,https://www.googleapis.com/auth/drive.metadata.readonly%%'', after which you should see three boxes ticked. Press 'Update', then press 'Save and continue' on the main Scopes page.
+  - In the following page, add your own email to the list of 'Test users' and press 'Save and continue'.
+  - After adding the scopes, go back to the Credentials page by selecting 'Credentials' on the bar on the left and select '+ Create Credentials' at the top. Choose 'OAuth client ID'.
+  - On the OAuth Client ID section, choose 'Desktop App' as the application type and name it something like 'gdrive'.
+  - You should now see a Client ID. Make sure you download the resulting JSON file. The Google Drive API is now complete!
+===== Loading and configuring the application =====
+On your terminal in Hábrók, run:
+<code bash>
+module load rclone/1.66.0
+rclone config
+</code>
+  - You will see prompt asking to create a new remote. Do so by responding with ''n''.
+  - Give it a name, such as ''gdrive''.
+  - Choose Google Drive from the list you obtain by responding with ''17'' to the prompt.
+  - When prompted for your client ID, paste this value, which you can find in the JSON file you downloaded in the previous section. Do the same when prompted for the client secret.
+  - Choose the highest access value, that is, 'Full access all files, excluding Application Data Folder' by responding with ''1''.
+  - When asked for a service account, leave the option blank.
+  - When prompted to use a web browser to authenticate, respond with ''Y'' and wait for a firefox window to open.
+  - On the web page you see, log in to your RUG Google Account as usual, and grant the application permission to access your Google Drive.
+  - The configuration is now complete! You can close the browser and refuse to configure a Shared Drive and to access the advanced setup.
+===== Using Rclone =====
+Once the initial setup is complete, transferring files to and from Google Drive is simple. ''rclone'' offers many options, which you can find [[https://rclone.org/docs/#subcommands|here]].
+A list of useful basic options for Google Drive, assuming you named your remote as 'gdrive':
+<code bash>
+# List the directories in your Google Drive
+rclone lsd gdrive:
+# Copy a file called my_file.txt from your $HOME directory to
+# a directory called backup on your Google drive
+rclone copy $HOME/my_file.txt gdrive:backup
+# Copy the my_file.txt in your Google Drive's directory named backup
+# to a directory on Hábrók in $HOME/habrok_backup
+rclone copy gdrive:backup/my_file.txt $HOME/habrok_backup/
+# Make the directory $HOME/results identical to the folder
+# results on your Google Drive.
+# !!!!! This will remove and modify files to make directories identical !!!!!
+rclone sync $HOME/results gdrive:results
+</code>