Differences

This shows you the differences between two versions of the page.

--- habrok:data_management:google_drive [2024/09/24 11:16] – Initial gupload instructions pedro
+++ habrok:data_management:google_drive [2024/09/27 16:05] (current) – [Using Google Drive Upload] pedro
@@ Line 1: / Line 1: @@
 ====== Data transfer with Google Drive ======
+There is a preinstalled software module on Hábrók named [[https://rclone.org/|rclone]] with which you can download and upload files and directories to and from your (RUG) Google Drive.
-Currently, there is no preinstalled application that lets users transfer files to and from Google Drive. It is possible, however, to install a third-party tool in your ''/home'' directory called [[https://labbots.github.io/google-drive-upload/|Google Drive Upload]] to achieve this. With this tool you can download and upload files and directories to your (RUG) Google Drive.
+**Please be mindful of your data management and data archival requirements**. Google Drive is a very helpful tool to save and access files but it is not recommended as an appropriate storage solution for research data archival. Consider using tools hosted at the university like the [[[[rdms:start|RDMS]]. For questions regarding data management, you can contact the [[https://www.rug.nl/digital-competence-centre/contact/contact-the-dcc|Digital Competence Center]] ([[dcc@rug.nl|dcc@rug.nl]]) or the data steward in your faculty.
-Please be mindful of your data management and data archival requirements. Google Drive is a very helpful tool to save and access files but it is not recommended as an appropriate storage solution for research data archival. Consider using tools hosted at the university like the [[[[rdms:start|RDMS]]. For questions regarding data management, you can contact the [[https://www.rug.nl/digital-competence-centre/contact/contact-the-dcc|Digital Competence Center]] ([[dcc@rug.nl|dcc@rug.nl]]) or the data steward in your faculty.
+===== Loading the application =====
-===== Installing the application =====
+Loading ''rclone'' is similar to loading other modules in Hábrók. You can do so by running ''module load rclone/1.66.0''. From that point on, you can use the application. However, it still needs to be configured to your own Google Drive, for which there are instructions in the next section.
-For most use cases, simply following the [[https://labbots.github.io/google-drive-upload/setup/install/|instructions]] in the installation page will be enough. The page contains more information in case you need to customize the installation further, but this is not a requirement.
-In essence, downloading and installing Google Drive Upload can be done by running one line of code on your bash console in your ''$HOME'' directory.
+===== Setting up Google Drive's API =====
-  - Log into Hábrók on any of the access nodes from a terminal, for example ''ssh pXXXXXX@login1.hb.hpc.rug.nl''.
-  - Run ''curl --compressed -Ls https://github.com/labbots/google-drive-upload/raw/master/install.sh | sh -s''
-  - Add a few lines to your ''.bashrc'' file by running the line suggested by the application. It will look something like: ''echo '[ -f "${HOME}/.google-drive-upload/bin/gupload" ] && [ -x "${HOME}/.google-drive-upload/bin" ] && PATH="${HOME}/.google-drive-upload/bin:${PATH}"' >> /homeN/pXXXXXX/.bashrc''. **If you are copying the line from this page, then make sure you replace ''/homeN/pXXXXXX/.bashrc'' with your own p/s- number and ''/home'' directory!**
-You can verify that this worked by restarting your terminal and calling ''gupload --help''. You should see a help message if the installation was successful.
+There are extended instructions in the initial setup page of the official [[https://rclone.org/drive/#making-your-own-client-id|documentation]], but here we detail the necessary steps to connect the application to your RUG Google Drive.
-===== Connecting to Google Drive =====
-There are extended instructions in the initial setup page of the official [[https://labbots.github.io/google-drive-upload/setup/initial_setup/|documentation]], but here we detail the necessary steps to connect the application to your RUG Google Drive.
   - Make sure you are logged in with your university account if you wish to access your RUG Google Drive and enable the Google Drive API on your Google account. You can do so by accessing the [[https://console.developers.google.com/|Google Developer console]].
-  - Create a new project. To do so, type "Create a project" in the search bar at the top. Create this project in the 'rug.nl' organization to use your University Google Drive. Choose a descriptive name for your project, such as 'gupload'.
+  - Create a new project. To do so, type "Create a project" in the search bar at the top. Create this project in the 'rug.nl' organization to use your University Google Drive. Choose a descriptive name for your project, such as 'gdrive'.
   - It will take a few seconds for your project to be created and the project dashboard should open automatically. If it doesn't, open it manually.
   - On the project dashboard, open the side bar menu on the top left and navigate to 'APIs & Services' -> 'Enable APIs & services'.
   - On the 'APIs & Services' page, select Enable 'APIs and Services' at the top. This will take you to the 'API Library' page. On this page, search for 'Google Drive API' and enable it.
-  - The previous step should open the 'Google Drive API' page. On it, click the 'Create credentials' link at the top of the page.
+  - The previous step should open the 'Google Drive API' page. On it, click the 'Credentials' tab on the left, no the 'Create credentials' link at the top of the page.
-  - On the Create credentials menu, select the Google Drive API and indicate that you will be accessing User data. Name the app something descriptive like 'gupload' and add your email in the support email and developer contact fields. You don't need to add any scope to the credential.
+  - At of the page you should see a message telling you that it is necessary to configure a 'Consent Screen'. Do so by selecting 'Configure Consent Screen'. Then, select 'External' and select 'Create'. Give your application a name, such as 'rclone' and add your own email to the 'User Support Email' and 'Developer Contact Email' fields. Save the form and proceed to adding scopes on the next page.
-  - On the OAuth Client ID section, choose 'Desktop App' as the application type and name it something like 'gupload app'.
+  - Select 'Add or remove scopes' and under 'Manually add scopes' add: ''%%https://www.googleapis.com/auth/docs,https://www.googleapis.com/auth/drive,https://www.googleapis.com/auth/drive.metadata.readonly%%'', after which you should see three boxes ticked. Press 'Update', then press 'Save and continue' on the main Scopes page.
-  - You should now see a Client ID. Make sure you download this. The Google Drive API is now complete!
+  - In the following page, add your own email to the list of 'Test users' and press 'Save and continue'.
-  - On your terminal in Hábrók, after also having done the steps in the [[habrok:data_management:google_drive:installing_application|previous section]].
+  - After adding the scopes, go back to the Credentials page by selecting 'Credentials' on the bar on the left and select '+ Create Credentials' at the top. Choose 'OAuth client ID'.
-    - Create a dummy file to upload from your ''$HOME'' directory. For example ''cd $HOME && echo "This is a dummy file!" >> dummy_file.txt''.
+  - On the OAuth Client ID section, choose 'Desktop App' as the application type and name it something like 'gdrive'.
-    - Upload the file by calling ''gupload dummy_file.txt''. You will be prompted for your ''Client ID'' and ''Client SECRET''. Using a regular text editor like notepad open the json file you downloaded earlier and paste the corresponding values.
+  - You should now see a Client ID. Make sure you download the resulting JSON file. The Google Drive API is now complete!
-    - When prompted for a refresh token, leave the field blank and press ''Enter''. You will now see a URL for a web page.
-    - Without closing the current session, start a new SSH connection on the same access node in Hábrók and open a browser window by writing ''firefox''. Paste the URL from the previous step on your browser. This should take you to a Google login page. Login with your university account and grant permissions to ''gupload''.
+===== Loading and configuring the application =====
-    - Go back to your terminal session and press ''Enter''. You will be asked for the root folder ID or URL of the Google Drive folder you want to upload to. Choose one, or press ''Enter'' to leave it to the default (the root directory of your account). The setup is now complete.
-    - Go to your Google drive and confirm the file ''dummy_file.txt'' is present and contains the text "This is a dummy file!".
+On your terminal in Hábrók, run:
+<code bash>
+module load rclone/1.66.0
+rclone config
+</code>
+  - You will see prompt asking to create a new remote. Do so by responding with ''n''.
+  - Give it a name, such as ''gdrive''.
+  - Choose Google Drive from the list you obtain by responding with ''17'' to the prompt.
+  - When prompted for your client ID, paste this value, which you can find in the JSON file you downloaded in the previous section. Do the same when prompted for the client secret.
+  - Choose the highest access value, that is, 'Full access all files, excluding Application Data Folder' by responding with ''1''.
+  - When asked for a service account, leave the option blank.
+  - When prompted to use a web browser to authenticate, respond with ''Y'' and wait for a firefox window to open.
+  - On the web page you see, log in to your RUG Google Account as usual, and grant the application permission to access your Google Drive.
+  - The configuration is now complete! You can close the browser and refuse to configure a Shared Drive and to access the advanced setup.
+===== Using Rclone =====
+Once the initial setup is complete, transferring files to and from Google Drive is simple. ''rclone'' offers many options, which you can find [[https://rclone.org/docs/#subcommands|here]].
+A list of useful basic options for Google Drive, assuming you named your remote as 'gdrive':
+<code bash>
+# List the directories in your Google Drive
+rclone lsd gdrive:
-===== Using Google Drive Upload =====
+# Copy a file called my_file.txt from your $HOME directory to
+# a directory called backup on your Google drive
+rclone copy $HOME/my_file.txt gdrive:backup
-Once the initial setup is complete, transferring files to and from Google Drive is simple as uploading the dummy file in the previous section. To upload a file to your Google Drive, a simple call to ''gupload my_file'' will place the file in the root directory of your Drive by default. You can upload to a different directory as well, by doing something like:
+# Copy the my_file.txt in your Google Drive's directory named backup
+# to a directory on Hábrók in $HOME/habrok_backup
+rclone copy gdrive:backup/my_file.txt $HOME/habrok_backup/
-''gupload filename/foldername/file_id/file_link -c gdrive_folder_name''
+# Make the directory $HOME/results identical to the folder
+# results on your Google Drive.
+# !!!!! This will remove and modify files to make directories identical !!!!!
+rclone sync $HOME/results gdrive:results
+</code>