Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision		 Previous revision
habrok:data_management:google_drive [2024/09/27 13:37] – pedrohabrok:data_management:google_drive [2026/09/30 17:30] (current) – pedro
Line 2: Line 2:
 There is a preinstalled software module on Hábrók named [[https://rclone.org/|rclone]] with which you can download and upload files and directories to and from your (RUG) Google Drive. There is a preinstalled software module on Hábrók named [[https://rclone.org/|rclone]] with which you can download and upload files and directories to and from your (RUG) Google Drive.
  
-**Please be mindful of your data management and data archival requirements**. Google Drive is a very helpful tool to save and access files but it is not recommended as an appropriate storage solution for research data archival. Consider using tools hosted at the university like the [[[[rdms:start|RDMS]]. For questions regarding data management, you can contact the [[https://www.rug.nl/digital-competence-centre/contact/contact-the-dcc|Digital Competence Center]] ([[dcc@rug.nl|dcc@rug.nl]]) or the data steward in your faculty.+<WRAP center round important 100%> 
 +**Please be mindful of your data management and data archival requirements**.\\ Google Drive is a very helpful tool to save and access files but it is not recommended as an appropriate storage solution for research data archival. Consider using tools hosted at the university like the [[[[rdms:start|RDMS]]. For questions regarding data management, you can contact the [[https://www.rug.nl/digital-competence-centre/contact/contact-the-dcc|Digital Competence Center]] ([[dcc@rug.nl|dcc@rug.nl]]) or the data steward in your faculty. 
 + 
 +</WRAP>
  
 ===== Loading the application ===== ===== Loading the application =====
Line 17: Line 20:
   - It will take a few seconds for your project to be created and the project dashboard should open automatically. If it doesn't, open it manually.   - It will take a few seconds for your project to be created and the project dashboard should open automatically. If it doesn't, open it manually.
   - On the project dashboard, open the side bar menu on the top left and navigate to 'APIs & Services' -> 'Enable APIs & services'.   - On the project dashboard, open the side bar menu on the top left and navigate to 'APIs & Services' -> 'Enable APIs & services'.
-  - On the 'APIs & Services' page, select Enable 'APIs and Services' at the top. This will take you to the 'API Library' page. On this page, search for 'Google Drive API' and enable it. +  - On the 'APIs & Services' page, select '+ Enable APIs and Services' at the top. This will take you to the 'API Library' page. On this page, search for 'Google Drive API' and enable it. 
-  - The previous step should open the 'Google Drive API' page. On it, click the 'Credentials' tab on the left, no the 'Create credentials' link at the top of the page. +  - The previous step should open the 'Google Drive API' page. On it, click the 'Credentials' tab on the left, **not** the 'Create credentials' link at the top of the page. 
-  - At of the page you should see a message telling you that it is necessary to configure a 'Consent Screen'. Do so by selecting 'Configure Consent Screen'. Then, select 'External' and select 'Create'. Give your application a name, such as 'rclone' and add your own email to the 'User Support Email' and 'Developer Contact Email' fields. Save the form and proceed to adding scopes on the next page. +  - At top of the page you should see a message telling you that it is necessary to configure a 'Consent Screen'. Do so by selecting 'Configure Consent Screen'. Then give your application a name, such as 'rclone', afterwards in 'Audience' select 'External' , and add your own email to the 'User Support Email' field. Lastly, tick the box that you agree to the 'Google API Services: User Data Policy' and press Create. 
-  - Select 'Add or remove scopes' and under 'Manually add scopes' add: ''%%https://www.googleapis.com/auth/docs,https://www.googleapis.com/auth/drive,https://www.googleapis.com/auth/drive.metadata.readonly%%'', after which you should see three boxes ticked. Press 'Update', then press 'Save and continue' on the main Scopes page. +  - On the side bar on the left-hand side of the screen, select 'Data Access' and on the following page choose 'Add or remove scopes'. On the pop-up that will appear, under 'Manually add scopes' add: ''%%https://www.googleapis.com/auth/docs,https://www.googleapis.com/auth/drive,https://www.googleapis.com/auth/drive.metadata.readonly%%'', after which you should see three boxes ticked. Press 'Update', then press 'Save ' on the main 'Data Access' page. 
-  - In the following page, add your own email to the list of 'Test users' and press 'Save and continue'. +  - On the side bar on the left, go to 'Audience' and add your email under to the list of 'Test users'. 
-  - After adding the scopes, go back to the Credentials page by selecting 'Credentials' on the bar on the left and select '+ Create Credentials' at the top. Choose 'OAuth client ID'.+  - After adding the scopes, press the 3 horizontal bars at the top left of the screen to open the side bar. Then, navigate to 'APIs & Services -> Credentials. On the Credentials select '+ Create Credentials' at the top. Choose 'OAuth client ID'.
   - On the OAuth Client ID section, choose 'Desktop App' as the application type and name it something like 'gdrive'.   - On the OAuth Client ID section, choose 'Desktop App' as the application type and name it something like 'gdrive'.
   - You should now see a Client ID. Make sure you download the resulting JSON file. The Google Drive API is now complete!   - You should now see a Client ID. Make sure you download the resulting JSON file. The Google Drive API is now complete!
 +
 +After going through these steps you don't need to use the Remote Desktop to use rclone and can use it from the login and interactive nodes (or in compute jobs).
  
 ===== Loading and configuring the application ===== ===== Loading and configuring the application =====
  
-On your terminal in Hábrók, run:+We will need a browser for the next section, so it is easier to do so via a Remote Desktop Session through the Hábrók Web Portal. Please log in with your university account at https://portal.hb.hpc.rug.nl. 
 + 
 +Then, click on 'Hábrók Remote Destkop' which will open a screen to queue a remote desktop job. Choose the regular partition, 2 cores, 1 hour and 8 GB and press launch. After a few moments you will see your job in the queue, soon after a blue Launch Hábrók Remote Desktop button will appear. Press it to enter the desktop environment. 
 + 
 +You will then see a Linux Desktop Environment with a few icons at the top. The small black square will open a terminal. Press it to open the terminal and load ''rclone'' by doing:
  
 <code bash> <code bash>
Line 35: Line 44:
 </code> </code>
  
-  - You will see prompt asking to create a new remote. Do so by responding with ''n''.+  - You will see a prompt asking to create a new remote. Do so by responding with ''n''.
   - Give it a name, such as ''gdrive''.   - Give it a name, such as ''gdrive''.
   - Choose Google Drive from the list you obtain by responding with ''17'' to the prompt.   - Choose Google Drive from the list you obtain by responding with ''17'' to the prompt.
-  - Choose the highest access value, that is, 'Full access all files, excluding Application Data Folder' by responding with ''1''.+  - When prompted for your client ID, paste this value, which you can find in the JSON file you downloaded in the previous section. Do the same when prompted for the client secret. <wrap em>Note! To paste text to the environment you need to copy it first to the clipboard. Open the clipboard by clicking the small white arrow on the left of your screen and then pressing the clipboard icon. Any text you paste in the resulting text box can then be copied inside the desktop environment.</wrap> 
 +  - Choose 'Full access all files, excluding Application Data Folder' by responding with ''1''.
   - When asked for a service account, leave the option blank.   - When asked for a service account, leave the option blank.
   - When prompted to use a web browser to authenticate, respond with ''Y'' and wait for a firefox window to open.   - When prompted to use a web browser to authenticate, respond with ''Y'' and wait for a firefox window to open.
Line 44: Line 54:
   - The configuration is now complete! You can close the browser and refuse to configure a Shared Drive and to access the advanced setup.   - The configuration is now complete! You can close the browser and refuse to configure a Shared Drive and to access the advanced setup.
    
-===== Using Google Drive Upload =====+===== Using Rclone =====
  
 Once the initial setup is complete, transferring files to and from Google Drive is simple. ''rclone'' offers many options, which you can find [[https://rclone.org/docs/#subcommands|here]].  Once the initial setup is complete, transferring files to and from Google Drive is simple. ''rclone'' offers many options, which you can find [[https://rclone.org/docs/#subcommands|here]].