Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision		 Previous revision
habrok:data_management:transferring_data [2022/12/20 15:54] – fokkehabrok:data_management:transferring_data [2025/06/20 13:27] (current) – [More advanced: rsync] typo pedro
Line 7: Line 7:
  
 The MobaXterm SSH client offers a file browser interface to files on the Hábrók cluster. As shown in the following figure. The MobaXterm SSH client offers a file browser interface to files on the Hábrók cluster. As shown in the following figure.
-{{habrok:introduction:what_is_a_cluster:mobaxterm_file_browser.png?nolink|}}+ 
 +{{habrok:additional_information:course_material:mobaxterm_file_browser.png?nolink|}}
  
 This interface can be used to create directories and for downloading and uploading files. The interface offers a upload and download icon above the file listing. Files can also be dragged and dropped between MobaXterm and the Windows file explorer. The last way also allows a full directory including files to be transferred to or from Hábrók. This interface can be used to create directories and for downloading and uploading files. The interface offers a upload and download icon above the file listing. Files can also be dragged and dropped between MobaXterm and the Windows file explorer. The last way also allows a full directory including files to be transferred to or from Hábrók.
Line 14: Line 15:
  
 There are also alternative file transfer clients available like: There are also alternative file transfer clients available like:
-  * [[https://filezilla-project.org/|Filezilla]]: Windows, OS X, Linux+  * [[https://filezilla-project.org/|Filezilla]]: Windows, OS X, Linux (See the page on [[habrok/data_management/clients/filezilla|FileZilla]] for more information on how to use it).
   * [[https://winscp.net|WinSCP]]: Windows   * [[https://winscp.net|WinSCP]]: Windows
  
Line 31: Line 32:
 Source and destination are mandatory. Either source or destination can be prefixed by a username and hostname for specifying the remote side. Here is an example that will transfer a single file from the local system to a directory, within the home directory of the user in Hábrók. Note that the ''user@host part'' is prefixed by a colon '':'': Source and destination are mandatory. Either source or destination can be prefixed by a username and hostname for specifying the remote side. Here is an example that will transfer a single file from the local system to a directory, within the home directory of the user in Hábrók. Note that the ''user@host part'' is prefixed by a colon '':'':
 <code> <code>
-alice@skries:~/workdir$ scp vars.yml username@habrok.hpc.rug.nl:my_project+alice@skries:~/workdir$ scp vars.yml username@login2.hb.hpc.rug.nl:my_project
 vars.yml                                      100% 3495   198.4KB/s   00:00     vars.yml                                      100% 3495   198.4KB/s   00:00    
 alice@skries:~/workdir$  alice@skries:~/workdir$ 
Line 42: Line 43:
 In order to copy the file back from Hábrók to the local system, source and destination have to be interchanged and adjusted where necessary. E.g.: In order to copy the file back from Hábrók to the local system, source and destination have to be interchanged and adjusted where necessary. E.g.:
 <code> <code>
-alice@skries:~/workdir$ scp username@habrok.hpc.rug.nl:my_project/vars.yml .+alice@skries:~/workdir$ scp username@login2.hb.rug.nl:my_project/vars.yml .
 vars.yml                                      100% 3495   191.7KB/s   00:00     vars.yml                                      100% 3495   191.7KB/s   00:00    
 alice@skries:~/workdir$  alice@skries:~/workdir$ 
Line 50: Line 51:
 In many cases we want to copy a whole directory from or to Hábrók. This can be done by using the option ''-r'', which like as for cp stands for recursive.  In many cases we want to copy a whole directory from or to Hábrók. This can be done by using the option ''-r'', which like as for cp stands for recursive. 
 <code> <code>
-alice@skries:~/workdir$ scp -r username@habrok.hpc.rug.nl:my_project .+alice@skries:~/workdir$ scp -r username@login2.hb.hpc.rug.nl:my_project .
 main.yml                                      100%  517    26.7KB/s   00:00     main.yml                                      100%  517    26.7KB/s   00:00    
 vars.yml                                      100% 3495   219.4KB/s   00:00     vars.yml                                      100% 3495   219.4KB/s   00:00    
Line 84: Line 85:
   * ''-v'' Verbose. This will show information about the files that are being copied.   * ''-v'' Verbose. This will show information about the files that are being copied.
   * ''-p'' Shows a progress indicator.   * ''-p'' Shows a progress indicator.
-  * ''--delete'' Delete files from the receiving end, which are missing at the sending side. +  * ''%%--%%delete'' Delete files from the receiving end, which are missing at the sending side. 
-  * ''--append'' Append data onto shorter files on the receiving end. The ''--append-verify'' option will check the contents of the file first. These options are useful for resuming transfer of large files after a connection failure.+  * ''%%--%%append'' Append data onto shorter files on the receiving end. The ''%%--%%append-verify'' option will check the contents of the file first. These options are useful for resuming transfer of large files after a connection failure.
  
 Source and destination can be given in the same way as for ''scp''. Here is an example of transferring a directory from Hábrók to our local system: Source and destination can be given in the same way as for ''scp''. Here is an example of transferring a directory from Hábrók to our local system:
  
 <code> <code>
-john@skries:~/testdir$ rsync -avz username@habrok.hpc.rug.nl:pydaal/pydaal-tutorials .+john@skries:~/testdir$ rsync -avz username@login2.hb.hpc.rug.nl:pydaal/pydaal-tutorials .
 receiving incremental file list receiving incremental file list
 pydaal-tutorials/ pydaal-tutorials/
Line 111: Line 112:
 At the end of the output you can see a summary of how much data was transferred and the average speed. At the end of the output you can see a summary of how much data was transferred and the average speed.
  
-**CAVEAT behaviour of rsync changes with final ''/'' at the source location. ** +**CAVEAT behavior of rsync changes with final ''/'' at the source location. ** 
-The behavour of ''rsync'' changes when a ''/'' is appended at the end of the source location. The man page explains this as follows:+The behavior of ''rsync'' changes when a ''/'' is appended at the end of the source location. The man page explains this as follows:
 <code> <code>
        A  trailing slash on the source changes this behavior to avoid creating        A  trailing slash on the source changes this behavior to avoid creating