Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Next revision		 Previous revision
habrok:data_management:transferring_data [2022/02/22 13:04] – external edit 127.0.0.1habrok:data_management:transferring_data [2025/05/26 13:37] (current) – bob
Line 2: Line 2:
 ====== Transferring data ===== ====== Transferring data =====
  
-For most use cases data to be analyzed needs to be transferred to Peregrine and results need to be transferred to other locations. In this page we will describe two basic file transfer methods, followed by a description of a more advanced tool.+For most use cases data to be analyzed needs to be transferred to Hábrók and results need to be transferred to other locations. In this page we will describe two basic file transfer methods, followed by a description of a more advanced tool.
  
 ===== Using MobaXterm ====== ===== Using MobaXterm ======
  
-The MobaXterm SSH client offers a file browser interface to files on the Peregrine cluster. As shown in the following figure. +The MobaXterm SSH client offers a file browser interface to files on the Hábrók cluster. As shown in the following figure.
-{{peregrine:introduction:what_is_a_cluster:mobaxterm_file_browser.png?nolink|}}+
  
-This interface can be used to create directories and for downloading and uploading files. The interface offers a upload and download icon above the file listing. Files can also be dragged and dropped between MobaXterm and the Windows file explorer. The last way also allows a full directory including files to be transferred to or from Peregrine.+{{habrok:additional_information:course_material:mobaxterm_file_browser.png?nolink|}} 
 + 
 +This interface can be used to create directories and for downloading and uploading files. The interface offers a upload and download icon above the file listing. Files can also be dragged and dropped between MobaXterm and the Windows file explorer. The last way also allows a full directory including files to be transferred to or from Hábrók.
  
 ==== Alternative applications ==== ==== Alternative applications ====
  
 There are also alternative file transfer clients available like: There are also alternative file transfer clients available like:
-  * [[https://filezilla-project.org/|Filezilla]]: Windows, OS X, Linux+  * [[https://filezilla-project.org/|Filezilla]]: Windows, OS X, Linux (See the page on [[habrok/data_management/clients/filezilla|FileZilla]] for more information on how to use it).
   * [[https://winscp.net|WinSCP]]: Windows   * [[https://winscp.net|WinSCP]]: Windows
  
Line 21: Line 22:
 ===== Using command line tools ===== ===== Using command line tools =====
  
-Another option is to use command line tools for copying data from or to Peregrine. This option is available in local terminals for Mac OS X or Linux. MobaXterm also offers a terminal on the local system, as does the Linux subsystem for Windows if you were to install that. The two commands that are available are ''scp'' and ''sftp''. Both must be started from the remote system, unless that system can also be accessed through SSH directly. This normally is not the case for your personal desktop or laptop. So in the examples we assume that you work in a terminal session on your local desktop or laptop. If ran from Peregrine source and destination have to be adjusted.+Another option is to use command line tools for copying data from or to Hábrók. This option is available in local terminals for Mac OS X or Linux. MobaXterm also offers a terminal on the local system, as does the Linux subsystem for Windows if you were to install that. The two commands that are available are ''scp'' and ''sftp''. Both must be started from the remote system, unless that system can also be accessed through SSH directly. This normally is not the case for your personal desktop or laptop. So in the examples we assume that you work in a terminal session on your local desktop or laptop. If ran from Hábrók source and destination have to be adjusted.
  
 ==== scp ==== ==== scp ====
Line 29: Line 30:
 scp [options] [user@host1:]source [user@host2:]destination scp [options] [user@host1:]source [user@host2:]destination
 </code> </code>
-Source and destination are mandatory. Either source or destination can be prefixed by a username and hostname for specifying the remote side. Here is an example that will transfer a single file from the local system to a directory, within the home directory of the user in Peregrine. Note that the ''user@host part'' is prefixed by a colon '':'':+Source and destination are mandatory. Either source or destination can be prefixed by a username and hostname for specifying the remote side. Here is an example that will transfer a single file from the local system to a directory, within the home directory of the user in Hábrók. Note that the ''user@host part'' is prefixed by a colon '':'':
 <code> <code>
-alice@skries:~/workdir$ scp vars.yml username@peregrine.hpc.rug.nl:my_project+alice@skries:~/workdir$ scp vars.yml username@login2.hb.hpc.rug.nl:my_project
 vars.yml                                      100% 3495   198.4KB/s   00:00     vars.yml                                      100% 3495   198.4KB/s   00:00    
 alice@skries:~/workdir$  alice@skries:~/workdir$ 
 </code> </code>
-In the example above the file vars.yml from the directory workdir on the local desktop or laptop is being sent to the users directory my_project on Peregrine.+In the example above the file vars.yml from the directory workdir on the local desktop or laptop is being sent to the users directory my_project on Hábrók.
 Note that if you don't specify the username, the username will be taken to be your username on the local system. Note that if you don't specify the username, the username will be taken to be your username on the local system.
  
-If the directory would not have existed the file would have been renamed as my_project in the users home directory on Peregrine.+If the directory would not have existed the file would have been renamed as my_project in the users home directory on Hábrók.
  
-In order to copy the file back from Peregrine to the local system, source and destination have to be interchanged and adjusted where necessary. E.g.:+In order to copy the file back from Hábrók to the local system, source and destination have to be interchanged and adjusted where necessary. E.g.:
 <code> <code>
-alice@skries:~/workdir$ scp username@peregrine.hpc.rug.nl:my_project/vars.yml .+alice@skries:~/workdir$ scp username@login2.hb.rug.nl:my_project/vars.yml .
 vars.yml                                      100% 3495   191.7KB/s   00:00     vars.yml                                      100% 3495   191.7KB/s   00:00    
 alice@skries:~/workdir$  alice@skries:~/workdir$ 
Line 48: Line 49:
 In this case we had to make the remote location explicit, because the source has to be specified exactly. The destination is now ''.'', which is the current directory on the local system.  In this case we had to make the remote location explicit, because the source has to be specified exactly. The destination is now ''.'', which is the current directory on the local system. 
  
-In many cases we want to copy a whole directory from or to Peregrine. This can be done by using the option ''-r'', which like as for cp stands for recursive. +In many cases we want to copy a whole directory from or to Hábrók. This can be done by using the option ''-r'', which like as for cp stands for recursive. 
 <code> <code>
-alice@skries:~/workdir$ scp -r username@peregrine.hpc.rug.nl:my_project .+alice@skries:~/workdir$ scp -r username@login2.hb.hpc.rug.nl:my_project .
 main.yml                                      100%  517    26.7KB/s   00:00     main.yml                                      100%  517    26.7KB/s   00:00    
 vars.yml                                      100% 3495   219.4KB/s   00:00     vars.yml                                      100% 3495   219.4KB/s   00:00    
Line 87: Line 88:
   * ''--append'' Append data onto shorter files on the receiving end. The ''--append-verify'' option will check the contents of the file first. These options are useful for resuming transfer of large files after a connection failure.   * ''--append'' Append data onto shorter files on the receiving end. The ''--append-verify'' option will check the contents of the file first. These options are useful for resuming transfer of large files after a connection failure.
  
-Source and destination can be given in the same way as for ''scp''. Here is an example of transferring a directory from Peregrine to our local system:+Source and destination can be given in the same way as for ''scp''. Here is an example of transferring a directory from Hábrók to our local system:
  
 <code> <code>
-john@skries:~/testdir$ rsync -avz username@peregrine.hpc.rug.nl:pydaal/pydaal-tutorials .+john@skries:~/testdir$ rsync -avz username@login2.hb.hpc.rug.nl:pydaal/pydaal-tutorials .
 receiving incremental file list receiving incremental file list
 pydaal-tutorials/ pydaal-tutorials/