While transferring data over a web application such as Open OnDemand, or over a client such as Cyberduck are easy-to-use, they are difficult to automate within a compute job. However, in some use cases, people may want to transfer data, run some computation on that data, transfer it back and so on. These type of tasks can be accomplished using the command line interface (CLI) on MedicineBow. There are many CLI options to use including the previously discussed Globus CLI, scp, SFTP, and rsync which will all work on MedicineBow, but in this module we will only discuss rclone because it is ARCC’s recommended command line tools due to it’s ability to work with desktops, HPC, and cloud storage systems as well as it’s ability to be multi-threaded to facilitate faster transfers.

...

• scp - Uses SSH (Secure Shell) to authenticate, then securely transfer data. This means the data is authenticated by the user initiating the connection.

  • Example for from local to MedicineBow on Linux/Mac

  • The structure will go ‘scp’ space ‘file’ space ‘username’@'server:'Basic syntax of a scp command: scp file username@server:directory to transfer to'

  • Code Block
    dylan@fireball:~$ scp transfer.file dperkin6@medicinebow.arcc.uwyo.edu:/project/arcc transfer.file 100% 0 0.0KB/s 00:00 dylan@fireball:~$

• SFTP - can also be used on the CLI as well as clients. Compared to the SCP protocol, which only allows file transfers, the SFTP protocol allows for a wider range of operations on remote files. SFTP clients provide extra capabilities include resuming interrupted transfers, directory listings, and remote file removal.

  • SFTP is generally more platform-independent than SCP.

  • Example of interactive use of SFTP

  • Code Block
    dylan@fireball:~$ sftp dperkin6@medicinebow.arcc.uwyo.edu Connected to medicinebow.arcc.uwyo.edu. sftp>

  • Helpful SFTP commands

    • ?- is how you access the help, put - to upload a file, mput - to upload multiple files

    • get - to download a file or directory, mget - to download multiple files

rclone config

Two Column Tables are nice ways to separate content/ Background info along with a code example on the same “Slide”. Please notice the table width. This should stop scroll bars from appearing

...

Bullets are nice to include for distinct points

...

yep

...

they

...

sure

...

rsync CLI Tool and example

rsync is another very useful tool, that has many options. Rsync (Remote Sync) is a most commonly used command for copying and synchronizing files and directories remotely as well as locally in Linux/Unix systems. With the help of rsync command you can copy and synchronize your data remotely and locally across directories, across disks and networks, perform data backups and mirroring between two Linux machines.

• Basic syntax of rsync command: rsync options source destination

• Code Block
  dylan@fireball:~$ rsync transfer.file dperkin6@medicinebow.arcc.uwyo.edu:/project/arcc/dperkin6

Some common options used with rsync commands

-v : verbose
-r : copies data recursively (but don’t preserve timestamps and permission while transferring data
-a : archive mode, archive mode allows copying files recursively and it also preserves symbolic links, file permissions, user & group ownership and timestamps
-z : compress file data
-h : human-readable, output numbers in a human-readable format

...

Straight Code - No context

...