While transferring data over a web application such as Open OnDemand, or over a client such as Cyberduck are easy-to-use, they are difficult to automate within a compute job. However, in some use cases, people may want to transfer data, run some computation on that data, transfer it back and so on. These type of tasks can be accomplished using the command line interface (CLI) on MedicineBow. There are many CLI options to use including the previously discussed Globus CLI, scp, SFTP, and rsync which will all work on MedicineBow, but in this module we will only discuss rclone due to it’s ability to work with desktops, HPC, and cloud storage systems as well as it’s ability to be multi-threaded to facilitate faster transfers.

scp and SFTP CLI Tools and Examples

Before diving into providing information on rclone, we’ll briefly cover some of the other command line tools and give examples for how to use them on MedicineBow.

• scp - Uses SSH (Secure Shell) to authenticate, then securely transfer data. This means the data is authenticated by the user initiating the connection.

  • Example for from local to MedicineBow on Linux/Mac

  • The structure will go ‘scp’ space ‘file’ space ‘username’@'server:'directory to transfer to'

  • dylan@fireball:~$ scp transfer.file dperkin6@medicinebow.arcc.uwyo.edu:/project/arcc
    transfer.file                                                                                                                 100%    0     0.0KB/s   00:00    
    dylan@fireball:~$

• SFTP - can also be used on the CLI as well as clients. Compared to the SCP protocol, which only allows file transfers, the SFTP protocol allows for a wider range of operations on remote files. SFTP clients provide extra capabilities include resuming interrupted transfers, directory listings, and remote file removal.

  • SFTP is generally more platform-independent than SCP.

  • Example of interactive use of SFTP

  • dylan@fireball:~$ sftp dperkin6@medicinebow.arcc.uwyo.edu
    Connected to medicinebow.arcc.uwyo.edu.
    sftp>

  • Helpful SFTP commands

    • ? - is how you access the help, put - to upload a file, mput - to upload multiple files

    • get - to download a file or directory, mget - to download multiple files

rclone config

Please use the "code snippet" in the + button when creating code examples. Also please do not go
past the width of the table. This is to prevent scroll bars appearing













This is the Max number of code lines to show on an example

Straight Code - No context

Limit to 16 lines in the example. 














This is the end

Same Thing With Images

Alternatively No Table

Finally The End