While transferring data over a web application such as Open OnDemand, or over a client such as Cyberduck are easy-to-use, they are difficult to automate within a compute job. However, in some use cases, people may want to transfer data, run some computation on that data, transfer it back and so on. These type of tasks can be accomplished using the command line interface (CLI) on MedicineBow. There are many CLI options to use including the previously discussed Globus CLI, scp, SFTP, and rsync which will all work on MedicineBow, but in this module we will only discuss detail rclone because it is ARCC’s recommended command line tools due to it’s ability to work with desktops, HPC, and cloud storage systems as well as it’s ability to be multi-threaded to facilitate faster transfers.

...

SSH password, leave blank to use ssh-agent.
y) Yes type in my own password
g) Generate random password
n) No leave this optional password blank (default)
y/g/n> 
Raw PEM-encoded private key, If specified, will override key_file parameter.
Enter a string value. Press Enter for the default ("").
key_pem> 
Path to PEM-encoded private key file, leave blank or set key-use-agent to use ssh-agent.

Leading `~` will be expanded in the file name as will environment variables such as `${RCLONE_CONFIG_DIR}`.

Enter a string value. Press Enter for the default ("").
key_file> /home/dylan/.ssh/id_rsaecdsa

The next options will be relating to passwords and certificates. For MedicineBow, none of this applies so we keep hitting enter until we get to the cipher where we will enter '1' for false. The final two steps are entering an advanced configuration and then saving before exiting the configuration setup.

...

• copy - to copy files/directories to or from somewhere

• sync - (one way) to make a directory identical

• move - files to cloud storage deleting the local after verification

• check - hashes and for missing/extra files

• mount - your cloud storage as a network disk

...

dylan@fireball:~$ rclone copy transfer.file medbow:/project/arcc

Finally The End

...

Next Steps