Page Comparison

...

Overview

Rclone is a command-line program to manage files on cloud storage. to support file transfers and syncing between a number of storage services. It is a feature rich alternative to cloud vendors' web storage interfaces. Over 40 cloud storage products support rclone including S3 object stores, business & consumer file storage services, and standard transfer protocols. Rclone has powerful cloud equivalents to the unix commands rsync, cp, mv, mount, ls, ncdu, tree, rm, and cat. Rclone's familiar syntax includes shell pipeline support, and --dry-run protection. Storage that can be managed using RClone includes: 

• All UW ARCC Hosted Storage (Beartooth/Teton-Creek, Alcova, Pathfinder)

• Cloud based storage (Google Drive, Dropbox, OneDrive, Sharepoint)

• Your local computer (or any host on which it is installed)  

It is a feature rich alternative to cloud vendors' web storage interfaces. Over 40 cloud storage products support rclone including S3 object stores, business & consumer file storage services, as well as standard transfer protocols. Rclone has powerful cloud equivalents to the unix commands rsync, cp, mv, mount, ls, ncdu, tree, rm, and cat. Rclone's familiar syntax includes shell pipeline support, and --dry-run protection. It can be used at the command line, in scripts or via its API. This page will describe rclone including , and include instructions for using Pathfinder with rclone.

Features

• MD5/SHA-1 hashes checked at all times for file integrity

• Timestamps preserved on files

• Partial syncs supported on a whole file basis

• Copy mode to just copy new/changed files

• Sync (one way) mode to make a directory identical

• Check mode to check for file hash equality

• Can sync to and from network, e.g. two different cloud accounts

• Optional large file chunking (Chunker)

• Optional encryption (Crypt)

• Optional cache (Cache)

• Optional FUSE mount (rclone mount)

• Multi-threaded downloads to local disk

• Can serve local or remote files over HTTP/WebDav/FTP/SFTP/dlna

What does rclone do?

Rclone can help you:

• Backup (and encrypt) files to cloud storage

• Restore (and decrypt) files from cloud storage

• Mirror cloud data to other cloud services or locally

• Migrate data to cloud, or between cloud storage vendors

• Mount multiple, encrypted, cached or diverse cloud storage as a disk

• Analyse and account for data held on cloud storage using lsf, ljson, size, ncdu

• Union file systems together to present multiple local and/or cloud file systems as one

...

How to Use rclone with Pathfinder

Since rclone is intended to be used with cloud technologies, any server that can use cloud protocols can use rclone to transfer data. ARCC’s on-premises cloud-like storage, Pathfinder, uses the S3 protocol that was developed by Amazon AWS. This enables researchers to store and share data on with all of the capabilities of cloud storage without the costs of a third-party vendor.

...

Step 1. Make sure rclone is installed

On a laptop or desktop: Before using Pathfinder with your own workstation with rclone, you will need to make sure you download it and it is installed properly from the rclone website. The rclone website also has much more information on using rclone including several commands and other documentation with installation instructions.

On Beartooth: Use the module spider rclone command to discover which version or versions of rclone are installed on Beartooth. Once you have identified which version, use the module load rclone command to enable rclone for this Beartooth session. Once that is done, you can check to see if the rclone module is available by using the module avail command.

...

Step 2. Setting up the rclone .config file

Before using rclone you must set up a configuration file that details the information about the remote server you want to transfer data to. Since Pathfinder uses the S3 protocol, our following examples will all use S3, but it’s important to know that there are more options available.

There are two different ways of setting up your rclone configuration file:

Create the .conf file manually

At the command prompt, do the following:

...

[pf-mybucket]​
type = s3
provider = 3
env_auth = false
access_key_id = <enter your access key here>
secret_access_key = <enter your secret key here>
endpoint = pathfinder.arcc.uwyo.edu

Using the rclone prompt to create the .conf file

Once rclone is available to use, run the command rclone config to see the options to create the configuration file.

...

It should look similar to the manual configuration that we mentioned earlier.

Step 3. Basic usage commands for rclone

The basic syntax goes as follows rclone <function> <source> <destination endpoint>:<bucket>.

...