Data/File Transfer Tools

Overview

In the world of HPC there is frequent need to copy or move files and folders. This page covers various methods of performing these basic file operations.

SMB/Samba Using a Graphic User Interface (GUI)

In brief, here’s how to connect to file shares from common GUI’s:

Windows: search for or click on the ‘File Explorer App’ (a manila folder held upright by a light blue rack), then in the address bar for the window, type in \\<servername>.arcc.uwyo.edu\<pathto>\<yourproject> and hit enter.

Mac: from the Finder, click ‘Go’ → ‘Connect to Server’. In the ‘Connect to Server’ window that pops up, enter: smb://<servername>.arcc.uwyo.edu/<pathto>/<yourproject> and click ‘Connect’.

Command Line Tools

Here are several command line tools to make managing files easier:

scp (secure copy)

Secure copy or SCP is a means of securely transferring computer files between a local host and a remote host or between two remote hosts. It is based on the Secure Shell (SSH) protocol.

Copy file or folder from a remote host to local host SCP example:

1 $ scp username@from_host:/path_to_file/file.txt /local/folder/
1 $ scp -r username@from_host:/path_to_folder/folder/  /local/folder/

Copy file or folder from local host to a remote host SCP example:

1 $ scp file.txt username@to_host:/remote/folder/
1 $ scp -r /local/folder/ username@to_host:/remote/folder/

Copy file from remote host to remote host SCP example:

1 $ scp username@from_host:/remote/folder/file.txt username@to_host:/remote/folder/

sftp

Secure Shell File Transfer Protocol is a command-line interface client program to securely transfer files using an encrypted Secure Shell connection. SFTP should not be confused with running an FTP client over an SSH connection.

Start the sftp interface: SFTP username@teton.uwyo.edu

Examples

  • Gets txt file from Teton to Local System:
    get hello_world.txt destination_directory

  • Puts txt file from Local System on to Big Horn:
    put source_directory/hello_world.txt destination_directory

rsync

  • rsync is a utility to keep copies of a file on two computer systems

    • functions as both a file synchronization and file transfer program

  • The rsync algorithm is a type of delta encoding and is used to minimize network usage. Zlib may be used for additional compression, and SSH or stunnel can be used for data security

  • Rsync is typically used to synchronize files and directories between two different systems.

    • For example, if the command rsync local-file user@teton.arcc.uwyo.edu:remote-file is run, rsync will use SSH to connect as the user to remote-host

Examples

  • To sync the contents of dir1 to dir2 on the same system:
    rsync -r dir1/ dir2

  • To sync with a remote system:
    rsync -a ~/dir1 username@teton.arcc.uwyo.edu:destination_directory

Globus Online

Globus manages file transfers between two computer systems. It is ideal for large files and available for many institutional clusters and networks.
This document covers the basics of using Globus and provides external links to more detailed information.
If you’re a first time user or simply need a refresher, please refer to Globus’ excellent step by step guide.

Using Globus in Brief

  1. Login to Globus' Web app.

    1. Click “Login”.

    2. To use UWyo’s organizational login, search for ‘University of Wyoming’.

      1. Note that this step may be skipped if Globus still has you cached.

  2. Enter or search for an endpoint or collection in the ‘Collection’ field or find recently used endpoints under ENDPOINTS in the left pane.

    1. The Teton/Alcova collection is ‘ARCC Teton’.

  3. Click ‘Transfer or Sync to…’ in the panel on the right side of the page.

  4. Enter/find a second collection.

  5. Browse to the appropriate source and destination folders in both collections.

  6. Select the files/folders you wish to copy and click ‘Start’.

  7. Click ‘Activity’ in the left pane to observe the transfer progress.

Globus file share instructions

See Globus' instructions on sharing data to learn how to create and share an upload/download repository that you can control access to.

Globus Connect Personal

Globus Connect Personal allows you to share and transfer files to and from your laptop or desktop computer. GCP supports the three primary OS’s. The default is for Mac; click the ‘Show me other supported operating systems’ toggle to download for Linux and Windows.

Note that GCP has a “High Assurance” option for Protected Health Information or Controlled Unclassified Information.

Globus Command Line Interface (CLI)

If you need to use a command line interface, please refer to Globus’ excellent step by step guide.