Linux - Data Transfer

Owned by Abigail Smith (Unlicensed)
Last updated: Mar 16, 2022 by TLBrewer
2 min read

This page goes over the methods used to transfer data including scp, sftp, and rsync in Linux. In the following examples, the remote system is always teton.arcc.uwyo.edu. While the local host is always a computer system hosted on the public campus network, i.e. a workstation or some other device.

Note: The use of the following commands when accessing ARCC resources is a one-way inbound connection. This means you must initiate these commands from a system outside of the ARCC resources, i.e. from your local workstation. Attempting to transfer files from an ARCC resource to a system on campus will fail due to security restrictions.

Note: When the amount of data to transfer exceeds around 100 GB, methods like scp, sftp, rsync may be too slow, and Globus will be faster for transferring collections of files due to doing so in parallel.

Contents

https://arccwiki.atlassian.net/wiki/spaces/DOCUMENTAT/pages/64192662

scp

  • Secure copy or SCP is a means of securely transferring computer files between a local host and a remote host or between two remote hosts.

    • It is based on the Secure Shell (SSH) protocol

  • Normally, a client initiates an SSH connection to the remote host and requests an SCP process to be started on the remote server. The remote SCP process can operate in one of two modes:

    • source mode, which reads files (usually from disk) and sends them back to the localhost.

    • sink mode, which accepts the files sent by the client and writes them (usually to disk) on the remote host

Examples

  • Local:
    scp source_directory/hello_world.txt destination_directory

  • Teton:
    scp source_directory/hello_world.txt username@teton.arcc.uwyo.edu:destination_directory

sftp

  • sftp is a command-line interface client program to transfer files using the SSH File Transfer Protocol (SFTP) as implemented by the sftp-server command by the OpenSSH project, which runs inside the encrypted Secure Shell connection.

  • It provides an interactive interface similar to that of traditional FTP clients

  • should not be confused with running an FTP client over an SSH connection.

Examples

SFTP username@teton.uwyo.edu

  • Gets txt file from Teton to Local System:
    get hello_world.txt destination_directory

  • Puts txt file from Local System on to Big Horn:
    put source_directory/hello_world.txt destination_directory

rsync

  • rsync is a utility to keep copies of a file on two computer systems

    • functions as both a file synchronization and file transfer program

  • The rsync algorithm is a type of delta encoding and is used to minimize network usage. Zlib may be used for additional compression, and SSH or stunnel can be used for data security

  • Rsync is typically used to synchronize files and directories between two different systems.

    • For example, if the command rsync local-file user@teton.arcc.uwyo.edu:remote-file is run, rsync will use SSH to connect as the user to remote-host

Examples

  • To sync the contents of dir1 to dir2 on the same system:
    rsync -r dir1/ dir2

  • To sync with a remote system:
    rsync -a ~/dir1 username@teton.arcc.uwyo.edu:destination_directory