The biggest change with using the new CLI is that the previous use of ssh has been replaced by a new command called "globus" (which is an application that must be installed).

Contents

https://arccwiki.atlassian.net/wiki/spaces/DOCUMENTAT/pages/64192662

https://arccwiki.atlassian.net/wiki/spaces/DOCUMENTAT/pages/33184

Installation

References

• The CLI application "globus" can be found at [2] (It isn't necessary to download it from here as it will be automatically retrieved during installation.)

• Installation instructions are at [3]

System Requirements and Background

• Python 2.7+ or 3.3+

• "python pip" — a package manager for python (used in a similar way to yum or apt-get)

• "virtualenv" — a python package for creating isolated python environments

• If using a version of Python >= 2.7 and < 2.7.9, then need to do an extra step before the provided instructions, which is to install "python pip". From version 2.7.9 onward, it is included with Python.

Steps

1. Install "python pip" if needed: If "python pip" isn't installed, then do the following (or the equivalent for your OS):
   sudo yum install python-pip

2. Install "virtualenv" if needed: Do "which virtualenv" to check if it's installed and if not, do:
   sudo pip install virtualenv

3. Use virtualenv to create an isolated python environment and install the CLI in it:

   virtualenv "$HOME/.globus-cli-virtualenv" source "$HOME/.globus-cli-virtualenv/bin/activate" pip install globus-cli deactivate

4. Add location of newly installed CLI to path:
   export PATH="$PATH:$HOME/.globus-cli-virtualenv/bin"
   To the end of $HOME/.bashrc (or other initialization file if using a different shell), add the line:
   export PATH="$PATH:$HOME/.globus-cli-virtualenv/bin"

5. Check if the CLI is installed:
   globus --help

Transferring Data

Logging in

To use the CLI to view data, transfer data, or look up any Globus information, start by logging in to Globus:

globus login

This command opens a web browser window/tab which requires a normal Globus login (unless you are already logged in on the Globus website), and then continues to a CLI setup page. Follow the instructions and accept the terms to complete the setup. Once this web interaction completes, the above "globus login" command will also complete with a success message, while maintaining a state of being logged in. (Will later need to logout with a separate Globus command.)

Notes:

• After logging in, although the prompt looks like a regular operating system prompt, it is now possible to give it Globus commands, until logging out with "globus logout".

• There is currently no way to complete the process of logging in to the command-line interface without having to click the "Allow" button on a webpage.

• If the system being used to login to Globus isn't able to detect and/or open a tab in a browser, give the "globus login" command the option --no-local-server. The login command will then display a link which must be copy-and-pasted into a browser, and the code provided by that webpage copy-and-pasted back into the command-line login interaction to complete the process of logging in.

Useful globus commands once logged in

• globus whoami : see login status/user

• globus get-identities 'go@globusid.org' : get ID of Globus Auth server (confirms that it is accessible)

• globus endpoint search 'ARCC' : search for a known endpoint

• globus endpoint activate --myproxy 'xxxxxxxx-yyyy-zzzz-wwww-uuuuuuuuuuuu' : login to endpoint (for endpoint that requires authentication)

• globus ls 'aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee' : see listing of endpoint — if no path is given, shows listing of default path. Can also provide a path (see examples below), in which case the full path must be given (i.e. it is NOT relative to the default path).

• globus endpoint show 'xxxxxxxx-yyyy-zzzz-wwww-uuuuuuuuuuuu' : show information about endpoint configuration and state

• globus logout : logout of Globus CLI

Notes:

• Commands that require access to the contents of an endpoint (e.g. 'ls' and 'transfer') won't work until the endpoint is activated. There are two ways to activate an endpoint:

  • either use the web interface to select the endpoint and login to it,

  • or use the 'globus endpoint activate' command shown above.

• (Workaround to bug) If commands such as 'endpoint search' (or 'ls' after activation) give access errors, then revoke Globus consents — and obtain new ones — by:

  1. Logout of CLI.

  2. Go to https://auth.globus.org/v2/web/consents (and login on website if necessary).

  3. Revoke consents for Globus Web App and Globus CLI by clicking the X on the right and choosing "Rescind Consent".

  4. Login to CLI again and follow prompts to grant new consents.

• To use an endpoint in a command (e.g. "globus ls"), it must be referred to by its ID (not the Display Name or Legacy Name). Since it can be awkward to keep copy-and-pasting the ID many times, the ID can be stored in a variable which can then be used when needed, e.g.:

Examples

Example 1: Transfer a File

To transfer a file from a server endpoint to a personal endpoint (with default options):

• First ensure that Globus Connect is running on the personal endpoint and has a status of "Connected".

globus transfer 'xxxxxxxx-yyyy-zzzz-wwww-uuuuuuuuuuuu:/full/path/to/ansible_notes.odt' 'personal-endpoint-id:/full/path/other/location/ansible_notes.odt'

where each of the source and destination have the form <endpoint_ID>:<PATH>

When the above transfer request is successfully submitted, the command displays the Task ID for the transfer. Take note of this ID — it can be used to find detailed information about the task (see below).

Notes:

• Trying to use the name of an endpoint in the above command, in place of its ID, doesn't work.

• Unlike scp, the destination MUST include the filename (even if it's the same as on the source).

• Personal endpoints do NOT need to be activated (but Globus Connect must be running on them and have a Connected status).

• The paths for the endpoints must use actual directory names (not SMB names).

• When a task has been created, some useful commands are:

  • globus task list — show all current and recent tasks, with their status, for current user.

  • globus task show <task-ID> — show more details about a particular task. The ID can be obtained either from the output of the transfer command, or from the task list command.

  • globus task event-list <task-ID> — If a task has, or might have, any errors, then events will show details of what the errors are. Can also see progress of transfer.

  • globus task wait <task-ID> — waits until task is complete, then returns, so can be used to detect task completion.

• If a task is created and then found to have an unrecoverable error (like trying to use a path or endpoint that doesn't exist), then can cancel that task — rather than have it retry for three days before failing — by:

globus task cancel <task-ID>

Example 2: Transfer a Directory

To transfer a directory from a personal endpoint to a server endpoint (with default options):

• First ensure that Globus Connect is running on the personal endpoint and has a status of "Connected".

globus transfer -r 'personal-endpoint-id:/~/dir_to_copy' 'xxxxxxxx-yyyy-zzzz-wwww-uuuuuuuuuuuu:/full/path/to/test-dir'

Notes:

• If a directory "test-dir" already exists at the destination, then (unlike scp) it will be replaced by "dir_to_copy" (renamed to "test-dir"), rather than putting "dir_to_copy" into "test-dir". If "test-dir" doesn't exist, it will be created.

• All the transfer options that can be chosen in the web interface — and some additional ones — are available for the CLI transfer command. These include:

  • -r : recursive — required to transfer directories

  • -s [exists|size|mtime|checksum] : synchronization level (exists: only copy files not at destination; size: only copy files whose sizes differ; mtime: only copy files where the source file is newer; checksum: only copy files whose checksums differ)

  • --preserve-mtime : preserve modification time if possible

  • --verify-checksum / --no-verify-checksum : Turn on/off verifying that destination has same checksum as source, and re-transferring any files that don't match (on by default).

  • --encrypt : Transfer data in encrypted form.

  • --delete : For recursive transfers, delete any files at destination that aren't are source.

  • --dry-run : Don't submit task, but show details of what it would be if it were submitted.

  • --label <text> : Set a human-readable label for the task (makes it easy to identify in the task list).

  • -h : help — display help information

  • -v : verbose — show additional information in output

Example 3: Remove a File

To remove a file from an endpoint:

globus delete 'aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee:/full/path/to/some/file'

Notes:

• It isn't possible to delete all files matching a pattern at once using wildcards with this method. It is limited to ether deleting a single file as in the example above, or a whole directory (using "-r") as in the example below.

Example 4: Remove a Directory

To remove a directory from an endpoint:

globus delete -r 'xxxxxxxx-yyyy-zzzz-wwww-uuuuuuuuuuuu:/full/path/some/dir'

Notes:

• The "-r" option must be used to remove a directory.

• The "-r" and "-f" options are similar to the Linux/Unix "rm" command.

Example 5: List Contents of a Directory

To list the contents of a directory:

globus ls 'aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee:/full/path/to/dir/'

Notes:

• The "-l" and "-a" options have a similar effect as for the Linux/Unix "ls" command (but "-a" omits "." and "..").

• The above command works with or without the final "/" at the end of the directory name.

• "ls" can only be used for a directory, not a file or selection of files from a directory (as can be done with the Linux/Unix "ls" command).

References

• docs.globus.org/cli/

• docs.globus.org/cli/using-the-cli/

• docs.globus.org/cli/reference/