Installing Software Yourself on an HPC Cluster

Owned by Abigail Smith (Unlicensed)
Last updated: Jun 16, 2022 by TLBrewer
6 min read
1 Overview | 2 To Install a Software Module In Your Home Directory or Project Directory | 3 Further Information | 4 HPC Cluster Software Requests | 5 Before Requesting a Software Installation, Please Consider the Following: | 6 Install Your Own Software

Overview

There are different ways that software modules can be installed on the Teton cluster, including modules that are available to everyone and modules that are available to a specific project or user. When new software installation is requested, knowing whether the expected users are an individual, a team/lab, or a wider group of users across different projects, and the software installation complexity, can help to determine the best way to install the software. In general, graphical applications are likely to be more complex to run in this type of environment.

To install the software yourself, please follow the process below. For more information on selecting the type of installation to use, or requesting system-wide installation, see Further Information below.

To Install a Software Module In Your Home Directory or Project Directory

Preparation

To keep software in your home directory organized, it is useful to create a subdirectory in which to install it, and a subdirectory for the corresponding module files. For this example, we'll use the following two subdirectories in the home directory:

drwxrwxr-x 5 uname uname 4096 Oct 15 16:34 mf drwxrwxr-x 6 uname uname 4096 Oct 15 11:45 software

  • Decide whether the installation destination will be the software directory itself, or subdirectories of this directory:

    • If you plan to install software modules so that all of them will be loaded at the same time, then installing the software directly into the above software directory could work.

    • If you want to be able to load one of your modules without loading the rest of them (e.g. to have the option of choosing between different versions of some software package), then the software directory could have different subdirectories for each module you create. This is because the module files will add selected locations to your active path to be able to run the software found there.

  • You may want to make a subdirectory (such as src) inside the software directory in which to put downloaded source code, to use for building it.

  • Download the source code of the software package, if available check its checksum to make sure that the correct file was obtained, place the source file in your chosen location (e.g. software/src), and unpack it according to to the archiving/compression format used.

Installation

  1. Read the installation documentation for the process of building the software from the source. Specifically, note:

    • whether it uses standard build tools already available on the system (compilers, make, cmake, etc.), or specialized build tools that would need to be installed as well

    • how to select an installation destination

    • dependencies on other software (including versions)

  2. Make all dependencies available by:

    • For any which are available modules, load the module (including the required compiler module).

    • For any which aren't otherwise available, use this process to create them as modules in your home directory, and then load them.

  3. Follow the installation instructions to install the software into the destination determined in the preparation step above. (For a "configure, make, make install" process, this is usually by an option to the configure command.)

Module Creation

  1. In your module files directory (called mf above), create a subdirectory which is the name of the software package as one word in all lower cases. (This name should generally not include the version number, as the version will be used later for a file name.)

  2. In the subdirectory just created, make a text file named software-version-number.lua, where software-version-number is the version of the software package just installed. (This organization allows modules to be created for multiple versions of a software package.) Edit this file to have content like:

-- sw-app-name ver.num -- -- -- -- help([[ sw-app-name is a package that does something. ]]) prepend_path("PATH", "/home/uname/software/bin")

where:

  • sw-app-name is the name of the software package,

  • ver.num is the version number of the software package,

  • sw-app-name is a package that does something is a brief description of what the software is or does,

  • /home/uname/software/bin is the full path to the directory containing the executable file for running the software.

Some software modules need additional details in this module file. The above example is a minimal one that can be used to start from. If you need help with a module file, please contact ARCC.

To Test the Use of the Module Just Made:

  1. Run:

    module use /home/uname/mf

    where /home/uname/mf is your module files directory.

  2. Run:

    and the module you just created should be listed.

  3. Run:

    where sw-app-name is the name of the directory (created in step 1 of Module creation) for the module in the module files directory.

  4. Run:

    and you should see the expected path.

  5. Try using sw-app-name as you normally would for some test cases.

Further Information

To Determine What Type of Installation to Use

  1. Search the installed modules for the software using Searching for Modules.

  2. If the software isn't yet installed, consider how widespread its expected use is.

    • If you expect it to be used beyond your own project or lab, then a system-wide installation request may make sense.

    • If you expect it to only be used by you or your team, then installing it into your home directory or project directory may be more appropriate, and it is possible for you to either do this yourself or request help from ARCC with it.

  3. Consider possible restrictions on installing the software:

    • Modules on the cluster are organized in a hierarchy, including multiple versions of some key packages, in a way designed to support a wide range of different projects with different requirements and types of jobs they need to run. As a consequence, if an application has requirements that conflict with how other modules on the system are set up, it may not be possible to add it.

    • Some software has licensing restrictions that limit how or whether it can be installed in a cluster environment. If you request licensed software, it is your responsibility to find out whether a license is currently available at UW, who holds the license, any constraints it may have on running on a cluster, and technical requirements for using the license, and to provide this information with your request.

    • Some software has dependencies or other technical requirements that may make it infeasible to install. This may include certain graphical applications, security risks, software not supported on Linux, or software with a specialized build process that doesn't allow a suitable installation destination to be selected to be able to install it as a module for an HPC system.

  4. If you need a Python module not currently available, you can install it to your home directory using Miniconda.

To Request System-Wide Installation of a Software Module

HPC Cluster Software Requests

There are different ways that software modules can be installed on HPC clusters, including modules that are available to everyone and modules that are available to a specific project or user. Knowing how wide the expected usage is, and the complexity, dependencies, and restrictions of the software can help to determine the best way to install it.

If there is some software that you would like to run on the cluster, please follow the process below.

Before Requesting a Software Installation, Please Consider the Following:

Is the Software Already Installed on the Cluster?

Search the installed modules for the software using Searching for Modules.

Consider Utility and Complexity

Are you the only user of this software?

Will it be used by team members in your project?

How complex or difficult is this software to install?

  • Software that is complex, conflicts with other packages, or has several dependencies may require extra effort. These types of software are excellent candidates for using miniConda or Singularity. If you are unsure or have complex software, contact us.

Install Your Own Software

  1. Obtain software website and installation instructions

  2. Determine required dependencies, they may already be modules installed on the cluster

  3. Determine if home or project directory install is appropriate

  4. Consider using miniConda or Singularity to create a sandboxed software environment:

  5. Consider possible restrictions on installing the software. Not all software is allowed or compatible on the cluster. If you are unsure, contact us.