While processing, you may also need to: | A key principle of any shared computing environment is that resources are shared among users and therefore must be scheduled. Please DO NOT simply log into the HPC and run your computations without requesting or scheduling resources from Slurm through a batch script or Interactive job. ARCC uses the Slurm Workload Manager to regulate and schedule user submitted jobs on our HPC systems. In order for your job to submit properly to Slurm, you must at minimum specify your account and a time in your submission. There are 2 ways to run your work on an ARCC HPC systems from the Command Line: Option 1: Run it as an Interactive JobThese are jobs that allow users access to computing nodes where applications can be run in real time. This may be necessary when performing heavy processing of files, or compiling large applications. Interactive jobs can be requested with an salloc command. ARCC has configured the clusters so that interactive jobs provide shell access on compute nodes themselves rather than running on the login node. An example of an salloc request can be expanded below. | Expand |
|---|
| title | Interactive Job Examples and Explanations ( Examples include using: salloc, --account, --time, --partition, --nodes, --cpus-per-task --mem) |
|---|
| The following is the simplest example of a command to start an interactive job. This command has the bare minimum information (account and time) in order to run any job on an ARCC cluster: | Code Block |
|---|
[cowboyjoe@hpclog1 ~]$ salloc --account=<your project name> --time=01:00 |
Breaking it down: [cowboyjoe@hpclog1 ~]$ This is our command prompt and specifies our username (cowboyjoe), the node on which we're currently working from (hpclog1), and the folder in which we're located on the HPC (~ which is short for our /home directory). To learn more about command line in Linux, see our Linux Command Line tutorial here.
salloc is the slurm command to allocate a work session on the cluster
--account is a flag specifying the account/project under which you’re performing your work in the session.
--timeis a flag specifying your “walltime limit” which is how long you will have access to the HPC resources you’re requesting in your work session. At the end of this time, you will be disconnected from your requested resources.
Syntax for time may be in the form of “minutes”, “minutes:seconds”, “hours:minutes:seconds”, “days-hours”, days-hours:minutes”, or “days-hours:minutes:seconds”. On ARCC HPC, the maximum time you may request for any job is 7 days (job runtime may be extended, upon request).
In this basic example, aside from time requested and account used, all allocated resources are set to the default. Total CPUs/cores available for this session is set to 1 by default. Total nodes we have access to in our session is set to 1 by default. Total memory allocated to the session is 1GB by default.
The next example is a set of commands. The first line is a command to allocate an interactive job requesting specific hardware to perform the computations in our session. The second line runs a python script: | Code Block |
|---|
[cowboyjoe@hpclog1 ~]$ salloc --account=arcc --time=40:00 --partition=moran --nodes=1 --cpus-per-task=8 --mem=8G
python my_job_sequential_steps.py |
Breaking it down: --account is a flag specifying the account/project under which you’re performing your work in the session. In this case, the project’s name is arcc.
--timeis a flag specifying the “walltime limit” for this job, in this case, 40 minutes.
--partition is a flag specifying which partition we want our nodes to come from for this session. You can view the partitions and learn about hardware specific to different partitions by viewing the hardware summary associated with the HPC you’re using. Since our overall computational needs are not significant, we requested the moran partition. These nodes have 16 cores/node and at least 32GB of RAM/node.
--nodes is a flag telling Slurm how many compute nodes we need available to us to run the computations we want to run in the session. In this example, we only ask for 1.
--cpus-per-task is a flag specifying how many cores/cpus we need available to run any single task. By default, ntasks (the number of tasks we run concurrently) is set to 1. Since our salloc command didn't specify --ntasks or any other "tasks-per" related parameters the total cpu for the requested session will be what was requested using the --cpus-per-task flag. In this example, 8 cores x (default) 1 tasks run concurrently = 8 cores total.
--mem is a flag to request the minimum memory/RAM per node that we’ll need for our job. Above we requested 8G, so 8GB. This is under the 32GB of RAM/node for the partition (moran) we asked for in our salloc command, so Slurm accepts this memory request.
The -mem flag should be followed by a unit prefix (G for GB). If a unit prefix is not specified and only an integer is provided, default prefix is M (megabytes).
In response, Slurm assigns us a single node (node1), with access to a total of 8 cores and 8GB RAM for 40 minutes.
| Code Block |
|---|
salloc: Granted job allocation 1012024
salloc: Nodes node1 are ready for job |
In the last line we have been granted our requested resources from the Slurm scheduler and use them to run a python script named ‘my_job_sequential_steps.py'using the default version of python installed on the cluster. If we need more hardware resources than what we asked for in our salloc command, we may get an error (such as a “oom-kill” or out of memory error indicating we didn’t request enough RAM to run our session), or our job will run for an extremely long time (which may mean we didn’t request enough CPU and/or we didn’t parse out our computational work appropriately in the script).
|
Option 2: Run it as a Batch JobThis means running of one or more tasks on a computer environment. Batch jobs are initiated using scripts or command-line parameters. They run to completion without further human intervention (fire and forget). Batch jobs are submitted to a job scheduler (on ARCC HPC, Slurm) and run on the first available compute node(s). | Expand |
|---|
| title | Batch Script Example and Explanation (Example using --account, --time, --partition, --mem, --job-name, --mail-type, --mail-user) |
|---|
| In the following example we need to create our own batch script which then gets run by Slurm to execute your jobs and any associated tasks. Below is an example of a batch script we created named myfirstjob.sh that then runs our computational work in a python script named my_job_sequential_steps.py: | Code Block |
|---|
#!/bin/bash
#SBATCH --account=myproject
#SBATCH --time=1-01:00:00
#SBATCH --partition=moran
#SBATCH --mem=8G
#SBATCH --job-name sequential_run
#SBATCH --mail-type=ALL
#SBATCH --mail-user=cowboyjoe@uwyo.edu
python my_job_sequential_steps.py |
Breaking it down: #!/bin/bash is the “shebang” line, telling which HPC to use the bash shell to interpret the script.
--account is a flag specifying the account/project under which you’re performing your work in the session. Here the account we’re using is myproject
--timeis a flag specifying your “walltime limit”. This is how long the script can run on HPC resources once it begins. At the end of this time, the script will end regardless of whether computations are completed. The example sets total time for our job to 1-02:15:45, 1 day, 2 hours, 15 minutes, 45 seconds.
Syntax for time may be in the form of “minutes”, “minutes:seconds”, “hours:minutes:seconds”, “days-hours”, days-hours:minutes”, or “days-hours:minutes:seconds”. On ARCC HPC, the maximum time you may request for any job is 7 days (job runtime may be extended, upon request).
--partition is a flag specifying which nodes we want to use for our job. You can view the partitions and learn about hardware specific to different partitions by viewing the hardware summary associated with the HPC you’re using. Since our overall computational needs are not significant, we requested to run on the moran partition. These nodes have 16 cores/node and at least 32GB of RAM/node.
--mem is a flag to request the minimum memory/RAM per node that we’ll need for our job. Above we specify 8G, so 8GB. This is under the 32GB of RAM/node for the partition (moran) we asked for in our salloc command, so Slurm accepts this memory request.
The -mem flag should be followed by a unit prefix (G for GB). If a unit prefix is not specified and only an integer is provided, default prefix is M (megabytes).
--job-name is a flag to specify the name of our job allocation. This will appear with the job id number if we query running jobs on the HPC.
--mail-type is a flag to specify which job events should trigger notification e-mails. Setting it to ALL means notification e-mails will be sent when a job begins, ends, fails, hits time limits, never runs due to problems with the request, or gets requeued.
--mail-user is a flag to specifies the e-mail address to notify when job events occur.
In this basic example, aside from time requested and account used, all allocated resources are set to the default. Total CPUs/cores available for this session is set to 1 by default. Total nodes we have access to in our session is set to 1 by default. Total tasks we will run concurrently is set to 1 by default.
In the last line we run a python script named ‘my_job_sequential_steps.py' using the default version of python installed on the cluster.
Assuming our batch script and the python script are complete and ready to run, we log into the HPC and run it on the cluster to submit our job by navigating to the location of our script, then running it with the following command: | Code Block |
|---|
sbatch myfirstjob.sh |
Since this batch script first makes a request to Slurm to schedule our job and allocate resources before performing any computations, we can submit it on the login node. |
To learn more about running parallel jobs, running jobs with GPUs, and avoid more common issues, see our SLURM tutorial. |