Slurm is the basis of which all jobs are to be submitted, this includes batch and interactive jobs. Slurm consists of several facing user commands, all of which have appropriate Unix man pages associated with them and should be consulted.

Contents

Glossary

Required Inputs and Default Values and Limits

There are some default limits set for Slurm jobs. By default the following is required for submission:

1. Walltime limit(--time=[days-hours:mins:secs]

2. Project account--account=account

Default Values

Additionally, the default submission has the following characteristics:

nodes is for one node (-N 1, --nodes=1)task count one tasks (-n 1, --ntasks-per-node=1)memory amount 1000 MB RAM / CPU (--mem-per-cpu=1000).

These can be changed by requesting different allocation schemes by modifying the appropriate flags. Please reference our Slurm documentation.

Default Limits

On Mount Moran, the default limits were specifically represented by concurrently used cores by each project account. Investors received an increase in concurrent core usage capability. To facilitate more flexible scheduling for all research groups, ARCC is looking at implementing limits based on concurrent usage of cores, memory, and walltime of jobs. This will be defined in the near future and will be subject to the FAC review.

Partitions

Commands

Batch Jobs

Batch jobs are jobs that are submitted via job script or commands that are input into the sbatch command interactively which will then enter the queueing system and prepare for the execution, then execute when possible. The execution could start immediately if the queue is not completely full, start after a short time period if preemption opted for, or after extensive time if the queue is full or running limits are already reached.

A simple sbatch script to submit a simple "Hello World!" type problem follows:

#!/bin/bash

### Assume this file is named hello.sh

#SBATCH --account=arcc
#SBATCH --time=24:00:00

echo "Hello World!"

The two '#SBATCH' directives above are required for all job submissions, whether interactive or batch. The values to account should be changed to the appropriate project account and the time should be changed to an appropriate walltime limit. This is walltime limit, not CPU time. These values could also be supplied when submitting jobs by providing them directly on the command line when submitting. Slurm will default jobs to use one node, one task per node, and once cpu per node.

Submitting Jobs

 $ sbatch hello.sh 

or, with account and time on the command line directly rather than as directives in the shell script:

 $ sbatch --account=arcc --time=24:00:00 test.sh 

Single Node, Multi-Core Jobs

Slurm creates allocations of resources and resources can vary depending on the work needing to be done with the cluster. A batch job that requires multiple cores can have a few different layouts depending on what is intending to be run. If the job is a multi-threaded application such as OpenMP or utilizes pthreads, it's best to set the number of tasks to 1. The below script will request that a single node with 4 cores available. The job script, assuming OpenMP, sets the number of threads to the job provided environment variable SLURM_CPUS_PER_TASK.

#!/bin/bash

#SBATCH --account=arcc
#SBATCH --time=24:00:00
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=1
#SBATCH --cpus-per-task=4

export $OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK
srun ./application

Single Node, Multi-Tasks

This could be a multi-tasked job where the application has it's own parallel processing engine or uses MPI, but experiences poor scaling over multiple nodes.

#!/bin/bash

#SBATCH --account=arcc
#SBATCH --time=24:00:00
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=4
#SBATCH --cpus-per-task=1

### Assuming MPI application
srun ./application

Multi-Node, Non-Multithreaded

An application that strictly uses MPI often can use multiple nodes. However, there is often a chance that MPI type programs do not implement multithreading capability. Therefore, the number of cpus per task should be set to a value of 1.

#!/bin/bash

#SBATCH --account=arcc
#SBATCH --time=24:00:00
#SBATCH --nodes=4
#SBATCH --ntasks-per-node=4
#SBATCH --cpus-per-task=1

### Assuming 'application' is on your $PATH environment variable
srun application

Multi-Node, Multithreaded

Some applications have been developed to take advantage of both distributed memory parallelism and shared memory parallelism such that they're capable of using MPI and threading together. This often requires the user to find the right balance based on additional resources required such as memory per task, network bandwidth, and node core count. The below example request that 4 nodes be allocated, each supporting 4 MPI ranks and each MPI rank supporting 4 threads. The total CPU request count aggregates to 64 (i.e., 4 x 4 x 4).

#!/bin/bash

#SBATCH --account=arcc
#SBATCH --time=24:00:00
#SBATCH --nodes=4
#SBATCH --ntasks-per-node=4
#SBATCH --cpus-per-task=4

export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK

srun application -arg1 -arg2

Checking Status and Canceling

You can use the squeue command to display the status of all your jobs:

$ squeue -u $USER

and scancel to delete a particular job from the queue:

$ scancel <jobid>

Viewing the Results

Once your job has completed, you should see two files in the directory from which you submitted the job. By default, these will be named <jobname>.oXXXXX and <jobname>.eXXXXX (where the <jobname> is replaced by the name of the SLURM script and the X's are replaced by the numerical portion of the job identifier returned by sbatch). In the Hello World example, any output from the job sent to "standard output" will be written to the hello.oXXXXX file and any output sent to "standard error" will be written to the hello.eXXXXX file.

Interactive Jobs

Interactive jobs are jobs that allow shell access to computing nodes where applications can be run interactively, heavy processing of files, or compiling large applications. They can be requested with similar arguments to batch jobs. ARCC has configured the clusters such that Slurm interactive allocations will give shell access on the compute nodes themselves rather than keeping the shell on the login node. The salloc command is appropriate to launch interactive jobs.

 $ salloc --account=arcc --time=40:00 --nodes=1 --ntasks-per-node=1 --cpus-per-task=8 

The value of interactive jobs is to allow users to work interactively with the CLI or interactive use of debuggers (ddt, gdb) , profilers (map, gprof), or language interpreters such as Python, R, or Julia.

Special Hardware / Configuration Requests

Slurm is a flexible and powerful workload manager. It has been configured to allow very good expressiveness to allocate certain features of nodes and specialized hardware. Certain features are requested by the use of Generic Resource or GRES while others are requested through the constraints option.

GPU Requests

Request that 16 cpus 2 GPUs be requested for an interactive session:

 $ salloc -A arcc --time=40:00 -N 1 --ntasks-per-node=1 --cpus-per-task=16 --gres=gpu:2 

Request 16 cpus, 1 GPU of type P100 in a batch script:

#!/bin/bash

#SBATCH --account=arcc
#SBATCH --time=1-00:00:00
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=1
#SBATCH --cpus-per-tasks=16
#SBATCH --gres:P100:1

srun gpu_application

Long Job QOS Configuration

To allow projects to temporarily run jobs for 14 days ARCC has established a special QOS (long-jobs-14) with the following limits:

14-day wall clock time limit
10 max running jobs
As needed ARCC can create other QOS's as needed with different limits.

QOS Creation

To create the QOS for this feature with the issue the following commands as root on tmgt1.

sacctmgr add qos <QOS name> set Flags=PartitionTimeLimit MaxWall=14-0 MaxJobsPA=10

As an example to create a 14 day wall time and max 10 running jobs

sacctmgr add qos long-jobs-14 set Flags=PartitionTimeLimit MaxWall=14-0 MaxJobsPA=10

Allow Access to the QOS

Once the QOS with the proper limits has been created you need to apply it to the project.

sacctmgr modify account <project name> where cluster=teton set qos+=long-jobs-14

Now that you have enabled the long-job-14 QOS on a project inform the users to add:

--qos=long-jobs-14

to there salloc, sbatch or srun command.

Remove Access to the QOS

Once the requirement for the project to run longer jobs is no longer required to remove access for the project to the QOS.

sacctmgr modify account <project name> where cluster=teton set qos-=long-jobs-14

Examples

            Account       User   Def QOS                  QOS 
-------------------- ---------- --------- --------------------

inv-arcc                            arcc          arcc,normal 
 arcc                               arcc          arcc,normal
  arcc                awillou2      arcc          arcc,normal 
  arcc                dperkin6      arcc          arcc,normal 
  arcc                 jbaker2      arcc          arcc,normal 
  arcc                  jrlang      arcc          arcc,normal 
  arcc                mkillean      arcc          arcc,normal 
  arcc                powerman      arcc          arcc,normal 
  arcc                salexan5      arcc          arcc,normal

sacctmgr modify account arcc where cluster=teton set qos+=long-jobs-14

            Account       User   Def QOS                  QOS 
-------------------- ---------- --------- --------------------

inv-arcc                            arcc            arcc,normal 
 arcc                               arcc arcc,long-job-14,norm+ 
  arcc                awillou2      arcc arcc,long-job-14,norm+
  arcc                dperkin6      arcc arcc,long-job-14,norm+
  arcc                 jbaker2      arcc arcc,long-job-14,norm+
  arcc                  jrlang      arcc arcc,long-job-14,norm+
  arcc                mkillean      arcc arcc,long-job-14,norm+
  arcc                powerman      arcc arcc,long-job-14,norm+
  arcc                salexan5      arcc arcc,long-job-14,norm+

 Keep it under wraps for now since this will be allowed on a per request basis.

 There are a couple of things in place to keep from abusing this:

Trouble Shooting

scontrol show node=XXX

root@tmgt1:/apps/s/lenovo/dsa# scontrol show node=mtest2
NodeName=mtest2 Arch=x86_64 CoresPerSocket=10 
   CPUAlloc=0 CPUTot=20 CPULoad=0.02
   AvailableFeatures=ib,dau,haswell,arcc
   ActiveFeatures=ib,dau,haswell,arcc
   Gres=(null)
   NodeAddr=mtest2 NodeHostName=mtest2 Version=18.08
   OS=Linux 3.10.0-693.21.1.el7.x86_64 #1 SMP Fri Feb 23 18:54:16 UTC 2018 
   RealMemory=64000 AllocMem=0 FreeMem=55805 Sockets=2 Boards=1
   State=IDLE+DRAIN ThreadsPerCore=1 TmpDisk=0 Weight=1 Owner=N/A MCS_label=N/A
   Partitions=arcc 
   BootTime=06.08-11:44:57 SlurmdStartTime=06.08-11:47:35
   CfgTRES=cpu=20,mem=62.50G,billing=20
   AllocTRES=
   CapWatts=n/a
   CurrentWatts=0 LowestJoules=0 ConsumedJoules=0
   ExtSensorsJoules=n/s ExtSensorsWatts=0 ExtSensorsTemp=n/s
   Reason=Low RealMemory [slurm@06.10-10:00:27]

free -m

Configuring Slurm for Investments

# Comment describing the investment
PartitionName=inv-<investment-name> AllowQos=<investment-name> \
  Default=No \
  Priority=10 \
  State=UP \
  Nodes=<nodelist> \
  PreemptMode=off \
  TRESBillingWeights="CPU=1.0,Mem=.00025"

Where:

Note: The nodes should also be added to the general partition list, i.e. teton

scontrol reconfigure

/root/bin/idm_scripts/add_slurm_inv inv-<investment-name>

/root/bin/idm_scripts/add_proj_to_inv inv-<investment-name> <project>