1. Allocates access to appropriate computer nodes specific to your requests.

2. Framework for starting, executing, monitoring, and even canceling your jobs.

3. Queue management and job state notification.

You’re there doing the work.

Suitable for developing and testing over a few hours.

# CPU only compute node.
[]$ salloc –A <project-name> –t 1:00 --reservation=<reservation-name>

Use the squeue command to find a list of jobs currently pending/running.

This list can be 10s/100s/1000s of lines long.

Use the -u option with your <username> to specifically look at your jobs.

[]$ salloc -A <project-name> -t 1:00 --reservation=<reservation-name>
salloc: Granted job allocation 13526337
salloc: Nodes m233 are ready for job
# Make a note of the job id.

# Notice the server/node name has changed.
[arcc-t05@m233 intro_to_hpc]$ squeue -u arcc-t05
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
          13526337     moran interact arcc-t05  R       0:19      1 m233
# For an interactive session: Name = interact
# You have the command-line interactively available to you.
[]$ 
...
[]$ squeue -u arcc-t05
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
          13526337     moran interact arcc-t05  R       1:03      1 m233
# Session will automatically time out
[]$ salloc: Job 13526337 has exceeded its time limit and its allocation has been revoked.
slurmstepd: error: *** STEP 13526337.interactive ON m233 CANCELLED AT 2024-03-22T09:36:53 DUE TO TIME LIMIT ***
exit
srun: Job step aborted: Waiting up to 32 seconds for job step to finish.

[]$ salloc -A <project-name> -t 1:00 --reservation=<reservation-name>
salloc: Granted job allocation 13526338
salloc: Nodes m233 are ready for job
[arcc-t05@m233 ...]$ Do stuff…
[]$ exit
exit
salloc: Relinquishing job allocation 13526338

• You submit a job to the queue and walk away.

• Monitor its progress/state using command-line and/or email notifications.

• Once complete, come back and analyze results.

The following is an example bash submission script that we will use to submit a job to the cluster.

It uses a short test python file defined here: python script.

Remember: Use the squeue command to find a list of your jobs currently pending/running.

[]$ sbatch run.sh
Submitted batch job 13526340

[]$ squeue -u <username>
             JOBID PARTITION     NAME      USER  ST       TIME  NODES NODELIST(REASON)
          13526340     moran   run.sh <username>  R       0:05      1 m233
[]$ ls
python01.py  run.sh  slurm-13526340.out

[]$ cat slurm-13526340.out
SLURM_JOB_ID: 13526340
Start: 03/22/24 09:38:36
Python version: 3.10.6 (main, OctSep  173 20222024, 1615:4713:3256) [GCC 1214.2.0]
Version info: sys.version_info(major=3, minor=10, micro=6, releaselevel='final', serial=0)

[]$ squeue -u <username>
             JOBID PARTITION     NAME       USER ST       TIME  NODES NODELIST(REASON)
          13526340     moran   run.sh <username>  R       0:17      1 m233

The squeue command only shows pending and running jobs.

If a job is no longer in the queue then it has finished.

For more information see the main Slurm squeue page and use.

[]$ squeue --help
[]$ man squeue

For example, using the --Format option you can display additional columns, such as how much time is left from your requested wall time using TimeLeft:

[]$ squeue -u <username> --Format="Account,UserName,JobID,SubmitTime,StartTime,TimeLeft"
[]$ squeue -u <username> --Format="Account,UserName,JobID,SubmitTime,StartTime,TimeLeft"
ACCOUNT             USER                JOBID               SUBMIT_TIME         START_TIME          TIME_LEFT
<project-name>      <username>          1795458             2024-08-14T10:31:07 2024-08-14T10:31:09 6-04:42:51
<project-name>      <username>          1795453             2024-08-14T10:31:06 2024-08-14T10:31:07 6-04:42:49
<project-name>      <username>          1795454             2024-08-14T10:31:06 2024-08-14T10:31:07 6-04:42:49
...

Remember from Linux, that your current location is your Current Working Directory - abbreviated to CWD.

By default Slurm will look for files, and write output, from the folder you submitted your script from i.e. your CWD.

In the example above, if I called sbatch run.sh from ~/intro_to_modules/ then the Python script should reside within this folder. Any output will be written into this folder.

Within the submission script you can define paths (absolute/relative) to other locations.

You can submit a script from any of your allowed locations /home, /project and/or /gscratch.

But you need to manage and describe paths to scripts, data, output appropriately.

If you have submitted a job, and for what ever reason you want/need to stop it early, then use scancel <job-id>.

This will stop the job at its current point within the computation, and return any associated resources back to the cluster.

[]$ sbatch run.sh
Submitted batch job 13526341
[]$ squeue -u <username>
             JOBID PARTITION     NAME       USER ST       TIME  NODES NODELIST(REASON)
          13526341     moran   run.sh <username>  R       0:03      1 m233

[]$ scancel 13526341

[]$ squeue -u <username>
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)

[]$ cat slurm-13526341.out
SLURM_JOB_ID: 13526341
Start: 03/22/24 09:40:09
Python version: 3.10.6 (main, OctSep  173 20222024, 1615:4713:3256) [GCC 1214.2.0]
Version info: sys.version_info(major=3, minor=10, micro=6, releaselevel='final', serial=0)
slurmstepd: error: *** JOB 13526341 ON m233 CANCELLED AT 2024-03-22T09:40:17 ***

Use the sacct command to view your jobs that have completed.

By default this will only list jobs from mid night of the that day.

View the -S, --starttime (and -E, --endtime=<end_time>) options to understand how to define a start (and end) time to configure different date/time intervals.

It too has a --format option allowing you to display additional columns:

For more information see the main Slurm sacct page and use:

[]$ sacct --help
[]$ man sacct

In my inbox, I also received two emails with the subjects:

1. medicinebow Slurm Job_id=1817260 Name=pytest Began, Queued time 00:00:00

   1. This will have no text within the email body.

2. medicinebow Slurm Job_id=1817260 Name=pytest Ended, Run time 00:01:01, COMPLETED, ExitCode 0

   1. The body of this email contained the seff results.