Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 25 Next »

Goal: Introduction to Slurm and how to start interactive sessions, submit jobs and monitor.

Workload Managers 

  1. Allocates access to appropriate computer nodes specific to your requests.

  2. Framework for starting, executing, monitoring, and even canceling your jobs.

  3. Queue management and job state notification.

ARCC: Slurm: Wiki Pages 

Slurm Related Commands

Interactive Session: salloc

  • You’re there doing the work.

  • Suitable for developing and testing over a few hours.

[]$ salloc -–help
[]$ man salloc
# Lots of options. 

# The bare minimum.
# This will provide the defaults of one node, one core and 1G of memory.
[]$ salloc –A <project-name> -t <wall-time>

  • As with other Linux commands, there are short and long form for the options.

  • Format for: -t/--time: Acceptable time formats include "minutes", "minutes:seconds", "hours:minutes:seconds", "days-hours", "days-hours:minutes" and "days-hours:minutes:seconds".

Interactive Session: salloc: workshop

  • You’ll only use the reservation for this (and/or other) workshop.

  • Once you have an account you typically do not need it.

  • But there are use cases when we can create a specific reservation for you.

[]$ salloc –A <project-name> –t 1:00 --reservation=<reservation-name>

Interactive Session: squeue: What’s happening?

[]$ salloc -A <project-name> -t 1:00 --reservation=<reservation-name>
salloc: Granted job allocation 13526337
salloc: Nodes m233 are ready for job
# Make a note of the job id.

# Notice the server/node name has changed.
[arcc-t05@m233 intro_to_hpc]$ squeue -u arcc-t05
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
          13526337     moran interact arcc-t05  R       0:19      1 m233
# For an interactive session: Name = interact
# You have the command-line interactively available to you.
[]$ 
...
[]$ squeue -u arcc-t05
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
          13526337     moran interact arcc-t05  R       1:03      1 m233
# Session will automatically time out
[]$ salloc: Job 13526337 has exceeded its time limit and its allocation has been revoked.
slurmstepd: error: *** STEP 13526337.interactive ON m233 CANCELLED AT 2024-03-22T09:36:53 DUE TO TIME LIMIT ***
exit
srun: Job step aborted: Waiting up to 32 seconds for job step to finish.

Interactive Session: salloc: Finished Early?

[]$ salloc -A <project-name> -t 1:00 --reservation=<reservation-name>
salloc: Granted job allocation 13526338
salloc: Nodes m233 are ready for job
[arcc-t05@m233 ...]$ Do stuff…
[]$ exit
exit
salloc: Relinquishing job allocation 13526338

Submit Jobs: sbatch

  • You submit a job to the queue and walk away.

  • Monitor its progress/state using command-line and/or email notifications.

  • Once complete, come back and analyze results.

Submit Jobs: sbatch: Example

The following is an example script that we will use to submit a job to the cluster.

It uses a short test python file defined here: python script.

#!/bin/bash                               
# Shebang indicating this is a bash script.
# Do NOT put a comment after the shebang, this will cause an error.
#SBATCH --account=<project-name>          # Use #SBATCH to define Slurm related values.
#SBATCH --time=10:00                      # Must define an account and wall-time.
#SBATCH --reservation=<reservation-name>
echo "SLURM_JOB_ID:" $SLURM_JOB_ID        # Can access Slurm related Environment variables.
start=$(date +'%D %T')                    # Can call bash commands.
echo "Start:" $start
module purge
module load gcc/13.2.0 python/3.10.6      # Load the modules you require for your environment.
python python01.py                        # Call your scripts/commands.
sleep 1m
end=$(date +'%D %T')
echo "End:" $end

Submit Jobs: squeue: What’s happening?

[]$ sbatch run.sh
Submitted batch job 13526340
[]$ squeue -u <username>
             JOBID PARTITION     NAME      USER  ST       TIME  NODES NODELIST(REASON)
          13526340     moran   run.sh <username>  R       0:05      1 m233
[]$ ls
python01.py  run.sh  slurm-13526340.out

[]$ cat slurm-13526340.out
SLURM_JOB_ID: 13526340
Start: 03/22/24 09:38:36
Python version: 3.10.6 (main, Oct 17 2022, 16:47:32) [GCC 12.2.0]
Version info: sys.version_info(major=3, minor=10, micro=6, releaselevel='final', serial=0)

[]$ squeue -u <username>
             JOBID PARTITION     NAME       USER ST       TIME  NODES NODELIST(REASON)
          13526340     moran   run.sh <username>  R       0:17      1 m233

  • By default, an output of the form: slurm-<job-id>.out will be generated.

  • You can view this file while the job is still running. Only view, do not edit.

Submit Jobs: squeue: What’s happening? Continued

[]$ squeue -u <username>
             JOBID PARTITION     NAME       USER ST       TIME  NODES NODELIST(REASON)
          13526340     moran   run.sh <username>  R       0:29      1 m233
[]$ squeue -u <username>
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)

[]$ cat slurm-13526340.out
SLURM_JOB_ID: 13526340
Start: 03/22/24 09:38:36
Python version: 3.10.6 (main, Oct 17 2022, 16:47:32) [GCC 12.2.0]
Version info: sys.version_info(major=3, minor=10, micro=6, releaselevel='final', serial=0)
End: 03/22/24 09:39:36

  • The squeue command only shows pending and running jobs.

  • If a job is no longer in the queue then it has finished.

  • Finished can mean success, failure, timeout... It’s just no longer running.

More squeue Information

The main Slurm squeue page.

# Lots more information
[]$ squeue --help
[]$ man squeue

# Display more columns:
# For example how much time is left of your requested wall time: TimeLeft
squeue -u <username> --Format="Account,UserName,JobID,SubmitTime,StartTime,TimeLeft"
[]$ squeue -u <username> --Format="Account,UserName,JobID,SubmitTime,StartTime,TimeLeft"
ACCOUNT             USER                JOBID               SUBMIT_TIME         START_TIME          TIME_LEFT
<project-name>      <username>          1795458             2024-08-14T10:31:07 2024-08-14T10:31:09 6-04:42:51
<project-name>      <username>          1795453             2024-08-14T10:31:06 2024-08-14T10:31:07 6-04:42:49
<project-name>      <username>          1795454             2024-08-14T10:31:06 2024-08-14T10:31:07 6-04:42:49
...

Submission from your Current Working Directory

Remember from Linux, that your current location is your Current Working Directory - abbreviated to CWD.

By default Slurm will look for files, and write output, from the folder you submitted your script from i.e. your CWD.

In the example above, if I called sbatch run.sh from ~/intro_to_modules/ then the Python script should reside within this folder. Any output will be written into this folder.

Within the submission script you can define paths (absolute/relative) to other locations.

You can submit a script from any of your allowed locations /home, /project and/or /gscratch.

But you need to manage and describe paths to scripts, data, output appropriately.

Submit Jobs: scancel: Cancel?

[]$ sbatch run.sh
Submitted batch job 13526341
[]$ squeue -u <username>
             JOBID PARTITION     NAME       USER ST       TIME  NODES NODELIST(REASON)
          13526341     moran   run.sh <username>  R       0:03      1 m233
[]$ scancel 13526341
[]$ squeue -u <username>
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)

[]$ cat slurm-13526341.out
SLURM_JOB_ID: 13526341
Start: 03/22/24 09:40:09
Python version: 3.10.6 (main, Oct 17 2022, 16:47:32) [GCC 12.2.0]
Version info: sys.version_info(major=3, minor=10, micro=6, releaselevel='final', serial=0)
slurmstepd: error: *** JOB 13526341 ON m233 CANCELLED AT 2024-03-22T09:40:17 ***

If you know your job no longer needs to be running please cancel it to free up resources - be a good cluster citizen.

Submit Jobs: sacct: What happened?

The main Slurm sacct page.

[]$ sacct -u <username> -X
JobID           JobName  Partition    Account  AllocCPUS      State ExitCode
------------ ---------- ---------- ---------- ---------- ---------- --------
13526337     interacti+      moran arccanetr+          1    TIMEOUT      0:0
13526338     interacti+      moran arccanetr+          1  COMPLETED      0:0
13526340         run.sh      moran arccanetr+          1  COMPLETED      0:0
13526341         run.sh      moran arccanetr+          1 CANCELLED+      0:0

# Lots more information
[]$ sacct --help
[]$ man sacct

# Display more columns:
[]$ sacct -u <username> --format="JobID,Partition,nnodes,NodeList,NCPUS,ReqMem,State,Start,Elapsed" -X
JobID         Partition   NNodes        NodeList      NCPUS     ReqMem      State               Start    Elapsed
------------ ---------- -------- --------------- ---------- ---------- ---------- ------------------- ----------
13526337          moran        1            m233          1      1000M    TIMEOUT 2024-03-22T09:35:25   00:01:28
13526338          moran        1            m233          1      1000M  COMPLETED 2024-03-22T09:37:41   00:00:06
13526340          moran        1            m233          1      1000M  COMPLETED 2024-03-22T09:38:35   00:01:01
13526341          moran        1            m233          1      1000M CANCELLED+ 2024-03-22T09:40:08   00:00:09

Submit Jobs: sbatch: Options

Here are some of the common options available:

[]$ sbatch –-help
#SBATCH --account=<prohect-name>        # Required: account/time
#SBATCH --time=72:00:00
#SBATCH --job-name=workshop             # Job name: Help to identify when using squeue.
#SBATCH --nodes=1                       # Options will typically have defaults.
#SBATCH --tasks-per-node=1              # Request resources in accordance to how you want
#SBATCH --cpus-per-task=1               # to parallelize your job, type of hardware partition
#SBATCH --partition=mb                  # and if you require a GPU.
#SBATCH --gres=gpu:1
#SBATCH --mem=100G                      # Request specific memory needs.
#SBATCH --mem-per-cpu=10G
#SBATCH --mail-type=ALL                 # Get email notifications of the state of the job.
#SBATCH --mail-user=<email-address>
#SBATCH --output=<prefix>_%A.out        # Define a named output file postfixed with the job id.

  • Both salloc and sbatch have 10s of options, in both short and long form.

  • Some options mimic functionality, for example -G works the same as --gres=gpu:1.

  • Please consult the command --help and man pages and/or web links to discover further options not listed.

Submit Jobs: sbatch: Options: Applied to Example

Let’s take the previous example, and add some of the additional options:

#!/bin/bash                               
#SBATCH --account=<project-name>
#SBATCH --time=10:00
#SBATCH --reservation=<reservation-name>

#SBATCH --job-name=pytest
#SBATCH --nodes=1                       
#SBATCH --cpus-per-task=1               
#SBATCH --mail-type=ALL                 
#SBATCH --mail-user=<email-address>
#SBATCH --output=slurms/pyresults_%A.out      

echo "SLURM_JOB_ID:" $SLURM_JOB_ID        # Can access Slurm related Environment variables.
start=$(date +'%D %T')                    # Can call bash commands.
echo "Start:" $start
module purge
module load gcc/13.2.0 python/3.10.6      # Load the modules you require for your environment.
python python01.py                        # Call your scripts/commands.
sleep 1m
end=$(date +'%D %T')
echo "End:" $end

Notice:

  • I’ve given the job a specific name and have requested email notifications.

  • The output is written to a sub folder slurm/ with a name of the form pytest_<jobid>.out

Extended Example: What Does the Run look Like?

With the above settings, a submission will look something like the following:

 Example Flow and Output:
# Submit the job:
[intro_to_modules]$ sbatch run.sh
Submitted batch job 1817260

# Notice the NAME is now 'pytest'
[intro_to_modules]$ squeue -u salexan5
             JOBID PARTITION     NAME       USER ST       TIME  NODES NODELIST(REASON)
           1817259        mb   pytest <username>  R       0:58      1 mbcpu-002

# I can view the output while the job is running.
# The output is now in a sub folder under slurm/
# It also uses the name 'pyresults_<job_id>.out'
[intro_to_modules]$ cat slurms/pyresults_1817260.out
SLURM_JOB_ID: 1817260
Start: 08/14/24 14:48:38
Python version: 3.10.6 (main, Apr 30 2024, 11:23:04) [GCC 13.2.0]
Version info: sys.version_info(major=3, minor=10, micro=6, releaselevel='final', serial=0)

[intro_to_modules]$ squeue -u <username>
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)

[intro_to_modules]$ cat slurms/pyresults_1817260.out
SLURM_JOB_ID: 1817260
Start: 08/14/24 14:48:38
Python version: 3.10.6 (main, Apr 30 2024, 11:23:04) [GCC 13.2.0]
Version info: sys.version_info(major=3, minor=10, micro=6, releaselevel='final', serial=0)
End: 08/14/24 14:49:38

In my inbox, I also received two emails with the subjects:

  1. medicinebow Slurm Job_id=1817260 Name=pytest Began, Queued time 00:00:00

    1. This will have no text within the email body.

  2. medicinebow Slurm Job_id=1817260 Name=pytest Ended, Run time 00:01:01, COMPLETED, ExitCode 0

    1. The body of this email contained the seff results.

 

Workshop Home

Intro to Job Scheduling

Next

Slurm: Workflows and Best Practices

 

 

  • No labels