Goal: Discuss common issues, what workflows can look like, being a good cluster citizen, and some best practices.

...

...

Common Questions

• How do I know what number of nodes, cores, memory etc to ask for my jobs?

• How do I find out whether a cluster/partition supports these resources?

• How do I find out whether these resources are available on the cluster?

• How long will I have to wait in the queue before my job starts? How busy is the cluster?

• How do I monitor the progress of my job?

...

• How do I know what number of nodes, cores, memory etc to ask for my jobs?

  • Understand your software and application. 

    • Read the docs – look at the help for commands/options.

    • Can it run multiple threads - use multi cores (OpenMP) / nodes (MPI)?

    • Can it use a GPU? Nvidia cuda.

    • Are their suggestions on data and memory requirements?

• How do I find out whether a cluster/partition supports these resources?

• How do I find out whether these resources are available on the cluster?

  • Consult the wiki: Beartooth Medicine Hardware Summary Table

• How long will I have to wait in the queue before my job starts? 

  • How busy is the cluster? 

  • Current Cluster utilization: Commands sinfo / arccjobs and SouthPass status / pestat and OnDemand’s MedicineBow System Status page.

• How do I monitor the progress of my job?

  • Slurm commands: squeue

...

Common Issues

• Not defining the account and time options.

• The account is the name of the project you are associated with. It is not your username.

• Requesting combinations of resources that can not be satisfied: Beartooth Hardware Summary Table

  • For example, you can not request 40 cores on a teton node (max of 32).

  • Requesting too much memory, or too many GPU devices with respect to a partition.

• My job is pending? Why? 

  • Because the resources are currently not available.

  • Have you unnecessarily defined a specific partition (restricted yourself) that is busy? 

  • We only have a small number of GPUs.

  • This is a shared resource - sometimes you just have to be patient…

  • Check current cluster utilization.

• Preemption: Users of an investment get priority on their hardware.

  • We have the non-investor partition.

...

Develop/Try/Test:

• Typically use an interactive session (salloc) where you’re typing/trying/testing.

• Are modules available? If not submit a New Software Request to get installed.

• Develop code/scripts.

• Understand how the command-line works – what commands/scripts to call with options.

• Understand if parallelization is available – can you optimize your code/application?

• Test against a subset of data. Something that runs quick – maybe a couple of minutes/hours.

• Do the results look correct?

...

• Put it all together within a bash Slurm script: 

  • Request appropriate resources using #SBATCH

  • Request appropriate wall time – hours, days…

  • Load modules: module load …

  • Run scripts/command-line.

• Finally, submit your job to the cluster (sbatch) using a complete set of data.

  • Use: sbatch <script-name.sh>

  • Monitor job(s) progress.

...

How can I be a good cluster citizen?

• Policies

• Don’t run intensive applications on the login nodes.

• Understand your software/application.

• Shared resource - multi-tenancy.

  • Other jobs running on the same node do not affect each other.

• Don’t ask for everything. Don’t use:

  • mem=0

  • exclusive tag.

  • Only ask for a GPU if you know it’ll be used.

• Use /lscratch for I/O intensive tasks rather than accessing /gscratch over the network. 

  • You will need to copy files back before the job ends.

• Track usage and job performance: seff <jobid>

...