Goal: List some common issues and how to resolve.

...

• How do I know what number of nodes, cores, memory etc to ask for my jobs?

  • Understand your software and application. 

    • Read the docs – look at the help for commands/options.

    • Can it run multiple threads - use multi cores (OpenMP) / nodes (MPI)?

    • Can it use a GPU? Nvidia cuda.

    • Are their suggestions on data and memory requirements?

• How do I find out whether a cluster/partition supports these resources?

• How do I find out whether these resources are available on the cluster?

  • Consult the wiki: Medicine Hardware Summary Table

• How long will I have to wait in the queue before my job starts? 

  • How busy is the cluster? 

  • Current Cluster utilization: Commands sinfo / arccjobs / pestat and OnDemand’s MedicineBow System Status page.

• How do I monitor the progress of my job?

  • Slurm commands: squeue

...

Common Issues

• Not defining the account and time options.

• The account is the name of the project you are associated with. It is not your username.

• Requesting combinations of resources that can not be satisfied: Medicine Hardware Summary Table

  • For example, you can not request 40 cores on a teton compute node (with a max of 32).

  • Requesting too much memory, or too many GPU devices with respect to a partition.

• My job is pending? Why? 

  • Because the resources are currently not available.

  • Have you unnecessarily defined a specific partition (restricted yourself) that is busy? 

  • We only have a small number of GPUs.

  • This is a shared resource - sometimes you just have to be patient…

  • Check current cluster utilization.

• Preemption: Users of an investment get priority on their hardware.

  • We have the non-investor partition.

...

[]$ pestat -n mbl40s-001
Select only nodes in hostlist=mbl40s-001
Hostname         Partition     Node Num_CPU  CPUload  Memsize  Freemem  Joblist
                              State Use/Tot  (15min)     (MB)     (MB)  JobID(JobArrayID) User ...
mbl40s-001         mb-l40s     mix   64  96   55.62*   765525   310618  1480439 hbalantr 1440260 vvarenth

...

...