The Slurm configuration on Teton is quite complicated to help with the layout of hardware, investors, and runtime limits. The following tables represent the partition on Teton. Some require a QoS which will be auto-assigned during job submission. The tables represent the Slurm allocatable units rather than hardware units.

General Partitions

Investor Partitions

Investor partitions are likely to be quite heterogeneous and may have a mix of hardware and are indicated below where appropriate. They require a special QoS for access.

Special Partitions

Special partitions require access to be given directly to user accounts or project accounts and likely require additional approval for access.

More Details

Generally, to run a job on a cluster you will need the following:

• User Account

• UW Sponsored Project

• SSH connection to the cluster

A handy migration reference to compare MOAB/Torque commands to SLURM commands can be found on the SLURM home site: Batch System Rosetta Stone.

• Node won't come online

If a node won't come online for some reason check the node information for a slurm reason. run

scontrol show node=XXX

The command output should include a reason for why slurm won't bring the node online. As an example:

root@tmgt1:/apps/s/lenovo/dsa# scontrol show node=mtest2
NodeName=mtest2 Arch=x86_64 CoresPerSocket=10 
   CPUAlloc=0 CPUTot=20 CPULoad=0.02
   AvailableFeatures=ib,dau,haswell,arcc
   ActiveFeatures=ib,dau,haswell,arcc
   Gres=(null)
   NodeAddr=mtest2 NodeHostName=mtest2 Version=18.08
   OS=Linux 3.10.0-693.21.1.el7.x86_64 #1 SMP Fri Feb 23 18:54:16 UTC 2018 
   RealMemory=64000 AllocMem=0 FreeMem=55805 Sockets=2 Boards=1
   State=IDLE+DRAIN ThreadsPerCore=1 TmpDisk=0 Weight=1 Owner=N/A MCS_label=N/A
   Partitions=arcc 
   BootTime=06.08-11:44:57 SlurmdStartTime=06.08-11:47:35
   CfgTRES=cpu=20,mem=62.50G,billing=20
   AllocTRES=
   CapWatts=n/a
   CurrentWatts=0 LowestJoules=0 ConsumedJoules=0
   ExtSensorsJoules=n/s ExtSensorsWatts=0 ExtSensorsTemp=n/s
   Reason=Low RealMemory [slurm@06.10-10:00:27]

This indicates that the memory definition for the node and what Slurm actually found are different. You can use

free -m

to see what the system thinks it has in terms of memory.

The node definition should have a memory definition less or equal to the total showed by the "free" command. You should verify that the settings are correct for the memory the node should have. If not, investigate and determine why the discrepancy.

The Teton cluster is the University of Wyoming's Condo cluster which provides computing resources to the general UW research community. Being a condo cluster researchers can invest funds into the cluster in order to expand its functionality. As an investor, a researcher is afforded special privileges specifically first access to the nodes their funds purchased.

To establish an investment within Slurm follow the following steps:

1. First, define an investor partition that refers to the purchased nodes. Create the partition definition, edit /apps/s/slurm/latest/etc/partitions-invest.conf. Add

# Comment describing the investment
PartitionName=inv-<investment-name> AllowQos=<investment-name> \
  Default=No \
  Priority=10 \
  State=UP \
  Nodes=<nodelist> \
  PreemptMode=off \
  TRESBillingWeights="CPU=1.0,Mem=.00025"

Where:

• investment-name is the name you wish to call the new investment

• nodelist is the list of nodes to be included in the investment definition, i.e. t[305-315],t317

• Adjust the TRESBillingWeights accordingly based on the node specifications

Note: The nodes should also be added to the general partition list, i.e. teton