Versions Compared

Old Version 7

changes.mady.by.user Simon Alexander

Saved on

compared with

New Version 14

changes.mady.by.user Simon Alexander

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Table of Contents
minLevel1
maxLevel1
outlinefalse
stylenone
typelist
printabletrue

...

Job Efficiency

Info

You can view the cpu and memory efficiency of a job using the seff command and providing a <job-id>.

Code Block
[]$ seff 13515489
Job ID: 13515489
Cluster: beartooth<cluster-name>
User/Group: salexan5<username>/salexan5<username>
State: COMPLETED (exit code 0)
Cores: 1
CPU Utilized: 00:00:05
CPU Efficiency: 27.78% of 00:00:18 core-walltime
Job Wall-clock time: 00:00:18
Memory Utilized: 0.00 MB (estimated maximum)
Memory Efficiency: 0.00% of 8.00 GB (8.00 GB/node)
Info

Note:

  • Only accurate

...

  • if the job is successful.

  • If the job fails with say an OOM: Out-Of-Memory the details will be inaccurate.

  • This is emailed out if you have Slurm email notifications turned on.

...

What’s the Current Cluster Utilization?

Info
title

There are a number of ways to see the current status of the cluster:

  • arccjobs: Prints a table showing active projects and jobs.

  • pestat: Prints a nodes node list with allocated jobs - can query individual nodes.

  • OnDemand’s MedicineBow System Status page.

Expand

  • sinfo: View the status of the Slurm partitions or nodes. Status of nodes that are drained can be seen using the -R flag.

  • OnDemand’s MedicineBow System Status page.

Expand
titlearccjobs example
Code Block
[]$ arccjobs
===============================================================================
Account                         Running                      Pending
  User                   jobs    cpus         cpuh    jobs    cpus         cpuh
===============================================================================
eap-amadson               500     500        30.42       3       3         2.00
  amadson                 500     500        30.42       3       3         2.00

eap-larsko                  1      32      2262.31       0       0         0.00
  fghorban                  1      32      2262.31       0       0         0.00

pcg-llps                    2      64      1794.41       0       0         0.00
  hbalantr                  1      32       587.68       0       0         0.00
  vvarenth                  1      32      1206.73       0       0         0.00

===============================================================================
TOTALS:                   503     596      4087.14       3       3         2.00
===============================================================================
Nodes                       9/51      (17.65%)
Cores                     596/4632    (12.87%)
Memory (GB)              2626/46952   ( 5.59%)
CPU Load               803.43         (17.35%)
===============================================================================
Expand
titlepestat example
Code Block
[salexan5@mblog1 ~]$ pestat
Hostname          Partition     Node Num_CPU  CPUload  Memsize  Freemem  Joblist
                               State Use/Tot  (15min)     (MB)     (MB)  JobID(JobArrayID) User ...
mba30-001            mb-a30    idle    0  96    0.00    765525   749441
mba30-002            mb-a30    idle    0  96    0.00    765525   761311
mba30-003            mb-a30    idle    0  96    0.00    765525   761189
...
mba30mbl40s-004            mb-a30l40s    idle    0  96    0.00    765525   761093761030
mba30mbl40s-005            mb-a30l40s    idle    0  96    0.00    765525   761357760728
mba30mbl40s-006  007          mb-a30l40s    idle    0  96    0.00    765525   761307761452
mba30-007wi001            mbinv-a30wildiris    idle    0  9648    0.00    765525506997   761264505745
mba30-008wi002            mbinv-a30wildiris    idle    0  9648    0.00    506997  765525 505726
wi003 761184 mba6000-001        mbinv-a6000wildiris    idle    0  6448    0.00   1023575  1020075506997 mbcpu-001  505746
wi004          inv-wildiris  mb*    idle    0  9648    0.00   1023575 506997 1019621 mbcpu-002 505729
wi005             mb*inv-wildiris    idle    0  9656    3.18*  1023575   479265
mbcpu-003               mb*    idle    0  96    2.56*.00  1023575   609488
mbcpu-004               mb*    idle    0  96    0.00   1023575  1019530
mbcpu-005               mb*    idle    0  96    0.38   1023575   280581
mbcpu-006               mb*    idle    0  96    2.06*  1023575   106400*
mbcpu-007               mb*    idle    0  96    1.60*  1023575   369035
mbcpu-008               mb*    idle    0  96    2.07*  1023575    38900*
mbcpu-009               mb*    idle    0  96    0.00   1023575  1019394
mbcpu-010               mb*    idle    0  96    0.00   1023575  1019942
mbcpu-011               mb*    idle    0  96    0.00   1023575   315295
mbcpu-012               mb*    idle    0  96   12.05*  1023575   397291
mbcpu-013               mb*    idle    0  96    2.47*  1023575   538938
mbcpu-014               mb*    idle    0  96   12.70*  1023575   953417
mbcpu-015               mb*     mix   22  96   35.56*  1023575  1002195  1544825 amadson 1544826 amadson 1544824 amadson 1544772 amadson 1544773 amadson 1544708 amadson 1544709 amadson 1544691 amadson 1544692 amadson 1544686 amadson 1544656 amadson 1544646 amadson 1544644 amadson 1544624 amadson 1544622 amadson 1544609 amadson 1544610 amadson 1544592 amadson 1544588 amadson 1544584 amadson 1544573 amadson 1544530 amadson
mbcpu-016               mb*    idle    0  96    0.17   1023575  1019397
mbcpu-017               mb*    idle    0  96    0.00   1023575  1019767
mbcpu-018               mb*    idle    0  96    0.00   1023575  1019609
mbcpu-019               mb*    idle    0  96    0.00   1023575  1019667
mbcpu-020               mb*   maint*   0  96    0.00   1023575  1019991
mbcpu-021               mb*     mix   94  96  143.19*  1023575    69205* 1541097 amadson 1541072 amadson 1541044 amadson 1541031 amadson 1541016 amadson 1544276 amadson 1544221 amadson 1543933 amadson 1544830 amadson 1544829 amadson 1544827 amadson 1544821 amadson 1544815 amadson 1544811 amadson 1544812 amadson 1544802 amadson 1544804 amadson 1544800 amadson 1544792 amadson 1544786 amadson 1544787 amadson 1544788 amadson 1544775 amadson 1544764 amadson 1544748 amadson 1544749 amadson 1544745 amadson 1544741 amadson 1544743 amadson 1544736 amadson 1544727 amadson 1544724 amadson 1544721 amadson 1544717 amadson 1544719 amadson 1544700 amadson 1544701 amadson 1544689 amadson 1544681 amadson 1544682 amadson 1544683 amadson 1544680 amadson 1544677 amadson 1544668 amadson 1544666 amadson 1544648 amadson 1544640 amadson 1544634 amadson 1544635 amadson 1544631 amadson 1544625 amadson 1544615 amadson 1544616 amadson 1544617 amadson 1544613 amadson 1544614 amadson 1544595 amadson 1544594 amadson 1544591 amadson 1544581 amadson 1544578 amadson 1544567 amadson 1544564 amadson 1544566 amadson 1544555 amadson 1544556 amadson 1544557 amadson 1544551 amadson 1544546 amadson 1544548 amadson 1544543 amadson 1544541 amadson 1544542 amadson 1544532 amadson 1544505 amadson 1544501 amadson 1544502 amadson 1544499 amadson 1544490 amadson 1544492 amadson 1544484 amadson 1544487 amadson 1544473 amadson 1544467 amadson 1544468 amadson 1544461 amadson 1544454 amadson 1544433 amadson 1544424 amadson 1544426 amadson 1544409 amadson 1544381 amadson 1544378 amadson 1544368 amadson
mbcpu-022               mb*   alloc   96  96  143.42*  1023575   100423* 1541136 amadson 1541134 amadson 1541104 amadson 1544275 amadson 1544271 amadson 1544817 amadson 1544809 amadson 1544810 amadson 1544798 amadson 1544791 amadson 1544789 amadson 1544782 amadson 1544779 amadson 1544776 amadson 1544766 amadson 1544768 amadson 1544760 amadson 1544746 amadson 1544744 amadson 1544742 amadson 1544740 amadson 1544731 amadson 1544726 amadson 1544716 amadson 1544718 amadson 1544713 amadson 1544710 amadson 1544704 amadson 1544705 amadson 1544706 amadson 1544699 amadson 1544703 amadson 1544696 amadson 1544698 amadson 1544688 amadson 1544684 amadson 1544678 amadson 1544675 amadson 1544673 amadson 1544671 amadson 1544670 amadson 1544669 amadson 1544665 amadson 1544664 amadson 1544652 amadson 1544650 amadson 1544647 amadson 1544649 amadson 1544643 amadson 1544645 amadson 1544641 amadson 1544626 amadson 1544619 amadson 1544611 amadson 1544608 amadson 1544604 amadson 1544602 amadson 1544596 amadson 1544582 amadson 1544575 amadson 1544576 amadson 1544574 amadson 1544568 amadson 1544565 amadson 1544561 amadson 1544558 amadson 1544552 amadson 1544547 amadson 1544540 amadson 1544536 amadson 1544538 amadson 1544535 amadson 1544519 amadson 1544521 amadson 1544510 amadson 1544512 amadson 1544503 amadson 1544506 amadson 1544493 amadson 1544489 amadson 1544485 amadson 1544478 amadson 1544479 amadson 1544483 amadson 1544472 amadson 1544470 amadson 1544458 amadson 1544453 amadson 1544449 amadson 1544440 amadson 1544441 amadson 1544415 amadson 1544402 amadson 1544398 amadson 1544370 amadson 1544314 amadson
mbcpu-023               mb*   alloc   96  96  134.43*  1023575    43715* 1544292 amadson 1544289 amadson 1544267 amadson 1544820 amadson 1544822 amadson 1544823 amadson 1544819 amadson 1544813 amadson 1544801 amadson 1544803 amadson 1544805 amadson 1544796 amadson 1544784 amadson 1544785 amadson 1544781 amadson 1544783 amadson 1544780 amadson 1544778 amadson 1544777 amadson 1544762 amadson 1544763 amadson 1544758 amadson 1544759 amadson 1544756 amadson 1544752 amadson 1544753 amadson 1544754 amadson 1544750 amadson 1544751 amadson 1544747 amadson 1544737 amadson 1544738 amadson 1544735 amadson 1544734 amadson 1544732 amadson 1544728 amadson 1544723 amadson 1544725 amadson 1544720 amadson 1544722 amadson 1544711 amadson 1544712 amadson 1544707 amadson 1544702 amadson 1544697 amadson 1544694 amadson 1544685 amadson 1544687 amadson 1544679 amadson 1544676 amadson 1544674 amadson 1544667 amadson 1544662 amadson 1544660 amadson 1544661 amadson 1544639 amadson 1544636 amadson 1544638 amadson 1544629 amadson 1544630 amadson 1544627 amadson 1544618 amadson 1544620 amadson 1544612 amadson 1544607 amadson 1544605 amadson 1544598 amadson 1544597 amadson 1544587 amadson 1544583 amadson 1544586 amadson 1544579 amadson 1544577 amadson 1544569 amadson 1544571 amadson 1544553 amadson 1544550 amadson 1544537 amadson 1544534 amadson 1544522 amadson 1544516 amadson 1544517 amadson 1544507 amadson 1544500 amadson 1544488 amadson 1544463 amadson 1544457 amadson 1544456 amadson 1544448 amadson 1544439 amadson 1544435 amadson 1544431 amadson 1544412 amadson 1544374 amadson 1544355 amadson 1544319 amadson
mbcpu-024               mb*   alloc   96  96   85.79*  1023575   799871  1541230 amadson 1541228 amadson 1541216 amadson 1541181 amadson 1541147 amadson 1541120 amadson 1541075 amadson 1541049 amadson 1541048 amadson 1541005 amadson 1541427 amadson 1541399 amadson 1541400 amadson 1541366 amadson 1541301 amadson 1541283 amadson 1541281 amadson 1541257 amadson 1542376 amadson 1542257 amadson 1542256 amadson 1542254 amadson 1542129 amadson 1542126 amadson 1542123 amadson 1542053 amadson 1542054 amadson 1542056 amadson 1542005 amadson 1542004 amadson 1542002 amadson 1541891 amadson 1541819 amadson 1541820 amadson 1541764 amadson 1541765 amadson 1541763 amadson 1541676 amadson 1541620 amadson 1541539 amadson 1544203 amadson 1544198 amadson 1544199 amadson 1544200 amadson 1544197 amadson 1544192 amadson 1544193 amadson 1544190 amadson 1544191 amadson 1544189 amadson 1544186 amadson 1544181 amadson 1544179 amadson 1544174 amadson 1544175 amadson 1544171 amadson 1544165 amadson 1544166 amadson 1544162 amadson 1544163 amadson 1544158 amadson 1544159 amadson 1544153 amadson 1544148 amadson 1544147 amadson 1544139 amadson 1544140 amadson 1544137 amadson 1544127 amadson 1544116 amadson 1544118 amadson 1544121 amadson 1544112 amadson 1544110 amadson 1544107 amadson 1544097 amadson 1544098 amadson 1544099 amadson 1544091 amadson 1544094 amadson 1544089 amadson 1544085 amadson 1544086 amadson 1544081 amadson 1544080 amadson 1544077 amadson 1544079 amadson 1544073 amadson 1544037 amadson 1544038 amadson 1543934 amadson 1543935 amadson 1543909 amadson 1543910 amadson 1543884 amadson 1543885 amadson
mbcpu-025                mb   alloc   96  96  133.55*  1023575    13523* 1544064 amadson 1544832 amadson 1544831 amadson 1544828 amadson 1544818 amadson 1544816 amadson 1544814 amadson 1544807 amadson 1544808 amadson 1544806 amadson 1544799 amadson 1544793 amadson 1544794 amadson 1544795 amadson 1544797 amadson 1544790 amadson 1544771 amadson 1544774 amadson 1544767 amadson 1544769 amadson 1544770 amadson 1544765 amadson 1544761 amadson 1544757 amadson 1544755 amadson 1544739 amadson 1544733 amadson 1544730 amadson 1544729 amadson 1544715 amadson 1544714 amadson 1544695 amadson 1544693 amadson 1544690 amadson 1544672 amadson 1544663 amadson 1544659 amadson 1544655 amadson 1544658 amadson 1544651 amadson 1544653 amadson 1544654 amadson 1544642 amadson 1544637 amadson 1544632 amadson 1544633 amadson 1544628 amadson 1544623 amadson 1544621 amadson 1544601 amadson 1544603 amadson 1544599 amadson 1544600 amadson 1544590 amadson 1544589 amadson 1544585 amadson 1544580 amadson 1544570 amadson 1544563 amadson 1544562 amadson 1544559 amadson 1544560 amadson 1544554 amadson 1544549 amadson 1544544 amadson 1544545 amadson 1544529 amadson 1544533 amadson 1544527 amadson 1544525 amadson 1544518 amadson 1544509 amadson 1544511 amadson 1544513 amadson 1544514 amadson 1544497 amadson 1544498 amadson 1544481 amadson 1544482 amadson 1544474 amadson 1544471 amadson 1544466 amadson 1544459 amadson 1544455 amadson 1544451 amadson 1544452 amadson 1544436 amadson 1544428 amadson 1544429 amadson 1544419 amadson 1544417 amadson 1544403 amadson 1544388 amadson 1544379 amadson 1544358 amadson 1544335 amadson
mbh100-001          mb-h100  drain$    0  96    0.00   1281554        0
mbh100-002          mb-h100    idle    0  96    0.01   1281554  1275170
mbh100-003          mb-h100    idle    0  96    0.00   1281554   961682
mbh100-004          mb-h100    idle    0  96    0.00   1281554   984596
mbh100-005          mb-h100     mix   32  96    8.00*  1281554   107608* 1243592 fghorban
mbh100-006          mb-h100   down$    0  96    0.00   1281554  1275587
mbl40s-001          mb-l40s     mix   64  96   55.83*   765525   310548  1480439 hbalantr 1440260 vvarenth
mbl40s-002          mb-l40s    idle    0  96    0.00    765525   760896
mbl40s-003          mb-l40s    idle    0  96    0.00    765525   761192
mbl40s-004 1031000  1020610
Expand
titlesinfo examples:
Code Block
# View overall cluster:
[]$ sinfo -eO "CPUs:8,Memory:9,Gres:14,NodeAIOT:16,NodeList:50"
CPUS    MEMORY   GRES          NODES(A/I/O/T)  NODELIST
96      1023575  (null)        6/19/0/25       mbcpu-[001-025]
96      765525   gpu:a30:8     0/8/0/8         mb-l40s mba30-[001-008]
96   idle   765525 0  96gpu:l40s:8    0.001/4/0/5     765525   761030 mbl40s-[001-005]
96      765525   mb-gpu:l40s:4    idle0/1/0/1      0  96 mbl40s-007
64  0.00    7655251023575  gpu:a6000:4 760728 mbl40s-007  0/1/0/1         mbmba6000-l40s001
48   idle   506997 0  96(null)    0.00    7655250/4/0/4   761452 wi001     wi[001-004]
56     inv-wildiris 1031000  gpu:a30:2 idle    0/1/0/1  48    0.00   wi005
50699796   505745 wi002  1281554  gpu:h100:8    1/3/2/6  inv-wildiris    idle   mbh100-[001-006]
0
# 48View a particular  0.00    506997   505726
wi003          (investment) partition:
[]$ sinfo -p inv-wildiris
PARTITION    AVAIL  TIMELIMIT  NODES  STATE NODELIST
inv-wildiris    idleup   infinite    0  485    0.00idle wi[001-005]

# View 506997compute nodes currently 505746
wi004   drained:
[]$ sinfo -R
REASON       inv-wildiris    idle    0USER  48    0.00TIMESTAMP    506997   505729 wi005   NODELIST
HW Status: Unknown - slurm  inv-wildiris    idle    0  562024-07-19T12:02:04 mbh100-001
Not responding       slurm 0.00   1031000  1020610 2024-07-30T13:49:06 mbh100-006

...

Prev

Slurm: Workflows and Best Practices 

Workshop Home

Intro to Job Scheduling

Next

Slurm: Common Questions and Issues and How to Resolve