- Created by Simon Alexander, last modified on Aug 02, 2024
You are viewing an old version of this page. View the current version.
Compare with Current View Page History
« Previous Version 9 Next »
Goal: Introduce some further features, such as job efficiency and cluster utilization.
Job Efficiency
[]$ seff 13515489 Job ID: 13515489 Cluster: beartooth User/Group: salexan5/salexan5 State: COMPLETED (exit code 0) Cores: 1 CPU Utilized: 00:00:05 CPU Efficiency: 27.78% of 00:00:18 core-walltime Job Wall-clock time: 00:00:18 Memory Utilized: 0.00 MB (estimated maximum) Memory Efficiency: 0.00% of 8.00 GB (8.00 GB/node)
Note:
Only accurate is the job is successful.
If the job fails with say an OOM: Out-Of-Memory the details will be inaccurate.
This is emailed out if you have Slurm email notifications turned on.
What’s the Current Cluster Utilization?
There are a number of ways to see the current status of the cluster:
arccjobs: Prints a table showing active projects and jobs.pestat: Prints a nodes list with allocated jobs - can query individual nodes.sinfo: View the status of the Slurm partitions or nodes. Status of nodes that are drained can be seen using the -R flag.OnDemand’s MedicineBow System Status page.
arccjobs example
[]$ arccjobs =============================================================================== Account Running Pending User jobs cpus cpuh jobs cpus cpuh =============================================================================== eap-amadson 500 500 30.42 3 3 2.00 amadson 500 500 30.42 3 3 2.00 eap-larsko 1 32 2262.31 0 0 0.00 fghorban 1 32 2262.31 0 0 0.00 pcg-llps 2 64 1794.41 0 0 0.00 hbalantr 1 32 587.68 0 0 0.00 vvarenth 1 32 1206.73 0 0 0.00 =============================================================================== TOTALS: 503 596 4087.14 3 3 2.00 =============================================================================== Nodes 9/51 (17.65%) Cores 596/4632 (12.87%) Memory (GB) 2626/46952 ( 5.59%) CPU Load 803.43 (17.35%) ===============================================================================
pestat example
[salexan5@mblog1 ~]$ pestat
Hostname Partition Node Num_CPU CPUload Memsize Freemem Joblist
State Use/Tot (15min) (MB) (MB) JobID(JobArrayID) User ...
mba30-001 mb-a30 idle 0 96 0.00 765525 749441
mba30-002 mb-a30 idle 0 96 0.00 765525 761311
mba30-003 mb-a30 idle 0 96 0.00 765525 761189
mba30-004 mb-a30 idle 0 96 0.00 765525 761093
mba30-005 mb-a30 idle 0 96 0.00 765525 761357
mba30-006 mb-a30 idle 0 96 0.00 765525 761307
mba30-007 mb-a30 idle 0 96 0.00 765525 761264
mba30-008 mb-a30 idle 0 96 0.00 765525 761184
mba6000-001 mb-a6000 idle 0 64 0.00 1023575 1020075
mbcpu-001 mb* idle 0 96 0.00 1023575 1019621
mbcpu-002 mb* idle 0 96 3.18* 1023575 479265
mbcpu-003 mb* idle 0 96 2.56* 1023575 609488
mbcpu-004 mb* idle 0 96 0.00 1023575 1019530
mbcpu-005 mb* idle 0 96 0.38 1023575 280581
mbcpu-006 mb* idle 0 96 2.06* 1023575 106400*
mbcpu-007 mb* idle 0 96 1.60* 1023575 369035
mbcpu-008 mb* idle 0 96 2.07* 1023575 38900*
mbcpu-009 mb* idle 0 96 0.00 1023575 1019394
mbcpu-010 mb* idle 0 96 0.00 1023575 1019942
mbcpu-011 mb* idle 0 96 0.00 1023575 315295
mbcpu-012 mb* idle 0 96 12.05* 1023575 397291
mbcpu-013 mb* idle 0 96 2.47* 1023575 538938
mbcpu-014 mb* idle 0 96 12.70* 1023575 953417
mbcpu-015 mb* mix 22 96 35.56* 1023575 1002195 1544825 amadson 1544826 amadson 1544824 amadson 1544772 amadson 1544773 amadson 1544708 amadson 1544709 amadson 1544691 amadson 1544692 amadson 1544686 amadson 1544656 amadson 1544646 amadson 1544644 amadson 1544624 amadson 1544622 amadson 1544609 amadson 1544610 amadson 1544592 amadson 1544588 amadson 1544584 amadson 1544573 amadson 1544530 amadson
mbcpu-016 mb* idle 0 96 0.17 1023575 1019397
mbcpu-017 mb* idle 0 96 0.00 1023575 1019767
mbcpu-018 mb* idle 0 96 0.00 1023575 1019609
mbcpu-019 mb* idle 0 96 0.00 1023575 1019667
mbcpu-020 mb* maint* 0 96 0.00 1023575 1019991
mbcpu-021 mb* mix 94 96 143.19* 1023575 69205* 1541097 amadson 1541072 amadson 1541044 amadson 1541031 amadson 1541016 amadson 1544276 amadson 1544221 amadson 1543933 amadson 1544830 amadson 1544829 amadson 1544827 amadson 1544821 amadson 1544815 amadson 1544811 amadson 1544812 amadson 1544802 amadson 1544804 amadson 1544800 amadson 1544792 amadson 1544786 amadson 1544787 amadson 1544788 amadson 1544775 amadson 1544764 amadson 1544748 amadson 1544749 amadson 1544745 amadson 1544741 amadson 1544743 amadson 1544736 amadson 1544727 amadson 1544724 amadson 1544721 amadson 1544717 amadson 1544719 amadson 1544700 amadson 1544701 amadson 1544689 amadson 1544681 amadson 1544682 amadson 1544683 amadson 1544680 amadson 1544677 amadson 1544668 amadson 1544666 amadson 1544648 amadson 1544640 amadson 1544634 amadson 1544635 amadson 1544631 amadson 1544625 amadson 1544615 amadson 1544616 amadson 1544617 amadson 1544613 amadson 1544614 amadson 1544595 amadson 1544594 amadson 1544591 amadson 1544581 amadson 1544578 amadson 1544567 amadson 1544564 amadson 1544566 amadson 1544555 amadson 1544556 amadson 1544557 amadson 1544551 amadson 1544546 amadson 1544548 amadson 1544543 amadson 1544541 amadson 1544542 amadson 1544532 amadson 1544505 amadson 1544501 amadson 1544502 amadson 1544499 amadson 1544490 amadson 1544492 amadson 1544484 amadson 1544487 amadson 1544473 amadson 1544467 amadson 1544468 amadson 1544461 amadson 1544454 amadson 1544433 amadson 1544424 amadson 1544426 amadson 1544409 amadson 1544381 amadson 1544378 amadson 1544368 amadson
mbcpu-022 mb* alloc 96 96 143.42* 1023575 100423* 1541136 amadson 1541134 amadson 1541104 amadson 1544275 amadson 1544271 amadson 1544817 amadson 1544809 amadson 1544810 amadson 1544798 amadson 1544791 amadson 1544789 amadson 1544782 amadson 1544779 amadson 1544776 amadson 1544766 amadson 1544768 amadson 1544760 amadson 1544746 amadson 1544744 amadson 1544742 amadson 1544740 amadson 1544731 amadson 1544726 amadson 1544716 amadson 1544718 amadson 1544713 amadson 1544710 amadson 1544704 amadson 1544705 amadson 1544706 amadson 1544699 amadson 1544703 amadson 1544696 amadson 1544698 amadson 1544688 amadson 1544684 amadson 1544678 amadson 1544675 amadson 1544673 amadson 1544671 amadson 1544670 amadson 1544669 amadson 1544665 amadson 1544664 amadson 1544652 amadson 1544650 amadson 1544647 amadson 1544649 amadson 1544643 amadson 1544645 amadson 1544641 amadson 1544626 amadson 1544619 amadson 1544611 amadson 1544608 amadson 1544604 amadson 1544602 amadson 1544596 amadson 1544582 amadson 1544575 amadson 1544576 amadson 1544574 amadson 1544568 amadson 1544565 amadson 1544561 amadson 1544558 amadson 1544552 amadson 1544547 amadson 1544540 amadson 1544536 amadson 1544538 amadson 1544535 amadson 1544519 amadson 1544521 amadson 1544510 amadson 1544512 amadson 1544503 amadson 1544506 amadson 1544493 amadson 1544489 amadson 1544485 amadson 1544478 amadson 1544479 amadson 1544483 amadson 1544472 amadson 1544470 amadson 1544458 amadson 1544453 amadson 1544449 amadson 1544440 amadson 1544441 amadson 1544415 amadson 1544402 amadson 1544398 amadson 1544370 amadson 1544314 amadson
mbcpu-023 mb* alloc 96 96 134.43* 1023575 43715* 1544292 amadson 1544289 amadson 1544267 amadson 1544820 amadson 1544822 amadson 1544823 amadson 1544819 amadson 1544813 amadson 1544801 amadson 1544803 amadson 1544805 amadson 1544796 amadson 1544784 amadson 1544785 amadson 1544781 amadson 1544783 amadson 1544780 amadson 1544778 amadson 1544777 amadson 1544762 amadson 1544763 amadson 1544758 amadson 1544759 amadson 1544756 amadson 1544752 amadson 1544753 amadson 1544754 amadson 1544750 amadson 1544751 amadson 1544747 amadson 1544737 amadson 1544738 amadson 1544735 amadson 1544734 amadson 1544732 amadson 1544728 amadson 1544723 amadson 1544725 amadson 1544720 amadson 1544722 amadson 1544711 amadson 1544712 amadson 1544707 amadson 1544702 amadson 1544697 amadson 1544694 amadson 1544685 amadson 1544687 amadson 1544679 amadson 1544676 amadson 1544674 amadson 1544667 amadson 1544662 amadson 1544660 amadson 1544661 amadson 1544639 amadson 1544636 amadson 1544638 amadson 1544629 amadson 1544630 amadson 1544627 amadson 1544618 amadson 1544620 amadson 1544612 amadson 1544607 amadson 1544605 amadson 1544598 amadson 1544597 amadson 1544587 amadson 1544583 amadson 1544586 amadson 1544579 amadson 1544577 amadson 1544569 amadson 1544571 amadson 1544553 amadson 1544550 amadson 1544537 amadson 1544534 amadson 1544522 amadson 1544516 amadson 1544517 amadson 1544507 amadson 1544500 amadson 1544488 amadson 1544463 amadson 1544457 amadson 1544456 amadson 1544448 amadson 1544439 amadson 1544435 amadson 1544431 amadson 1544412 amadson 1544374 amadson 1544355 amadson 1544319 amadson
mbcpu-024 mb* alloc 96 96 85.79* 1023575 799871 1541230 amadson 1541228 amadson 1541216 amadson 1541181 amadson 1541147 amadson 1541120 amadson 1541075 amadson 1541049 amadson 1541048 amadson 1541005 amadson 1541427 amadson 1541399 amadson 1541400 amadson 1541366 amadson 1541301 amadson 1541283 amadson 1541281 amadson 1541257 amadson 1542376 amadson 1542257 amadson 1542256 amadson 1542254 amadson 1542129 amadson 1542126 amadson 1542123 amadson 1542053 amadson 1542054 amadson 1542056 amadson 1542005 amadson 1542004 amadson 1542002 amadson 1541891 amadson 1541819 amadson 1541820 amadson 1541764 amadson 1541765 amadson 1541763 amadson 1541676 amadson 1541620 amadson 1541539 amadson 1544203 amadson 1544198 amadson 1544199 amadson 1544200 amadson 1544197 amadson 1544192 amadson 1544193 amadson 1544190 amadson 1544191 amadson 1544189 amadson 1544186 amadson 1544181 amadson 1544179 amadson 1544174 amadson 1544175 amadson 1544171 amadson 1544165 amadson 1544166 amadson 1544162 amadson 1544163 amadson 1544158 amadson 1544159 amadson 1544153 amadson 1544148 amadson 1544147 amadson 1544139 amadson 1544140 amadson 1544137 amadson 1544127 amadson 1544116 amadson 1544118 amadson 1544121 amadson 1544112 amadson 1544110 amadson 1544107 amadson 1544097 amadson 1544098 amadson 1544099 amadson 1544091 amadson 1544094 amadson 1544089 amadson 1544085 amadson 1544086 amadson 1544081 amadson 1544080 amadson 1544077 amadson 1544079 amadson 1544073 amadson 1544037 amadson 1544038 amadson 1543934 amadson 1543935 amadson 1543909 amadson 1543910 amadson 1543884 amadson 1543885 amadson
mbcpu-025 mb alloc 96 96 133.55* 1023575 13523* 1544064 amadson 1544832 amadson 1544831 amadson 1544828 amadson 1544818 amadson 1544816 amadson 1544814 amadson 1544807 amadson 1544808 amadson 1544806 amadson 1544799 amadson 1544793 amadson 1544794 amadson 1544795 amadson 1544797 amadson 1544790 amadson 1544771 amadson 1544774 amadson 1544767 amadson 1544769 amadson 1544770 amadson 1544765 amadson 1544761 amadson 1544757 amadson 1544755 amadson 1544739 amadson 1544733 amadson 1544730 amadson 1544729 amadson 1544715 amadson 1544714 amadson 1544695 amadson 1544693 amadson 1544690 amadson 1544672 amadson 1544663 amadson 1544659 amadson 1544655 amadson 1544658 amadson 1544651 amadson 1544653 amadson 1544654 amadson 1544642 amadson 1544637 amadson 1544632 amadson 1544633 amadson 1544628 amadson 1544623 amadson 1544621 amadson 1544601 amadson 1544603 amadson 1544599 amadson 1544600 amadson 1544590 amadson 1544589 amadson 1544585 amadson 1544580 amadson 1544570 amadson 1544563 amadson 1544562 amadson 1544559 amadson 1544560 amadson 1544554 amadson 1544549 amadson 1544544 amadson 1544545 amadson 1544529 amadson 1544533 amadson 1544527 amadson 1544525 amadson 1544518 amadson 1544509 amadson 1544511 amadson 1544513 amadson 1544514 amadson 1544497 amadson 1544498 amadson 1544481 amadson 1544482 amadson 1544474 amadson 1544471 amadson 1544466 amadson 1544459 amadson 1544455 amadson 1544451 amadson 1544452 amadson 1544436 amadson 1544428 amadson 1544429 amadson 1544419 amadson 1544417 amadson 1544403 amadson 1544388 amadson 1544379 amadson 1544358 amadson 1544335 amadson
mbh100-001 mb-h100 drain$ 0 96 0.00 1281554 0
mbh100-002 mb-h100 idle 0 96 0.01 1281554 1275170
mbh100-003 mb-h100 idle 0 96 0.00 1281554 961682
mbh100-004 mb-h100 idle 0 96 0.00 1281554 984596
mbh100-005 mb-h100 mix 32 96 8.00* 1281554 107608* 1243592 fghorban
mbh100-006 mb-h100 down$ 0 96 0.00 1281554 1275587
mbl40s-001 mb-l40s mix 64 96 55.83* 765525 310548 1480439 hbalantr 1440260 vvarenth
mbl40s-002 mb-l40s idle 0 96 0.00 765525 760896
mbl40s-003 mb-l40s idle 0 96 0.00 765525 761192
mbl40s-004 mb-l40s idle 0 96 0.00 765525 761030
mbl40s-005 mb-l40s idle 0 96 0.00 765525 760728
mbl40s-007 mb-l40s idle 0 96 0.00 765525 761452
wi001 inv-wildiris idle 0 48 0.00 506997 505745
wi002 inv-wildiris idle 0 48 0.00 506997 505726
wi003 inv-wildiris idle 0 48 0.00 506997 505746
wi004 inv-wildiris idle 0 48 0.00 506997 505729
wi005 inv-wildiris idle 0 56 0.00 1031000 1020610
sinfo examples:
# View overall cluster: []$ sinfo -eO "CPUs:8,Memory:9,Gres:14,NodeAIOT:16,NodeList:50" CPUS MEMORY GRES NODES(A/I/O/T) NODELIST 96 1023575 (null) 6/19/0/25 mbcpu-[001-025] 96 765525 gpu:a30:8 0/8/0/8 mba30-[001-008] 96 765525 gpu:l40s:8 1/4/0/5 mbl40s-[001-005] 96 765525 gpu:l40s:4 0/1/0/1 mbl40s-007 64 1023575 gpu:a6000:4 0/1/0/1 mba6000-001 48 506997 (null) 0/4/0/4 wi[001-004] 56 1031000 gpu:a30:2 0/1/0/1 wi005 96 1281554 gpu:h100:8 1/3/2/6 mbh100-[001-006] # View a particular (investment) partition: []$ sinfo -p inv-wildiris PARTITION AVAIL TIMELIMIT NODES STATE NODELIST inv-wildiris up infinite 5 idle wi[001-005] # View compute nodes currently drained: []$ sinfo -R REASON USER TIMESTAMP NODELIST HW Status: Unknown - slurm 2024-07-19T12:02:04 mbh100-001 Not responding slurm 2024-07-30T13:49:06 mbh100-006
Prev | Workshop Home | Next |
- No labels