Versions Compared

Old Version 10

changes.mady.by.user Abigail Smith (Unlicensed)

Saved on

compared with

New Version 11

changes.mady.by.user Abigail Smith (Unlicensed)

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

...

Overview

...

Teton is an Intel x86_64 cluster connected via a Mellanox FDR/EDR InfiniBand and has a 1.3 PB IBM Spectrum Scale global parallel filesystem available to all nodes. The system requires UWYO two-factor authentication (2FA) for login via SSH. The default shell is BASH with Lmod modules system is leveraged for dynamic user environments to help switch software stacks rapidly and easily. The Slurm workload manager is employed to schedule jobs, provide submission limits, and implement fair share as well as provide the Quality of Service (QoS) levels for research groups who have invested in the cluster.

...

Type

Series

Arch

Count

Sockets

Cores

Threads / Core

Clock (GHz)

RAM (GB)

GPU Type

GPU Count

Local Disk Type

Local Disk Capacity (GB)

IB Network

Operating System

Status

Teton Regular

Intel Broadwell

x86_64

180

2

32

1

2.1

128

N/A

N/A

SSD

240

EDR

RHEL 7.6

Mo longer available for purchase

Intel Cascade Lake

x86_64

15

2

40

1

2.1

128

N/A

N/A

SSD

240

EDR

RHEL 7.6

Current Model

Teton BigMem GPU

Intel Broadwell

x86_64

8

2

32

1

2.1

512

NVIDIA P100 16G

2

SSD

240

EDR

RHEL 7.6

Mo longer available for purchase

Teton HugeMem

Intel Broadwell

x86_64

10

2

32

1

2.1

1024

N/A

N/A

SSD

240

EDR

RHEL 7.6

Mo longer available for purchase

Teton KNL

Intel Knights Landing

x86_64

12

1

72

4

1.5

384 + 16

N/A

N/A

SSD

240

EDR

RHEL 7.6

Mo longer available for purchase

Teton DGX

Intel Broadwell

x86_64

1

2

40

2

2.2

512

NVIDIA V100 32G

8

SSD

7 TB

EDR

Ubuntu 18.04.2 LTS

Available as special order

Moran Regular

Intel Sandbridge/Ivybridge

x86_64

283

2

16

1

2.6

64 or 128

k20 on some

2

HD

1T

FDR

RHEL 7.6

Mo longer available for purchase

Moran BigMem

Intel Sandbridge/Ivybridge

x86_64

2

2

16

1

2.6

512

K80

8

HD

1T

FDR

RHEL 7.6

Mo longer available for purchase

Moran Debug

Intel Sandbridge/Ivybridge

x86_64

2

2

16

1

2.6

64

k20m

2

HD

1T

FDR

RHEL 7.6

Mo longer available for purchase

Moran HugeMem

Intel Sandbridge/Ivybridge

x86_64

2

2

16

1

2.6

1024

k20

2

HD

1T

FDR

RHEL 7.6

Mo longer available for purchase

Moran DGX

Intel Broadwell

x86_64

1

2

40

2

2.2

512

NVIDIA V100 16G

8

SSD

7 TB

EDR

Ubuntu 18.04.2 LTS

Available as special order

TOTAL Nodes

516


Global Filesystems

...

The Teton global parallel filesystem configured with a 160 TB SSD tier for active data and 1.2 PB HDD capacity tier for less-used data. The system policy engine moves data automatically between pools (disks and tiers). The system will automatically migrate data to HDD when the SSD tier reaches 70% used capacity. Teton has several spaces that are available for users to access described in the table below.

...

  • Project PIs can purchase additional scratch and/or project space at a cost of $100 / TB / year.

  • Additionally, PIs can request allocation increases at no cost for scratch and/or project space by submitting proposals that must be renewed when substantial cluster or storage changes occur:

    • the scientific gain and insights that will be or have been obtained by using the system,

    • how data is organized and accessed in efforts to maximize performance and usage.

    • Projects are limited to 1 no-cost increase.

  • To request more information, please contact ARCC.

Special Filesystems

...

Certain filesystems exist on different nodes of the cluster where specialized requirements exist. The table below summarizes these specialized filesystems.

...

The Bighorn filesystems will be provided for a limited amount of time in order for researchers to move data to either the petaLibrary, Teton storage or to some other storage media. The actual date that these mounts will be removed is still TBD.

Project and Account Requests

...

For research projects, UWYO faculty members (Principal Investigators) can request a Project be created on Teton. PIs can then add access to the project for UWYO students, faculty and external collaborators. User Accounts on Teton require a valid UWYO e-mail address and a UWYO-Affiliated PI sponsor. UWYO faculty members can sponsor their own accounts, while students, post-doctoral researchers, or research associates must use their PI as their sponsor. Non-UWYO external collaborators must be sponsored by a current UWYO faculty member.

...

To request access for instructional use, send an email to arcc-info@uwyo.edu with the course number, section, and student list. If the PI prefers generic accounts can be created instead of providing a student list. Instructional accounts are usually valid for a single semester and access to the project is terminated at the beginning of the next semester.

System Access

...

SSH Access

Teton has login nodes for users to access the cluster. Login nodes are available publicly using the hostname teton.arcc.uwyo.edu or teton.uwyo.edu. SSH can be done natively on MacOS or Linux based operating systems using the terminal and the ssh command. Although X11 forwarding is supported, and if you need graphical support, we recommend using FastX if at all possible. Additionally, you may want to configure your OpenSSH client to support connection multiplexing if you require multiple terminal sessions. For those instances where you have unreliable network connectivity, you may want to use either tmux or screen once you login to keep sessions alive during disconnects. This will allow you to later reconnect to these sessions.

...

Shell

Path

Version

Notes

bash

/bin/bash

4.2.46

Recommended

zsh

/bin/zsh

5.0.2

csh

/bin/csh

6.18.01

Implemented by TCSH

tcsh

/bin/tcsh

6.18.01

GPUs and Accelerators

...

The ARCC Teton cluster has a number of compute nodes that contain GPUs. This section describes the hardware, as well as access and usage of the GPU nodes.

...

The following tables list each node that has GPUs and the type of GPU installed.

Table #1

Expand
titleTable #1

Node

Partition

GPU Type

Number of Devices

GPU Memory Size (GB)

Compute Capability

GRES Flag

Teton Partition

Notes

m025

moran

K20m

2

4

3.5

gpu:k20m:{1-2}

Yes

m026

moran

K20m

2

4

3.5

gpu:k20m:{1-2}

Yes

m027

moran

K20m

2

4

3.5

gpu:k20m:{1-2}

Yes

m028

moran

K20m

2

4

3.5

gpu:k20m:{1-2}

Yes

m029

moran

K20m

2

4

3.5

gpu:k20m:{1-2}

Yes

m030

moran

K20m

2

4

3.5

gpu:k20m:{1-2}

Yes

m031

moran

K20m

2

4

3.5

gpu:k20m:{1-2}

Yes

m032

moran

K20m

2

4

3.5

gpu:k20m:{1-2}

Yes

m075

moran

K20m

2

4

3.5

gpu:k20m:{1-2}

Yes

m076

moran

K20m

2

4

3.5

gpu:k20m:{1-2}

Yes

m077

moran

K20m

2

4

3.5

gpu:k20m:{1-2}

Yes

m078

moran

K20m

2

4

3.5

gpu:k20m:{1-2}

Yes

m079

moran

K20m

2

4

3.5

gpu:k20m:{1-2}

Yes

m080

moran

K20m

2

4

3.5

gpu:k20m:{1-2}

Yes

m081

moran

K20m

2

4

3.5

gpu:k20m:{1-2}

Yes

m082

moran

K20m

2

4

3.5

gpu:k20m:{1-2}

Yes

m083

moran

K20m

2

4

3.5

gpu:k20m:{1-2}

Yes

Disabled due to memory ECC errors

m084

moran

K20m

2

4

3.5

gpu:k20m:{1-2}

Yes

Disabled due to memory ECC errors

m085

moran

K20m

2

4

3.5

gpu:k20m:{1-2}

Yes

Disabled due to memory ECC errors

m086

moran

K20m

2

4

3.5

gpu:k20m:{1-2}

Yes

m219

moran

K20Xm

2

5

3.5

gpu:k20xm:{1-2}

Yes

m220

moran

K20Xm

2

5

3.5

gpu:k20xm:{1-2}

Yes

m227

moran

K20Xm

2

5

3.5

gpu:k20xm:{1-2}

Yes

m228

moran

K20Xm

2

5

3.5

gpu:k20xm:{1-2}

Yes

m235

moran

K20Xm

2

5

3.5

gpu:k20xm:{1-2}

Yes

m236

moran

K20Xm

2

5

3.5

gpu:k20xm:{1-2}

Yes

m243

moran

K20Xm

2

5

3.5

gpu:k20xm:{1-2}

Yes

m244

moran

20Xm

2

5

3.5

gpu:k20xm:{1-2}

Yes

m251

moran

K20Xm

2

5

3.5

gpu:k20xm:{1-2}

Yes

m252

moran

K20Xm

2

5

3.5

gpu:k20xm:{1-2}

Yes

m259

moran

K20Xm

2

5

3.5

gpu:k20xm:{1-2}

Yes

m260

moran

K20Xm

2

5

3.5

gpu:k20xm:{1-2}

Yes

m267

moran

K20Xm

2

5

3.5

gpu:k20xm:{1-2}

Yes

m268

moran

K20Xm

2

5

3.5

gpu:k20xm:{1-2}

Yes

mdbg01

moran

GTX Titan X

1

12

5.2

gpu:TitanX:{1-1}

Yes

moran

GTX Titan

2

6

6.0

gpu:Titan:{1-2}

Yes

mdbg02

moran

K40c

2

11

3.5

gpu:k40c:{1-2}

Yes

moran

GTX Titan X

2

12

5.2

gpu:TitanX:{1-2}

Yes

mbm01

moran-gpu

K80

8

11

3.7

gpu:k80:{1-8}

No

mbm02

moran-gpu

K80

8

11

3.7

gpu:k80:{1-8}

No

tbm03

teton-gpu

Tesla P100

2

16

6.0

gpu:P100:{1-2}

No

tbm04

teton-gpu

Tesla P100

2

16

6.0

gpu:P100:{1-2}

No

tbm05

teton-gpu

Tesla P100

2

16

6.0

gpu:P100:{1-2}

No

tbm06

teton-gpu

Tesla P100

2

16

6.0

gpu:P100:{1-2}

No

tbm07

teton-gpu

Tesla P100

2

16

6.0

gpu:P100:{1-2}

No

tbm08

teton-gpu

Tesla P100

2

16

6.0

gpu:P100:{1-2}

No

tbm09

teton-gpu

Tesla P100

2

16

6.0

gpu:P100:{1-2}

No

tbm10

teton-gpu

Tesla P100

2

16

6.0

gpu:P100:{1-2}

No

The following two GPU nodes are reserved for AI use.

...