Versions Compared

Old Version 7

changes.mady.by.user Simon Alexander

Saved on

compared with

New Version 8

changes.mady.by.user Lisa Stafford

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Table of Contents
minLevel1
maxLevel1
outlinefalse
stylenone
typelist
printabletrue

...

What do we have available?

  • Compilers and eco systems: GNU family, Intel’s oneAPI, Nvidia’s hpc-sdk

  • Languages: C/C+, Fortran, Go, Java, Julia, Perl, Python, R, Ruby, Rust

  • Scientific libraries and toolkits: Built with a specific compiler: GNU by default

  • Standalone applications and utilities: Installed using:

    • Conda

    • Containers: running Singularity (not Docker)

      • We can create a Singularity image from a Docker image.

    • Binaries/Executables

Beartooth Software List

...

Setting up your environment 

...

Exercises

  • Environment variables: env, echo

  • Find and load modules.

  • Commands: module spider/load/purge/ml

  • Understand dependencies and compiler stacks.

  • Run our python script using a specific version of python.

  • Load R (and Python).

  • Default modules – why define the module version.

...

Environment Variables

Code Block
[]$ env

[]$ echo $PATH

...

What’s being used?

Code Block
[]$ which python
/usr/bin/python

...

Context: Run a Python Script on the System

Copy files:

Code Block
[]$ cd 
[~]$ cp -r /project/arccanetrain/intro_to_hpc/ .
[~]$ cd intro_to_hpc/
[intro_to_hpc]$ ls
Intro_to_hpc.pdf  python01.py  python01.py.fixed  run_gpu.sh  run.sh

...

Let's run a python script:

Code Block
# If you have NOT already copied the files.
# Navigate back to your home folder.
[]$ cd
[~]$ mkdir intro_to_hpc
[~]$ cd intro_to_hpc/
[intro_to_hpc]$ ls
[intro_to_hpc]$ pwd
/home/<username>/intro_to_hpc
[intro_to_hpc]$ vim python01.py
"python01.py" [New File] 
# Using vim: Press ESC followed by ‘i’ to INSERT: Start typing:
import sys
print("Python version: " + sys.version)
print("Version info: " + sys.version_info)
# Using vim: ESC followed by ‘:wq’, then Return

...

Let's run a python script: Fixed:

Code Block
[]$ python python01.py
Python version: 3.8.16 (default, May 31 2023, 12:44:21)
[GCC 8.5.0 20210514 (Red Hat 8.5.0-18)]
Traceback (most recent call last):
  File "python01.py", line 3, in <module>
    print("Version info: " + sys.version_info)
TypeError: can only concatenate str (not "sys.version_info") to str
# Let's update the code:
From: print("Version info: " + sys.version_info)
To:   print("Version info: " + str(sys.version_info))
[]$ python python01.py
Python version: 3.8.17 (default, Aug 10 2023, 12:50:17)
[GCC 8.5.0 20210514 (Red Hat 8.5.0-20)]
Version info: sys.version_info(major=3, minor=8, micro=17, releaselevel='final', serial=0)
[]$ python --version
Python 3.8.17

...

You can’t break the System

Code Block
[]$ ls /usr/bin

[]$ cd /usr/bin
[bin]$ ls
[]$ pwd
/usr/bin

# Permissions (ugo:rwx) and Ownership (user:group)
[bin]$ ls -al
[bin]$ ls -al python*

You can't break the system.
[bin]$ rm python
rm: cannot remove 'python': Permission denied

...

Warning

sudo: you will not be granted sudo access – do not ask!

...

What’s available?

Code Block
[bin]$ cd ~
[~]$ cd intro_to_hpc/

[]$ ml
[]$ module avail

[]$ module load gcc/12.2.0
[]$ ml
[]$ module avail

# What is different compared to the first time we called ml?


[]$ module load gcc/11.2.0
# What happened?

...

What’s available? Compiler tree

Code Block
[]$ module load gcc/11.2.0
# What happened?
Due to MODULEPATH changes, the following have been reloaded:
  1) gmp/6.2.1     2) mpfr/4.1.0     3) zlib/1.2.12     4) zstd/1.5.2

The following have been reloaded with a version change:
  1) gcc/12.2.0 => gcc/11.2.0

# You can only have one compiler loaded at a time within a session.

[]$ ml
[]$ module avail

[]$ module purge
[]$ ml
# What do you notice?

...

What’s available? Modules loaded by default

Code Block
[]$ module purge
The following modules were not unloaded:
  (Use "module --force purge" to unload all):
  1) slurm/latest   2) arcc/1.0


[]$ ml
Currently Loaded Modules:
  1) slurm/latest (S)   2) arcc/1.0 (S)
  Where:
   S:  Module is Sticky, requires --force to unload or purge

...

Module load/spider: Dependencies

Code Block
[]$ module load python/3.10.6
Lmod has detected the following error:  These module(s) or extension(s) exist but cannot be loaded as
requested: "python/3.10.6"
   Try: "module spider python/3.10.6" to see how to load the module(s).

...

What’s different between these command-lines?

Code Block
[]$ module spider python/3.10.6

[]$ module spider python/3.10.8

...

What’s different between these command-lines? Dependencies

Code Block
[]$ module spider python/3.10.6
----------------------------------------
  python: python/3.10.6
----------------------------------------
    You will need to load all module(s) on any one of the lines below before the "python/3.10.6" module is available to load.
      arcc/1.0  gcc/12.2.0
    Help:
      The Python programming language.


[]$ module spider python/3.10.8
----------------------------------------
  python: python/3.10.8
----------------------------------------
    You will need to load all module(s) on any one of the lines below before the "python/3.10.8" module is available to load.
      arcc/1.0  gcc/11.2.0
    Help:
      The Python programming language.

...

Setup Python environment

Code Block
[]$ module purge
[]$ module load gcc/12.2.0
[]$ module load python/3.10.6
[]$ python –version


# Single line:
# Order matters:
[]$ module purge
[]$ module load python/3.10.6 gcc/12.2.0
vs
[]$ module load gcc/12.2.0 python/3.10.6

...

What’s happened to the PATH environment variable?

Code Block
[]$ module purge
[]$ echo $PATH


[]$ module load gcc/12.2.0 python/3.10.6
[]$ echo $PATH
[]$ which python
/apps/u/spack/gcc/12.2.0/python/3.10.6-7ginwsd/bin/python

...

Can we use the R language?

Code Block
[]$ module purge
[]$ r
[]$ R

# Can we find an 'R' module?
[]$ module avail
[]$ module spider r

# What do we see and why?
[]$ module load r/4.2.2

# How do we fix it?

...

Can we use the R language? Fixed

Code Block
[]$ module purge
[]$ r
[]$ R

# Can we find an 'R' module?
[]$ module avail
[]$ module spider r

# What do we see and why?
[]$ module load r/4.2.2

# How do we fix it?
[]$ module load gcc/12.2.0 r/4.2.2
[]$ ml
[]$ R --version

...

Can we also use the Python language?

Code Block
[]$ module load python/3.10.6
[]$ python --version
Python 3.10.6

# Where’s the gcc/12.2.0?

# What happens if we:
[]$ module load gcc/11.2.0 python/3.10.8
[]$ python –-version
[]$ R --version

...

  • Only one compiler/version can be loaded into your environment at a time.

  • Can only load languages/applications built with the same compiler. 

  • But, even this can introduce dependency issues.

...

Defaults

Code Block
[]$ module purge
[]$ module avail
[]$ module load python
[]$ python --version
Python 2.7.18 :: Anaconda, Inc.

...

Defaults: Why Consider?

Modules change and are updated.

...

Helps to replicate and triage.

...

Advanced: Why do versions matter?

Consider Python Packages installed using pip:

...

It is the same concept for R libraries.

...

Summary

  • LMOD: Setting up an Environment.

...