Goal: Introduce the LMOD Module System and the concept of dependencies.
Table of Contents | ||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|
|
...
What Have We Seen
Note |
---|
Although on the login nodes we do have a We can see that R is not available, nor most of the applications that researchers will require. There are:
We can not provide a system that provides everything all at once. |
Info |
---|
What we need is a method to enable users to find and select that they specifically need, and a method to allow them to configure and setup their sessions/environment to allow them to perform their analysis and research. |
...
LMOD: An Environment Module System
ARCC: Lmod – Software
...
What do we have available?
Info |
---|
|
Check Software Pages
Setting up your environment
...
What’s available?
Code Block |
---|
[bin]$ cd ~
[~]$ cd intro_to_hpc/
[]$ ml
[]$ module avail
[]$ module load gcc/12.2.0
[]$ ml
[]$ module avail
# What is different compared to the first time we called ml? Check the MedicineBow Software Page. |
...
What is Available?
Info |
---|
|
...
What’s available: Using the Module System
Info |
---|
Use |
Code Block |
---|
# Modules currently loaded.
[]$ module list
Currently Loaded Modules:
1) slurm/latest (S) 2) arcc/1.0 (S) |
Info |
---|
Use |
Code Block |
---|
# What's available within my 'current' environment. # With respect to what modules I 'currently' have loaded. []$ module load gcc/11.2.0 # What happened?avail ... |
Info |
---|
Notice: There are various categories of available software: containers, compilers, conda environments, Linux binaries… |
Info |
---|
Modules have a name and a version: |
Note |
---|
The two modules These make available various commands and scripts that ALL users require (such as Slurm commands). If you |
...
What’s available? Compiler tree
Info |
---|
Some software has to be built with respect to a compiler. By default we build using the GNU family of compilers for C/C++ and Fortran. |
Info |
---|
Some commands have shortcuts. Using |
Code Block |
---|
[]$ module purge []$ module load gcc/1113.2.0 #[]$ Whatml happened?Currently DueLoaded toModules: MODULEPATH changes, the following have been reloaded: 11) slurm/latest (S) 3) gmp/6.2.1 25) mpfrmpc/41.3.1.0 37) zlibzstd/1.25.125 42) zstdarcc/1.5.20 The following have been reloaded(S) with a version change:4) mpfr/4.2.0 16) gcczlib-ng/122.21.0 =>4 8) gcc/1113.2.0 # You can only have one compiler loaded at a time within a session. []$ ml []$ module avail []$ module purge []$ ml # What do you notice? |
What’s available? Modules loaded by default
Code Block |
---|
[]$ module purge
The following modules were not unloaded:
(Use "module --force purge" to unload all):
1) slurm/latest 2) arcc/1.0
|
Info |
---|
The There are libraries that need to be loaded for gcc to work. They are loaded automatically so you don’t have to. |
Code Block |
---|
[] module avail |
Info |
---|
Question What changed? We can now see all the modules available that have been compiled with |
Note |
---|
These are only available after the |
...
Change Compiler
Info |
---|
Let’s see what happens to our environment if we load a different compiler. |
Code Block |
---|
[]# module load nvhpc-sdk/24.3
Lmod is automatically replacing "gcc/13.2.0" with "nvhpc-sdk/24.3". |
Info |
---|
Again, what happened? |
Code Block |
---|
[]$ ml
Currently Loaded Modules:
1) slurm/latest (S) 2) arcc/1.0 (S) Where:
S: Module is Sticky, requires --force to unload or purge |
...
3) nvhpc-sdk/24.3
[]$ module avail |
Info |
---|
Only modules relating to |
Note |
---|
You can only have one module compiler loaded at a time. Compilers of the same name, but different versions ( |
...
Purge Your Modules
Info |
---|
Use the |
Code Block |
---|
[]$ module load python/3.10.6 Lmod has detected the following errorpurge The following modules were not unloaded: These module(s)Use or extension(s) exist but cannot be loaded as requested: "python/3.10.6" Try: "module spider python/3.10.6" to see how to load the module(s). |
...
"module --force purge" to unload all):
1) slurm/latest 2) arcc/1.0 |
Info |
---|
Again, what has changed? |
Code Block |
---|
[]$ module spider python/3.10.6 ml []$ module spider python/3.10.8 |
...
avail |
Info |
---|
We are back to having the two modules |
Note |
---|
The more modules you have loaded, the more dependencies you will have loaded and the more likely you are to forget what you have loaded… |
Note |
---|
As mentioned earlier, do not use the |
...
Looking for Modules: module spider
Info |
---|
|
Code Block |
---|
[]$ module spider python/3.10.6 openmpi ---------------------------------------------------------------------------- pythonopenmpi: pythonopenmpi/34.101.6 ---------------------------------------------------------------------------- Other possible modules matches: nvhpc-openmpi3 You will need to load all module(s) on any one of the lines below before the "pythonopenmpi/34.101.6" module is available to load. arcc/1.0 gcc/1213.2.0 Help: The Python programming language. ... |
...
Module load/spider: Dependencies
Info |
---|
If you |
Code Block |
---|
[]$ module spider pythonopenmpi/34.101.8 6 ---------------------------------------------------------------------------- pythonopenmpi: pythonopenmpi/34.101.8 6 ---------------------------------------------------------------------------- You will need to load all module(s) on any one of the lines below before the "pythonopenmpi/34.101.86" module is available to load. arcc/1.0 gcc/1113.2.0 Help: TheAn Pythonopen programming language. |
Setup Python environment
Code Block |
---|
[]$ module purge []$ module load gcc/12source Message Passing Interface implementation. ... |
Note |
---|
Try loading before loading : |
Code Block |
---|
[]$ module load pythonopenmpi/34.101.6 []$Lmod pythonhas –versiondetected #the Singlefollowing lineerror: # Order matters: []$ module purge []$ module load python/3.10.6 gcc/12.2.0 vs []$ module load gcc/12.2.0 python/3.10.6 |
...
These module(s) or extension(s) exist
but cannot be loaded as requested: "openmpi/4.1.6"
Try: "module spider openmpi/4.1.6" to see how to load the module(s). |
Info |
---|
Load the required dependencies, then re-try: |
Code Block |
---|
[]$ module purge []$ echo $PATHload gcc/13.2.0 []$ module load gccopenmpi/124.2.0 python/3.10.1.6 []$ echo $PATH |
Info |
---|
Let’s check everything has been loaded successfully: |
Code Block |
---|
[]$ which pythonmpicc /apps/u/spackopt/gcc/1213.2.0/pythonopenmpi/34.101.6-7ginwsd/bin/python |
Can we use the R language?
Code Block |
---|
mpicc []$ mpicc module--version purge []$ r []$ R # Can we find an 'R' module? []$ module avail []$ module spider r # What do we see and why? []$ module load r/4.2.2 # How do we fix it? |
...
gcc (Spack GCC) 13.2.0
Copyright (C) 2023 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. |
...
What Do You Notice?
Info |
---|
After loading |
Code Block |
---|
[]$ moduleml purge []$ r []$ R # Can we find an 'R' module? []$ module avail []$ module spider r # What do we see and why? []$ module load r/4.2.2 # How do we fix it? []$ module load gcc/12.2.0 r/4.2.2 []$ ml []$ R --version |
...
module avail |
Info |
---|
Notice we have opened up an additional set of modules: |
Code Block |
---|
---- /apps/s/lmod/mf/spack/linux-rhel9-x86_64/openmpi/4.1.6-ipshyxb/gcc/13.2.0 ----
adios2/2.9.2-ompi netcdf-cxx4/4.3.1-ompi
cgns/4.4.0-ompi netlib-scalapack/2.2.0-ompi
elpa/2023.05.001-ompi parallel-netcdf/1.12.3-ompi
... |
Info |
---|
Due to dependencies, there can be a hierarchy of core modules that need to be loaded to make other libraries and software available. For example, some libraries, such as Typically their version number will be prefixed with Try: |
...
What’s different between these command-lines?
Info |
---|
What do we notice about the required dependencies for the following versions of Python? |
Code Block |
---|
[]$ module spider python/3.10.6
[]$ module spider python/3.12.0
[]$ module spider python2 |
...
What’s different between these command-lines? Dependencies
Info |
---|
The two |
Code Block |
---|
[]$ module loadspider python/3.10.6 []$ module spider python --version Python 3.10.6 # Where’s the gcc/12.2.0? # What happens if we: []$ module load gcc/11.2.0 python/3.10.8 []$ python –-version []$ R --version |
Remember:
Only one compiler/version can be loaded into your environment at a time.
Can only load languages/applications built with the same compiler.
But, even this can introduce dependency issues.
...
...
/3.12.0
...
You will need to load all module(s) on any one of the lines below before the "python/3.12.0" module is available to load.
arcc/1.0 gcc/13.2.0
arcc/1.0 gcc/14.2.0
... |
Info |
---|
The |
Code Block |
---|
[]$ module spider python2
----------------------------------------------------------------------------
python2: python2/2.7.18
----------------------------------------------------------------------------
You will need to load all module(s) on any one of the lines below before the "python2/2.7.18" module is available to load.
arcc/1.0 |
...
More module commands
Info |
---|
The |
Code Block |
---|
[]$ module
Modules based on Lua: Version 8.7 2022-04-16 13:49 -05:00
by Robert McLay mclay@tacc.utexas.edu
module [options] sub-command [args ...]
Help sub-commands:
------------------
help prints this message
help module [...] print help message from module(s)
Loading/Unloading sub-commands:
-------------------------------
load | add module [...] load module(s)
... |
...
Prev | Workshop Home | Next |