Goal: Understand how to load modules, reset your environment by purging, and potential dependency issues.
...
| Code Block |
|---|
[]$ module purge
[]$ module load gcc/1314.2.0
[]$ module load python/3.12.0
[]$ python --version |
...
| Code Block |
|---|
[]$ module purge
[]$ module load python/3.12.0 gcc/1314.2.0
vs
[]$ module purge
[]$ module load gcc/1314.2.0 python/3.12.0 |
| Info |
|---|
Order matters! Dependencies must be loaded first, so must be listed first, from left to right. |
...
| Code Block |
|---|
[]$ module purge
[]$ echo $PATH
/apps/s/arcc/1.0/bin:/apps/s/slurm/latest/bin:
/home/<username>/.local/bin:/home/<username>/bin:
/usr/share/Modules/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin
[]$ which python
/usr/bin/python |
...
| Code Block |
|---|
[]$ module load gcc/1314.2.0 python/3.12.0
[]$ echo $PATH
/apps/u/spack/gcc/1314.2.0/python/3.12.0-ovfqpv24e5he6r/bin:
/apps/u/spack/gcc/1314.2.0/util-linux-uuid/2.38.1-x54sns76f6zqay/bin:
...
/apps/u/spack/gcc/13.2.0/zstd/1.5.5-ibvf7gj/bin:
/apps/s/arcc/1.0/bin:
/apps/s/slurm/latest/bin:
/home/<username>/.local/bin:/home/<username>/bin:
/usr/share/Modules/bin:
/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin
[]$ which python
/apps/u/spack/gcc/13.2.0/python/3.12.0-ovfqpv2/bin/python |
...
| Info |
|---|
Remember: The order that the system will look within paths is from left to right. The /apps/u/spack/gcc/1314.2.0/python/3.12.0-ovfqpv24e5he6r/bin path is before the /usr/bin/ so it is picking up python/3.12.0 before the system’s version. |
...
| Info |
|---|
Notice from the module spider r/4.4.0 that this module has a dependency on gcc/1314.2.0. Until gcc/1314.2.0 has been loaded, we will not see r/4.4.0 being available. |
| Code Block |
|---|
[]$ module purge
[]$ module load gcc/1314.2.0 r/4.4.0 |
| Info |
|---|
Once r/4.4.0 has been loaded, how many additional dependencies/libraries have also been loaded? |
...
| Info |
|---|
We know we can only have one compiler loaded at a time. Can we have more than one language loaded at a time? First load r/4.4.0, and then python/133.212.0 - what do we notice? |
| Code Block |
|---|
[]$ module purge
[]$ module load gcc/1314.2.0 r/4.4.0
[]$ module load python/3.12.0
-------------------------------------------------------------------------------
The following dependent module(s) are not currently loaded: python/3.10.6 (required by: glib/2.78.0, xcb-proto/1.15.2, gobject-introspection/1.76.1)
-------------------------------------------------------------------------------
The following have been reloaded with a version change:
1) python/3.10.6 => python/3.12.0
[]$ R --version
R version 4.4.0 (2024-04-24) -- "Puppy Cup"
[]$ python --version
Python 3.12.0 |
...
| Code Block |
|---|
[]$ module purge
[]$ module load gcc/1314.2.0 r/4.4.0
[]$ R --version
R version 4.4.0 (2024-04-24) -- "Puppy Cup"
...
[]$ python --version
Python 3.10.6 |
...
| Code Block |
|---|
[]$ module load oneapi/2024.1.0
Lmod is automatically replacing "gcc/1314.2.0" with "oneapi/2024.1.0".
Inactive Modules:
1) cairo/1.16.0 30) libxcb/1.14
...
29) libxau/1.0.8 58) xtrans/1.4.0
Due to MODULEPATH changes, the following have been reloaded:
1) berkeley-db/18.1.40 6) libiconv/1.17 11) pigz/2.7 16) zlib-ng/2.1.4
...
5) libbsd/0.11.7 10) perl/5.38.0 15) xz/5.4.1
[]$ R --version
-bash: R: command not found
[]$ python --version
Python 3.9.18
[]$ which python
/usr/bin/python |
...
| Note |
|---|
Will changing the version of Python affect R? In this case probably not. But if underlying versions of libraries are changing - then maybe - and that’s the best we can say… Remember: The more complicated your environments, the more dependencies there’ll be, the more potential for dependency hell. |
...
Exercises:
| Note |
|---|
Try answers the following questions: Why do we need a module system? What modules are available that relate to netcdf? What modules become available after loading nvhpc-sdk/24.3? How would you identify modules that have no dependencies?
|
...
Exercises: Answers:
| Expand |
|---|
| title | Why do we need a module system? |
|---|
|
| Info |
|---|
To enable researchers to individually configure sessions with the compilers/libraries/applications they require for their specific workflows. |
|
| Expand |
|---|
| title | What modules are available that relate to netcdf? |
|---|
|
| Info |
|---|
Use the module spider command to search for modules. You will see something of the form: | Code Block |
|---|
[]$ module spider netcdf
----------------------------------------------------------------------------
netcdf-c: netcdf-c/4.9.2-ompi
----------------------------------------------------------------------------
You will need to load all module(s) on any one of the lines below before the "netcdf-c/4.9.2-ompi" module is available to load.
arcc/1.0 gcc/13.2.0 openmpi/4.1.6
Help:
NetCDF (network Common Data Form) is a set of software libraries and
machine-independent data formats that support the creation, access, and
sharing of array-oriented scientific data. This is the C distribution.
----------------------------------------------------------------------------
netcdf-cxx4: netcdf-cxx4/4.3.1-ompi
----------------------------------------------------------------------------
You will need to load all module(s) on any one of the lines below before the "netcdf-cxx4/4.3.1-ompi" module is available to load.
arcc/1.0 gcc/13.2.0 openmpi/4.1.6
Help:
NetCDF (network Common Data Form) is a set of software libraries and
machine-independent data formats that support the creation, access, and
sharing of array-oriented scientific data. This is the C++ distribution.
----------------------------------------------------------------------------
parallel-netcdf: parallel-netcdf/1.12.3-ompi
----------------------------------------------------------------------------
You will need to load all module(s) on any one of the lines below before the "parallel-netcdf/1.12.3-ompi" module is available to load.
arcc/1.0 gcc/13.2.0 openmpi/4.1.6
Help:
PnetCDF (Parallel netCDF) is a high-performance parallel I/O library for
accessing files in format compatibility with Unidata's NetCDF,
specifically the formats of CDF-1, 2, and 5. |
|
|
| Expand |
|---|
| title | What modules become available after loading nvhpc-sdk/24.3? |
|---|
|
| Info |
|---|
First, module load this compiler, and then use the module avail to see what additional modules are now available. | Code Block |
|---|
[]$ module purge
[]$ module load nvhpc-sdk/24.3
[]$ module avail
...
----------------- /apps/u/opt/compilers/nvhpc/24.3/modulefiles -----------------
nvhpc-byo-compiler/24.3 nvhpc-hpcx/24.3 nvhpc-openmpi3/24.3
nvhpc-hpcx-cuda12/24.3 nvhpc-nompi/24.3 nvhpc/24.3
... |
|
|
| Expand |
|---|
| title | How would you identify modules that have no dependencies? |
|---|
|
| Info |
|---|
Modules that are available from the containers |
|
...
and conda-envs and linux trees are typically installed with no dependencies. | Code Block |
|---|
[]$ module avail
...
-------------- /apps/s/lmod/mf/opt/linux-rhel9-x86_64/containers ---------------
|
|
|
...
...
...
8.0 regtools/1.0.0 stress-ng/0.17.08
...
-------------- /apps/s/lmod/mf/opt/linux-rhel9-x86_64/conda-envs ---------------
|
|
|
...
...
...
...
...
...
qiime2-amplicon/2024.5
bowtie2/2.5.4 |
|
|
...
multiqc/1.24.1 qiime2-metagenome/2024.5
|
|
|
...
...
...
...
...
...
...
...
...
| Note |
|---|
Modules that are based on Linux binaries might/might not have dependencies, but these will be automatically loaded. For example: | Code Block |
|---|
[]$ module avail
...
----------------- /apps/s/lmod/mf/opt/linux-rhel9-x86_64/linux -----------------
...
fastqc/0.12.1 matlab/2024a sratoolkit/3.1.1
gaussian/16.AVX2.b01 miniconda3/24.3.0 subread/2.0.6
gsutil/491.0.0 muscle/5.2 tophat/2.1.1
guppy-cpu/6.5.7 nextflow/23.10.1 trimmomatic/0.39
... |
matlab
| Code Block |
|---|
[powersw@mblog1 linux]$ module spider matlab/2024a
----------------------------------------------------------------------------
matlab: matlab/2024a
----------------------------------------------------------------------------
You will need to load all module(s) on any one of the lines below before the "matlab/2024a" module is available to load.
arcc/1.0 |
versus trimmomatic: | Code Block |
|---|
[]$ module spider trimmomatic/0.39
----------------------------------------------------------------------------
trimmomatic: trimmomatic/0.39
----------------------------------------------------------------------------
You will need to load all module(s) on any one of the lines below before the "trimmomatic/0.39" module is available to load.
arcc/1.0
Help:
trimmomativ : 0.39
A flexible read trimming tool for Illumina NGS data.
http://www.usadellab.org/cms/?page=trimmomatic
Loads dependencies: gcc/14.2.0 and openjdk/11.0.20.1_1 |
Notice that gcc/14.2.0 and openjdk/11.0.20.1_1 will be automatically loaded: | Code Block |
|---|
[]$ module purge
[]$ module load trimmomatic/0.39
[]$ gcc --version
gcc (Spack GCC) |
|
...
...
2.0
...
[]$ java -version
openjdk |
|
...
version "11.0.20.1" 2023-08-24
... |
|
...
Consider: How does this effect you environment and any existing loaded modules? Look at any warning messages. |
...
...