Goal: Understand how to load modules, reset your environment by purging, and potential dependency issues.
...
Info |
---|
Can we combine the two What is the difference, what happens, when we try the following two methods: |
...
Info |
---|
Order matters! Dependencies must be loaded first, so must be listed first, from left to right. |
...
Code Block |
---|
[]$ module load gcc/13.2.0 python/3.12.0 []$ echo $PATH /apps/u/spack/gcc/13.2.0/python/3.12.0-ovfqpv2/bin: /apps/u/spack/gcc/13.2.0/util-linux-uuid/2.38.1-x54sns7/bin: ... /apps/u/spack/gcc/13.2.0/sqlitezstd/31.435.25-vkxcwyjibvf7gj/bin: /apps/us/spack/gcc/13.2.0/openssl/3.1.3-shbr5xoarcc/1.0/bin: /apps/us/spackslurm/gcc/13.2.0/gettext/0.22.3-54ib6sn/bin:/apps/u/spack/gcc/13.2.0/tar/1.34-drux4fh/bin: latest/bin:/home/<username>/.local/bin:/home/<username>/bin:/usr/share/Modules/bin: /usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin []$ which python /apps/u/spack/gcc/13.2.0/pigzpython/23.12.70-6mr4kciovfqpv2/bin:/apps/u/spack/gcc/13.2.0/libxml2/2.10.3-5toq4pi/bin: /apps/u/spack/gcc/13.2.0/xz/5.4.1-uainak7/bin:/python |
Info |
---|
Loading modules updates a number of environment variables exposing the compiler/application/software we want to use. |
...
What’s Happening to My Environment?
Info |
---|
Loading modules does not change anything in your What is being updated are the environment variables within your current session. These are not persistent across other sessions. |
Info |
---|
You can have two or more separate/independent sessions opened at the same time, each with different modules loaded - but all still sharing the same file storage system. |
Info |
---|
Remember: The order that the system will look within paths is from left to right. The
path is before the |
...
Can we use the R language?
Info |
---|
Can we use R within an environment? |
Code Block |
---|
[]$ module purge
[]$ module avail |
Info |
---|
Do we see R available within the default list of available modules? No. Can we search for it? |
Code Block |
---|
[]$ module spider r |
Info |
---|
From what is returned by Can we load it? |
Code Block |
---|
[]$ module load r/4.4.0 |
Note |
---|
How do we fix this? |
...
Can we use the R language? Fixed
Info |
---|
Notice from the . Until |
Code Block |
---|
[]$ module purge []$ module load gcc/13.2.0/bzip2/1.0.8-7q5awx3/bin:/apps/u/spack/gcc/11.4.1/gcc/13.2.0-sxnvmb2/bin: /apps/u/spack/gcc/13.2.0/zstd/1.5.5-ibvf7gj/bin:/apps/s/arcc/1.0/bin: /apps/s/slurm/latest/bin:/home/<username>/.local/bin:/home/<username>/bin:/usr/share/Modules/bin: /usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin []$ which python /apps/u/spack/gcc/13.2.0/python/3.12.0-ovfqpv2/bin/python |
Info |
---|
Loading modules updates a number of environment variables exposing the compiler/application/software we want to use. |
What’s Happening to My Environment?
Info |
---|
Loading modules does not change anything in your What is being updated are the environment variables within your current session. These are not persistent across other sessions. You can have two or more sessions opened at the same time, each with different modules loaded - but all still sharing the same file storage system. |
Info |
---|
Remember, that the order that the system will look within paths is from left to right. The |
Can we use the R language?
Info |
---|
Can we use R within an environment? |
Code Block |
---|
[]$ module purge
[]$ module avail |
Info |
---|
Do we see R available within the default list of available modules? No. Can we search for it? |
Code Block |
---|
[]$ module spider r |
Info |
---|
From what is returned by module spider, can you see why it isn’t currently available? Can we load it? |
Code Block |
---|
[]$ module load r/4.4.0 |
Info |
---|
How do we fix this? |
Can we use the R language? Fixed
Info |
---|
The module Until this module has been loaded, we will not see |
Code Block |
---|
[]$ module purge
[]$ module load gcc/13.2.0 r/4.4.0 |
Info |
---|
Once |
Code Block |
---|
[]$ ml
Currently Loaded Modules:
1) slurm/latest (S) 42) libxau/1.0.8
...
40) libpthread-stubs/0.4 81) r/4.4.0
41) xproto/7.0.31 |
Info |
---|
A lot! The module system hides the loading of all these additional libraries. |
Info |
---|
Let’s check R is loaded: |
Code Block |
---|
[]$ R --version
R version 4.4.0 (2024-04-24) -- "Puppy Cup" |
Can we also use the Python language?
Info |
---|
We know we can only have one compiler loaded at a time. Can we have more than one language loaded at a time? First load |
Code Block |
---|
[]$ module purge []$ module load gcc/13.2.0 r/4.4.0 r/4.4.0 |
Info |
---|
Once |
Code Block |
---|
[]$ ml
Currently Loaded Modules:
1) slurm/latest (S) 42) libxau/1.0.8
...
40) libpthread-stubs/0.4 81) r/4.4.0
41) xproto/7.0.31 |
Info |
---|
A lot! The module system hides the loading of all these additional libraries. |
Info |
---|
Let’s check R is loaded: |
Code Block |
---|
[]$ R --version
R version 4.4.0 (2024-04-24) -- "Puppy Cup" |
...
Can we also use the Python language?
Info |
---|
We know we can only have one compiler loaded at a time. Can we have more than one language loaded at a time? First load |
Code Block |
---|
[]$ module purge
[]$ module load gcc/13.2.0 r/4.4.0
[]$ module load python/3.12.0
-------------------------------------------------------------------------------
The following dependent module(s) are not currently loaded: python/3.10.6 (required by: glib/2.78.0, xcb-proto/1.15.2, gobject-introspection/1.76.1)
-------------------------------------------------------------------------------
The following have been reloaded with a version change:
1) python/3.10.6 => python/3.12.0
[]$ R --version
R version 4.4.0 (2024-04-24) -- "Puppy Cup"
[]$ python --version
Python 3.12.0 |
Info |
---|
Notice: Loading R automatically loaded After loading a module, consider calling Loading |
Note |
---|
Question: Does replacing Potentially: We do not know how this is being used and thus can’t say definitively if it does/doesn’t affect the running of R. We would suggest airing on the side of caution and check everything is running as expected if you notice modules being replaced. |
...
Load Another Compiler:
Info |
---|
After loading R and Python, what happens if you replace the |
Code Block |
---|
[]$ module purge []$ module load pythongcc/313.122.0 ------------------------------------------------------------------------------- The following dependent module(s) are not currently loaded: python/3.10.6 (required by: glib/2.78.0, xcb-proto/1.15.2, gobject-introspection/1.76.1) ------------------------------------------------------------------------------- The following have been reloaded with a version changer/4.4.0 []$ R --version R version 4.4.0 (2024-04-24) -- "Puppy Cup" ... []$ python --version Python 3.10.6 |
Info |
---|
Load the Intel |
Code Block |
---|
[]$ module load oneapi/2024.1.0 Lmod is automatically replacing "gcc/13.2.0" with "oneapi/2024.1.0". Inactive Modules: 1) cairo/1.16.0 30) libxcb/1.14 ... 29) libxau/1.0.8 58) xtrans/1.4.0 Due to MODULEPATH changes, the following have been reloaded: 1) pythonberkeley-db/318.10.6 => python/3.12.0 []$ R --version R version 4.4.0 (2024-04-24) -- "Puppy Cup" []$ python --version Python 3.12.0 |
Info |
---|
Notice: Loading R automatically loaded After loading a module, consider calling Loading |
Note |
---|
Does replacing Potentially: We do not know how this is being used and thus can’t say definitively if it does/doesn’t affect the running of R. We would suggest airing on the side of caution and check everything is running as expected if you notice modules being replaced. |
Load Another Compiler:
Info |
---|
After loading R and Python, what happens if I replace the |
Code Block |
---|
[]$ module purge
[]$ module load gcc/13.2.0 r/4.4.0
[]$ R --version
R version 4.4.0 (2024-04-24) -- "Puppy Cup"
...
[]$ python --version
Python 3.10.6 |
Info |
---|
Load the Intel oneApi compiler and see what happens to your environment: |
Code Block |
---|
[]$ module load oneapi/2024.1.0 Lmod is automatically replacing "gcc/13.2.0" with "oneapi/2024.1.0". Inactive Modules: 1) cairo/1.16.0 30) libxcb/1.14 ... 29) libxau/1.0.81.40 6) libiconv/1.17 11) pigz/2.7 16) zlib-ng/2.1.4 ... 5) libbsd/0.11.7 10) perl/5.38.0 15) xz/5.4.1 []$ R --version -bash: R: command not found []$ python --version Python 3.9.18 []$ which python /usr/bin/python |
Info |
---|
Remember: You can only have one compiler loaded at a time. Replacing one compiler with another will affect modules that have the first compiler as a dependency. So, with the above, R has been made inactive and is no longer available, and Python has also been made inactive and reverts back to the System version. |
...
But R is still in my Module List?
Info |
---|
What can you see if you list the modules loaded? |
Code Block |
---|
[]$ ml Currently Loaded Modules: 1) slurm/latest (S) 8) xz/5.4.1 15) berkeley-db/18.1.40 ... 6) bzip2/1.0.8 13) gettext/0.22.3 20) libbsd/0.11.7 7) libiconv/1.17 14) pkgconf/1.9.5 Where: S: Module is Sticky, requires --force to unload or purge Inactive Modules: ... 13) python/3.10.6 42) libtirpc/1.3.3 ... 5829) xtranspixman/10.442.02 Due to MODULEPATH changes, the following have been reloaded: 158) berkeley-dbr/184.1.40 6) libiconv/1.17 11) pigz/2.7 16) zlib-ng/2.1.4 ... 5) libbsd/0.11.7 10) perl/5.38.0 15) xz/5.4.1 []$ R --version -bash: R: command not found []$ python --version Python 3.9.18 []$ which python /usr/bin/python |
Info |
---|
Remember: You can only have one compiler loaded at a time. Replacing one compiler with another will also remove and modules that have this first compiler as a dependency. So, with the above, R has been removed and is no longer available, and Python has been removed and reverted back to the System version. |
But R is still in my Module List?
Info |
---|
But, what can we see if we list the modules available? |
Code Block |
---|
[]$ ml
Currently Loaded Modules:
1) slurm/latest (S) 8) xz/5.4.1 15) berkeley-db/18.1.40
2) arcc/1.0 (S) 9) libxml2/2.10.3 16) readline/8.2
3) oneapi/2024.1.0 10) ncurses/6.4 17) gdbm/1.23
4) zlib-ng/2.1.4 11) pigz/2.7 18) perl/5.38.0
5) zstd/1.5.5 12) tar/1.34 19) libmd/1.0.4
6) bzip2/1.0.8 13) gettext/0.22.3 20) libbsd/0.11.7
7) libiconv/1.17 14) pkgconf/1.9.5
Where:
S: Module is Sticky, requires --force to unload or purge
Inactive Modules:
...
13) python/3.10.6 42) libtirpc/1.3.3
...
29) pixman/0.42.2 58) r/4.4.0 |
Note |
---|
Yes, both R and Python are listed. But these are listed as |
Info |
---|
We would suggest, as a good practice, always perform a |
Code Block |
---|
[]$ module purge
[]$ module load oneapi/2024.1.0
[]$ ml
Currently Loaded Modules:
1) slurm/latest (S) 2) arcc/1.0 (S) 3) oneapi/2024.1.0 |
Remember:
Info |
---|
|
Note |
---|
Will changing the version of Python affect R? In this case probably not. But if underlying versions of libraries are changing - then maybe - and that’s the best we can say… Remember: The more complicated your environments, the more dependencies there’ll be, the more potential for dependency hell4.0 |
Note |
---|
Yes, both R and Python are listed. But these are listed as |
Info |
---|
We would suggest, as a good practice, always perform a |
Code Block |
---|
[]$ module purge
[]$ module load oneapi/2024.1.0
[]$ ml
Currently Loaded Modules:
1) slurm/latest (S) 2) arcc/1.0 (S) 3) oneapi/2024.1.0 |
...
Remember:
Info |
---|
|
Note |
---|
Will changing the version of Python affect R? In this case probably not. But if underlying versions of libraries are changing - then maybe - and that’s the best we can say… Remember: The more complicated your environments, the more dependencies there’ll be, the more potential for dependency hell. |
...
Exercises:
Note |
---|
Try answers the following questions:
|
...
Exercises: Answers:
Expand | ||||
---|---|---|---|---|
| ||||
|
Expand | ||||
---|---|---|---|---|
| ||||
|
Expand | ||||
---|---|---|---|---|
| ||||
|
Info |
---|
Essentially, anything that hasn’t been built with a compiler. |
...
Prev | Workshop Home | Next |
...