Goal: Understand how to load modules, reset your environment by purging, and potential dependency issues.
Table of Contents | ||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|
|
...
Setup Python environment
Info |
---|
Let’s set up an environment to enable us to run Python scripts: |
Code Block |
---|
[]$ module purge
[]$ module load gcc/13.2.0
[]$ module load python/3.12.0
[]$ python –version
# Single line:
# Order matters:
|
Info |
---|
Can we combine the two What is the difference, what happens, when we try the following two methods: |
Code Block |
---|
[]$ module purge
[]$ module load python/3.12.0 gcc/13.2.0
vs
[]$ module purge
[]$ module load gcc/13.2.0 python/3.12.0 |
Info |
---|
Order matters! Dependencies must be loaded first, so must be listed first, from left to right. |
...
What’s happened to the PATH environment variable?
Info |
---|
What is happening to our environment when we’re loading modules? |
Code Block |
---|
[]$ module purge []$ echo $PATH []$ module load gcc/13.2.0 python/3.12.0 []$ echo $PATH /apps/us/spack/gcc/13.2.0/python/3.12.0-ovfqpv2arcc/1.0/bin:/apps/us/spackslurm/gcc/13.2.0/util-linux-uuid/2.38.1-x54sns7/bin: /apps/u/spack/gcc/13.2.0/sqlite/3.43.2-vkxcwyjlatest/bin:/home/<username>/.local/bin:/apps/u/spack/gcc/13.2.0/openssl/3.1.3-shbr5xohome/<username>/bin: /appsusr/ushare/spack/gcc/13.2.0/gettext/0.22.3-54ib6sn/Modules/bin:/apps/u/spack/gcc/13.2.0/tar/1.34-drux4fh/bin: /apps/u/spack/gcc/13.2.0/pigz/2.7-6mr4kci/bin:/apps/u/spack/gcc/13.2.0/libxml2/2.10.3-5toq4pi/bin: /apps/u/spack/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin []$ which python /usr/bin/python |
Info |
---|
Now lets load a version of Python and see what happens: |
Code Block |
---|
[]$ module load gcc/13.2.0 python/xz/53.4.1-uainak7/bin:/apps/u/spack/gcc/13.2.0/libiconv/1.17-3dj22ny/bin:12.0 []$ echo $PATH /apps/u/spack/gcc/13.2.0/gdbmpython/1.23-idyayjv/bin:/apps/u/spack/gcc/13.2.0/readline/8.2-j6zrtuz3.12.0-ovfqpv2/bin: /apps/u/spack/gcc/13.2.0/ncurses/6.4-y257enz/bin:/apps/u/spack/gcc/13.2.0/expat/2.5.0-3aw2urqutil-linux-uuid/2.38.1-x54sns7/bin: /apps/u/spack/gcc/13.2.0/bzip2/1.0.8-7q5awx3/bin:/apps/u/spack/gcc/11.4.1/gcc/13.2.0-sxnvmb2/bin: /apps/u/spack/gcc/13.2.0/zstd/1.5.5-ibvf7gj/bin: /apps/s/arcc/1.0/bin: /apps/s/slurm/latest/bin:/home/<username>/.local/bin:/home/<username>/bin:/usr/share/Modules/bin: /usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin []$ which python /apps/u/spack/gcc/13.2.0/python/3.12.0-ovfqpv2/bin/python |
Info |
---|
Loading modules updates a number of environment variables exposing the compiler/application/software we want to use. |
...
What’s Happening to My Environment?
Info |
---|
Loading modules does not change anything in your What is being updated are the environment variables within your current session. These are not persistent across other sessions. |
Info |
---|
Remember, that the You can have two or more separate/independent sessions opened at the same time, each with different modules loaded - but all still sharing the same file storage system. |
Info |
---|
Remember: The order that the system will look within paths is from left to right. The |
...
Can we use the R language?
Info |
---|
Can we use R within an environment? |
Code Block |
---|
[]$ module purge # Is R available? []$ module avail |
Info |
---|
Do we see R available within the default list of available modules? No. Can we search for it? |
Code Block |
---|
[]$ module spider r
# What do we see and why?
# Does this work?
|
Info |
---|
From what is returned by Can we load it? |
Code Block |
---|
[]$ module load r/4.4.0
# How do we fix it |
Note |
---|
How do we fix this? |
...
Can we use the R language? Fixed
Info |
---|
Notice from the Until |
Code Block |
---|
[]$ module purge
[]$ module load gcc/13.2.0 r/4.4.0 |
Info |
---|
Once |
Code Block |
---|
[]$ ml Currently Loaded Modules: 1) slurm/latest (S) 42) libxau/1.0.8 ... # How many dependencies have automatically been loaded? []$ ml 40) libpthread-stubs/0.4 81) r/4.4.0 41) xproto/7.0.31 |
Info |
---|
A lot! The module system hides the loading of all these additional libraries. |
Info |
---|
Let’s check R is loaded: |
Code Block |
---|
[]$ R --version R version 4.4.0 (2024-04-24) -- "Puppy Cup" |
...
Can we also use the Python language?
Info |
---|
We know we can only have one compiler loaded at a time. Can we have more than one language loaded at a time? First load |
Code Block |
---|
[]$ module purge
[]$ module load gcc/13.2.0 r/4.4.0
[]$ module load python/3.12.0
-------------------------------------------------------------------------------
The following dependent module(s) are not currently loaded: python/3.10.6 (required by: glib/2.78.0, xcb-proto/1.15.2, gobject-introspection/1.76.1)
-------------------------------------------------------------------------------
The following have been reloaded with a version change:
1) python/3.10.6 => python/3.12.0
[]$ R --version
R version 4.4.0 (2024-04-24) -- "Puppy Cup"
[]$ python --version
Python 3.12.0 |
Info |
---|
Notice: Loading R automatically loaded After loading a module, consider calling Loading |
Note |
---|
Question: Does replacing Potentially: We do not know how this is being used and thus can’t say definitively if it does/doesn’t affect the running of R. We would suggest airing on the side of caution and check everything is running as expected if you notice modules being replaced. |
...
Load Another Compiler:
Info |
---|
After loading R and Python, what happens if you replace the |
Code Block |
---|
[]$ module purge []$ module load gcc/13.2.0 r/4.4.0 []$ R --version R version 4.4.0 (2024-04-24) -- "Puppy Cup" ... []$ python --version Python 3.10.6 |
Info |
---|
Load the Intel |
Code Block |
---|
[]$ module load oneapi/2024.1.0 Lmod is automatically replacing "gcc/13.2.0" with "oneapi/2024.1.0". Inactive Modules: 1) cairo/1.16.0 30) libxcb/1.14 ... 29) libxau/1.0.8 58) xtrans/1.4.0 Due to MODULEPATH changes, the following have been reloaded: 1) berkeley-db/18.1.40 6) libiconv/1.17 11) pigz/2.7 16) zlib-ng/2.1.4 ... 5) libbsd/0.11.7 10) perl/5.38.0 15) xz/5.4.1 []$ R --version -bash: R: command not found []$ python --version Python 3.9.18 []$ which python /usr/bin/python |
Info |
---|
Remember: You can only have one compiler loaded at a time. Replacing one compiler with another will affect modules that have the first compiler as a dependency. So, with the above, R has been made inactive and is no longer available, and Python has also been made inactive and reverts back to the System version. |
...
But R is still in my Module List?
Info |
---|
What can you see if you list the modules loaded? |
Code Block |
---|
[]$ ml
Currently Loaded Modules:
1) slurm/latest (S) 8) xz/5.4.1 15) berkeley-db/18.1.40
...
6) bzip2/1.0.8 13) gettext/0.22.3 20) libbsd/0.11.7
7) libiconv/1.17 14) pkgconf/1.9.5
Where:
S: Module is Sticky, requires --force to unload or purge
Inactive Modules:
...
13) python/3.10.6 42) libtirpc/1.3.3
...
29) pixman/0.42.2 58) r/4.4.0 |
Note |
---|
Yes, both R and Python are listed. But these are listed as |
Info |
---|
We would suggest, as a good practice, always perform a |
Code Block |
---|
[]$ module purge
[]$ module load oneapi/2024.1.0
[]$ ml
Currently Loaded Modules:
1) slurm/latest (S) 2) arcc/1.0 (S) 3) oneapi/2024.1.0 |
...
Remember:
Info |
---|
|
Note |
---|
Will changing the version of Python affect R? In this case probably not. But if underlying versions of libraries are changing - then maybe - and that’s the best we can say… Remember: The more complicated your environments, the more dependencies there’ll be, the more potential for dependency hell. |
...
Exercises:
Note |
---|
Try answers the following questions:
|
...
Exercises: Answers:
Expand | ||||
---|---|---|---|---|
| ||||
|
Expand | ||||
---|---|---|---|---|
| ||||
|
Expand | ||||
---|---|---|---|---|
| ||||
|
Info |
---|
Essentially, anything that hasn’t been built with a compiler. |
...
Prev | Workshop Home | Next |
...