Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Goal: Understand how to load modules, reset your environment by purging, and potential dependency issues.

...

Code Block
[]$ module load gcc/13.2.0 python/3.12.0
[]$ echo $PATH
/apps/u/spack/gcc/13.2.0/python/3.12.0-ovfqpv2/bin:
/apps/u/spack/gcc/13.2.0/util-linux-uuid/2.38.1-x54sns7/bin:
...
/apps/u/spack/gcc/13.2.0/sqlitezstd/31.435.25-vkxcwyjibvf7gj/bin:
/apps/us/spack/gcc/13.2.0/openssl/3.1.3-shbr5xoarcc/1.0/bin:
/apps/us/spackslurm/gcc/13.2.0/gettext/0.22.3-54ib6snlatest/bin:/home/<username>/.local/bin:/appshome/u<username>/spackbin:/gcc/13.2.0/tar/1.34-drux4fhusr/share/Modules/bin:
/appsusr/ulocal/spack/gcc/13.2.0/pigz/2.7-6mr4kcibin:/usr/bin:/appsusr/ulocal/spacksbin:/gcc/13.2.0/libxml2/2.10.3-5toq4pi/bin:usr/sbin

[]$ which python
/apps/u/spack/gcc/13.2.0/xzpython/53.412.10-uainak7ovfqpv2/bin:/apps/u/spack/gcc/13.2.0/libiconv/1.17-3dj22ny/bin:
/apps/u/spack/gcc/13.2.0/gdbm/1.23-idyayjv/bin:/python
Info

Loading modules updates a number of environment variables exposing the compiler/application/software we want to use.

...

What’s Happening to My Environment?

Info

Loading modules does not change anything in your /home nor /project storage space - nothing is being installed.

What is being updated are the environment variables within your current session. These are not persistent across other sessions.

Info

You can have two or more separate/independent sessions opened at the same time, each with different modules loaded - but all still sharing the same file storage system.

Info

Remember: The order that the system will look within paths is from left to right.

The /apps/u/spack/gcc/13.2.0/

readline

python/

8

3.12.

2

0-

j6zrtuz

ovfqpv2/bin

: /apps/u/spack/gcc/13.2.0/ncurses/6.4-y257enz/bin:/apps/u/spack/gcc/13.2.0/expat/2.5.0-3aw2urq/bin: /apps/u/spack/gcc/13.2.0/bzip2/1.0.8-7q5awx3/bin:/apps/u/spack/gcc/11.4.1/gcc/13.2.0-sxnvmb2/bin: /apps/u/spack/

path is before the /usr/bin/ so it is picking up python/3.12.0 before the system’s version.

...

Can we use the R language?

Info

Can we use R within an environment?

Code Block
[]$ module purge
[]$ module avail
Info

Do we see R available within the default list of available modules? No.

Can we search for it?

Code Block
[]$ module spider r
Info

From what is returned by module spider, can you see why it isn’t currently available?

Can we load it?

Code Block
[]$ module load r/4.4.0
Note

How do we fix this?

...

Can we use the R language? Fixed

Info

Notice from the module spider r/4.4.0 that this module has a dependency on gcc/13.2.0

/zstd/1.5.5-ibvf7gj/bin:/apps/s/arcc/1.0/bin: /apps/s/slurm/latest/bin:/home/<username>/.local/bin:/home/<username>/bin:/usr/share/Modules/bin: /usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin []$ which python /apps/u/spack/gcc/13.2.0/python/3.12.0-ovfqpv2/bin/python
Info

Loading modules updates a number of environment variables exposing the compiler/application/software we want to use.

What’s Happening to My Environment?

Info

Loading modules does not change anything in your /home nor /project storage space - nothing is being installed.

What is being updated are the environment variables within your current session. These are not persistent across other sessions.

You can have two or more sessions opened at the same time, each with different modules loaded - but all still sharing the same file storage system.

Info

Remember, that the order that the system will look within paths is from left to right.

The /apps/u/spack/.

Until gcc/13.2.0 has been loaded, we will not see r/4.4.0 being available.

Code Block
[]$ module purge
[]$ module load gcc/13.2.0 r/4.4.0
Info

Once r/4.4.0 has been loaded, how many additional dependencies/libraries have also been loaded?

Code Block
[]$ ml
Currently Loaded Modules:
  1) slurm/latest           (S)  42) libxau/1.0.8
...
 40) libpthread-stubs/0.4        81) r/4.4.0
 41) xproto/7.0.31
Info

A lot! The module system hides the loading of all these additional libraries.

Info

Let’s check R is loaded:

Code Block
[]$ R --version
R version 4.4.0 (2024-04-24) -- "Puppy Cup"

...

Can we also use the Python language?

Info

We know we can only have one compiler loaded at a time.

Can we have more than one language loaded at a time?

First load r/4.4.0, and then python/13.2.0 - what do we notice?

Code Block
Code Block
[]$ module purge
[]$ module load gcc/13.2.0 r/
python/3
4.
12.0-ovfqpv2/bin path is before the /usr/bin/ so it is picking up
4.0

[]$ module load python/3.12.0
before the system’s version.

Can we use the R language?

Info

Can we use R within an environment?

Code Block
[]$ module purge
[]$ module avail
Info

Do we see R available within the default list of available modules? No.

Can we search for it?

Code Block
[]$ module spider r
Info

From what is returned by module spider, can you see why it isn’t currently available?

Can we load it?

Code Block
[]$ module load r/4.4.0
Info

How do we fix this?

Can we use the R language? Fixed

Info

The module r/4.4.0 has a dependency on gcc/13.2.0.

Until this module has been loaded, we will not see r/4.4.0 being available.

Code Block
[]$ module purge
[]$ module load gcc/13.2.0 r/4.4.0
Info

Once r/4.4.0 has been loaded, how many additional dependencies/libraries have also been loaded?

Code Block
[]$ ml
Currently Loaded Modules:
  1) slurm/latest           (S)  42) libxau/1.0.8
...
 40) libpthread-stubs/0.4        81) r/4.4.0
 41) xproto/7.0.31
Info

A lot! The module system hides the loading of all these additional libraries.

Info

Let’s check R is loaded:

Code Block
[]$ R --version
R version 4.4.0 (2024-04-24) -- "Puppy Cup"

Can we also use the Python language?

Info

We know we can only have one compiler loaded at a time.

Can we have more than one language loaded at a time?

First load r/4.4.0, and then python/13.2.0 - what do we notice?

Code Block
[]$ module purge
[]$ module load gcc/13.2.0 r/4.4.0

[]$ module load python/3.12.0
-------------------------------------------------------------------------------
The following dependent module(s) are not currently loaded: python/3.10.6 (required by: glib/2.78.0, xcb-proto/1.15.2, gobject-introspection/1.76.1)
-------------------------------------------------------------------------------
The following have been reloaded with a version change:
  1) python/3.10.6 => python/3.12.0

[]$  R --version
R version 4.4.0 (2024-04-24) -- "Puppy Cup"
[]$ python --version
Python 3.12.0
Info

Notice: Loading R automatically loaded python/3.10.6. This is actually a dependency that was loaded during the loading of R.

After loading a module, consider calling ml to see what dependencies have also been loaded.

Loading python/3.12.0 replaced this version.

Note

Does replacing python/3.10.6 with python/3.12.0 affect R?

Potentially: We do not know how this is being used and thus can’t say definitively if it does/doesn’t affect the running of R.

We would suggest airing on the side of caution and check everything is running as expected if you notice modules being replaced.

Load Another Compiler:

Info

After loading R and Python, what happens if I replace the gcc compiler?

[]$ module purge
[]$ module load gcc/13.2.0 r/4.4.0
[]$
-------------------------------------------------------------------------------
The following dependent module(s) are not currently loaded: python/3.10.6 (required by: glib/2.78.0, xcb-proto/1.15.2, gobject-introspection/1.76.1)
-------------------------------------------------------------------------------
The following have been reloaded with a version change:
  1) python/3.10.6 => python/3.12.0

[]$  R --version
R version 4.4.0 (2024-04-24) -- "Puppy Cup"
...
[]$ python --version
Python 3.1012.60
Info

Load the Intel oneApi compiler and see what happens to your environment:

Code Block
[]$ module load oneapi/2024.1.0 Lmod is automatically replacing "gcc/13.2.0" with "oneapi/2024.1.0". Inactive Modules: 1) cairo/1.16.0 30) libxcb/1.14 ... 29) libxau/1.0.8 58) xtrans/1.4.0 Due to MODULEPATH changes, the following have been reloaded: 1) berkeley-db/18.1.40 6) libiconv/1.17 11) pigz/2.7 16) zlib-ng/2.1.4 ... 5) libbsd/0.11.7 10) perl/5.38

Notice: Loading R automatically loaded python/3.10.6. This is actually a dependency that was loaded during the loading of R.

After loading a module, consider calling ml to see what dependencies have also been loaded.

Loading python/3.12.0 replaced this version.

Note

Question: Does replacing python/3.10.6 with python/3.12.0 affect R?

Potentially: We do not know how this is being used and thus can’t say definitively if it does/doesn’t affect the running of R.

We would suggest airing on the side of caution and check everything is running as expected if you notice modules being replaced.

...

Load Another Compiler:

Info

After loading R and Python, what happens if you replace the gcc compiler?

Code Block
[]$ module purge
[]$ module load gcc/13.2.0 r/4.4.0
[]$ R --version
R version 4.4.0 (2024-04-24) -- "Puppy Cup"
...
[]$ python --version
Python 3.10.6
Info

Load the Intel oneapi compiler and see what happens to your environment:

Code Block
[]$ module load oneapi/2024.1.0
Lmod is automatically replacing "gcc/13.2.0" with "oneapi/2024.1.0".
Inactive Modules:
  1) cairo/1.16.0       15) xz/5.4.1  []$ R --version -bash: R: command not found  []$ python --version
Python 3.9.18
[]$ which python
/usr/bin/python
Info

Remember: You can only have one compiler loaded at a time. Replacing one compiler with another will also remove and modules that have this first compiler as a dependency.

So, with the above, R has been removed and is no longer available, and Python has been removed and reverted back to the System version.

But R is still in my Module List?

Info

But, what can we see if we list the modules available?

Code Block
[]$ ml
Currently Loaded Modules30) libxcb/1.14
...
 29) libxau/1.0.8                    58) xtrans/1.4.0
Due to MODULEPATH changes, the following have been reloaded:
  1) slurm/latestberkeley-db/18.1.40     (S)6) libiconv/1.17    8 11) xzpigz/5.4.12.7         1516) berkeleyzlib-dbng/182.1.4
...40
  25) arcclibbsd/10.11.07        (S)   910) libxml2perl/25.1038.30   16) readline/8.2   315) oneapixz/20245.4.1.0

[]$ R --version
-bash: R: 10) ncurses/6.4     17) gdbm/1.23
  4) zlib-ng/2.1.4        11) pigz/2.7        18) perl/5.38.0
  5) zstd/1.5.5           12) tar/1.34command not found

[]$ python --version
Python 3.9.18

[]$ which python
/usr/bin/python
Info

Remember: You can only have one compiler loaded at a time.

Replacing one compiler with another will affect modules that have the first compiler as a dependency.

So, with the above, R has been made inactive and is no longer available, and Python has also been made inactive and reverts back to the System version.

...

But R is still in my Module List?

Info

What can you see if you list the modules loaded?

Code Block
[]$ ml
Currently Loaded Modules:
  1) slurm/latest    (S)   8) xz/5.4.1        1915) libmdberkeley-db/18.1.40
..0.4
  6) bzip2/1.0.8          13) gettext/0.22.3  20) libbsd/0.11.7
  7) libiconv/1.17        14) pkgconf/1.9.5

  Where:
   S:  Module is Sticky, requires --force to unload or purge

Inactive Modules:
...
  13) python/3.10.6            42) libtirpc/1.3.3
...
  29) pixman/0.42.2            58) r/4.4.0
Note

Yes, both R and Python are listed.

But these are listed as Inactive Modules, so are no longer available active to use - and thus R can not be called.

Info

We would suggest, as a good practice, always perform a module purge before changing compilers and generate a clean environment.

...

Note

Will changing the version of Python affect R? In this case probably not.

But if underlying versions of libraries are changing - then maybe - and that’s the best we can say…

Remember: The more complicated your environments, the more dependencies there’ll be, the more potential for dependency hell.

...

...