Goal: Understand where pip installs packages within a user’s home folder with respect to different versions of Python.

Python on the System

Python is available on the System (but not necessarily on the compute nodes due to keeping node images lite).

[salexan5@mblog2 ~]$ python --version
Python 3.9.18

[salexan5@mblog1 ~]$ which python
/usr/bin/python

Note:

Version might update during cluster maintenance.
Pip may/may note be installed.
Recommended to not use to due potential updates.

Python as a Module

Different versions are available via loading a module

[]$ module spider python
----------------------------------------------------------------------------
  python:
----------------------------------------------------------------------------
     Versions:
        python/3.10.6
        python/3.12.0
     Other possible modules matches:
        python2
        
[]$ module load gcc/13.2.0 python/3.10.6
[]$ python --version
Python 3.10.6
[]$ which pip
/apps/u/spack/gcc/13.2.0/python/3.10.6-jh3qs5v/bin/pip

Note:

Recommended to use a module version for reproducibility.
Pip will be available.

Pip Install Numpy - Python 3.10.6

Let’s pip install numpy and then run a test to verify that it works.

# Using python/3.10.6
[]$ pip install numpy
Defaulting to user installation because normal site-packages is not writeable
Collecting numpy
  Downloading numpy-2.0.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (60 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 60.9/60.9 kB 2.0 MB/s eta 0:00:00
Downloading numpy-2.0.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (19.3 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 19.3/19.3 MB 131.3 MB/s eta 0:00:00
Installing collected packages: numpy
Successfully installed numpy-2.0.0

[notice] A new release of pip is available: 24.0 -> 24.1
[notice] To update, run: pip install --upgrade pip

py_test.py

import sys
import numpy
print("Python version: " + sys.version)
print("Numpy: " + str(numpy.version.version))

[]$ python py_test.py
Python: 3.10.6 (main, Apr 30 2024, 11:23:04) [GCC 13.2.0]
Numpy: 2.0.0

Use Python 3.12.0 - Is numpy available?

[]$ module purge
[]$ module load gcc/13.2.0 python/3.12.0
[]$ python --version
Python 3.12.0
[]$ which python
/apps/u/spack/gcc/13.2.0/python/3.12.0-ovfqpv2/bin/python

[]$ python py_test.py
Traceback (most recent call last):
  File "/cluster/medbow/home/<username>/py_test.py", line 2, in <module>
    import numpy
ModuleNotFoundError: No module named 'numpy

But didn’t we previously install the numpy package?

Python System Configuration

sysconfig: The sysconfig module provides access to Python’s configuration information like the list of installation paths and the configuration variables relevant for the current platform.

[]$ module purge
[]$ module load gcc/13.2.0 python/3.10.6
[]$ python -m sysconfig
Platform: "linux-x86_64"
Python version: "3.10"
Current installation scheme: "posix_prefix"
Paths:
        ...
		scripts = "/apps/u/spack/gcc/13.2.0/python/3.10.6-jh3qs5v/bin"
        stdlib = "/apps/u/spack/gcc/13.2.0/python/3.10.6-jh3qs5v/lib/python3.10"
...
Variables:
        ...   
        py_version = "3.10.6"
        ...
        py_version_short = "3.10"
        ...
        userbase = "/home/<username>/.local"   
# Full list has over 700 lines!

What is the Installation Scheme?

Prefix scheme: The “prefix scheme” is useful when you wish to use one Python installation to perform the build/install (i.e., to run the setup script), but install modules into the third-party module directory of a different Python installation (or something that looks like a different Python installation). If this sounds a trifle unusual, it is—that’s why the user and home schemes come before. However, there are at least two known cases where the prefix scheme will be useful.

posix_prefix:

Path	Installation directory
stdlib	`prefix/lib/pythonX.Y`
scripts	`prefix/bin`
data	`prefix`

The prefix will be the userbase - so in this case: /home/<username>/.local

What is under the userbase?

[~]$ ls .local
bin  lib  share

[]$ ls -al .local/bin
total 1
drwxr-xr-x 2 <username> <username> 4096 Jun 24 11:16 .
drwxr-xr-x 2 <username> <username> 4096 Jun 24 11:16 ..
-rwxr-xr-x 1 <username> <username> 257 Jun 24 11:16 f2py
-rwxr-xr-x 1 <username> <username> 257 Jun 24 11:16 numpy-config

[]$ ls .local/lib/
python3.10
[]$ ls .local/lib/python3.10/site-packages/
numpy  numpy-2.0.0.dist-info  numpy.libs

# ~/.local/lib/python3.10

Why is they no lib/python3.12?

Because you have not pip installed anything for this particular version.

When using python/3.12.0 it is looking under ~/.local/lib/python3.12 for pip installed packages.

Since we have not pip installed numpy using version 3.12.0, no package exists under this folder.

Pip Install Numpy - Python 3.12.0

[]$ module purge
[]$ module load gcc/13.2.0 python/3.12.0
[]$ pip install numpy==1.26.4
[]$ python py_test.py
Python: 3.12.0 (main, Apr 30 2024, 11:30:37) [GCC 13.2.0]
Numpy: 1.26.4

[~]$ ls .local/lib/
python3.10  python3.12

[~]$ ls .local/lib/python3.12/site-packages/
numpy  numpy-1.26.4.dist-info  numpy.libs

What has changed under the .local/bin folder?

# Previously when installing numpy for python/3.10.6
[~]$ ls -al .local/bin
total 1
-rwxr-xr-x 1 salexan5 salexan5  257 Jun 24 11:16 f2py
-rwxr-xr-x 1 salexan5 salexan5  257 Jun 24 11:16 numpy-config
[~]$ cat .local/bin/f2py
#!/apps/u/spack/gcc/13.2.0/python/3.10.6-jh3qs5v/bin/python
...

# Now after installing numpy for python/3.12.0
[~]$ ls -al .local/bin/
-rwxr-xr-x 1 salexan5 salexan5  257 Jun 24 11:45 f2py
-rwxr-xr-x 1 salexan5 salexan5  257 Jun 24 11:16 numpy-config
[~]$ cat .local/bin/f2py
#!/apps/u/spack/gcc/13.2.0/python/3.12.0-ovfqpv2/bin/python
...

The python/3.10.6 numpy related files have been overridden.

Workshop Home

Using Python, Conda and Pip on the Cluster

Next

Pip Install within a Conda Environment

Python Pip Installs on the Cluster