Goal: Understand where pip installs packages within a user’s home folder with respect to different versions of Python.
Python on the System
Python is available on the System (but not necessarily on the compute nodes due to keeping node images lite).
[salexan5@mblog2 ~]$ python --version Python 3.9.18 [salexan5@mblog1 ~]$ which python /usr/bin/python
Note:
Version might update during cluster maintenance.
Pip may/may note be installed.
Recommended to not use to due potential updates.
Python as a Module
Different versions are available via loading a module
[salexan5@mblog2 ~]$ module spider python
----------------------------------------------------------------------------
python:
----------------------------------------------------------------------------
Versions:
python/3.10.6
python/3.12.0
Other possible modules matches:
python2
[salexan5@mblog2 ~]$ module load gcc/13.2.0 python/3.10.6
[salexan5@mblog2 ~]$ python --version
Python 3.10.6
[salexan5@mblog2 ~]$ which pip
/apps/u/spack/gcc/13.2.0/python/3.10.6-jh3qs5v/bin/pip
Note:
Recommended to use a module version for reproducibility.
Pip will be available.
Pip Install Numpy - Python 3.10.6
# Using python/3.10.6
[salexan5@mblog2 ~]$ pip install numpy
Defaulting to user installation because normal site-packages is not writeable
Collecting numpy
Downloading numpy-2.0.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (60 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 60.9/60.9 kB 2.0 MB/s eta 0:00:00
Downloading numpy-2.0.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (19.3 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 19.3/19.3 MB 131.3 MB/s eta 0:00:00
Installing collected packages: numpy
Successfully installed numpy-2.0.0
[notice] A new release of pip is available: 24.0 -> 24.1
[notice] To update, run: pip install --upgrade pip
Use Python 3.12.0 - Is numpy available?
[salexan5@mblog2 ~]$ module purge
[salexan5@mblog2 ~]$ module load gcc/13.2.0 python/3.12.0
[salexan5@mblog2 ~]$ python --version
Python 3.12.0
[salexan5@mblog2 ~]$ which python
/apps/u/spack/gcc/13.2.0/python/3.12.0-ovfqpv2/bin/python
[salexan5@mblog2 ~]$ python py_test.py
Traceback (most recent call last):
File "/cluster/medbow/home/salexan5/py_test.py", line 2, in <module>
import numpy
ModuleNotFoundError: No module named 'numpy
But didn’t we previously install the numpy package?
Python System Configuration
sysconfig: The sysconfig module provides access to Python’s configuration information like the list of installation paths and the configuration variables relevant for the current platform.
[salexan5@mblog2 ~]$ module purge
[salexan5@mblog2 ~]$ module load gcc/13.2.0 python/3.10.6
[salexan5@mblog2 ~]$ python -m sysconfig
Platform: "linux-x86_64"
Python version: "3.10"
Current installation scheme: "posix_prefix"
Paths:
...
scripts = "/apps/u/spack/gcc/13.2.0/python/3.10.6-jh3qs5v/bin"
stdlib = "/apps/u/spack/gcc/13.2.0/python/3.10.6-jh3qs5v/lib/python3.10"
...
Variables:
...
py_version = "3.10.6"
...
py_version_short = "3.10"
...
userbase = "/home/salexan5/.local"
# Full list has over 700 lines!
What is the Installation Scheme?
Prefix scheme: The “prefix scheme” is useful when you wish to use one Python installation to perform the build/install (i.e., to run the setup script), but install modules into the third-party module directory of a different Python installation (or something that looks like a different Python installation). If this sounds a trifle unusual, it is—that’s why the user and home schemes come before. However, there are at least two known cases where the prefix scheme will be useful.
posix_prefix:
Path | Installation directory |
|---|---|
stdlib |
|
scripts |
|
data |
|
The prefix will be the userbase - so in this case: /home/<username>/.local
What is under the userbase?
[salexan5@mblog2 ~]$ ls .local bin lib share [salexan5@mblog2 ~]$ ls -al .local/bin total 1 drwxr-xr-x 2 salexan5 salexan5 4096 Jun 24 11:16 . drwxr-xr-x 2 salexan5 salexan5 4096 Jun 24 11:16 .. -rwxr-xr-x 1 salexan5 salexan5 257 Jun 24 11:16 f2py -rwxr-xr-x 1 salexan5 salexan5 257 Jun 24 11:16 numpy-config [salexan5@mblog2 ~]$ ls .local/lib/ python3.10 [salexan5@mblog2 ~]$ ls .local/lib/python3.10/site-packages/ numpy numpy-2.0.0.dist-info numpy.libs # ~/.local/lib/python3.10
Why is they no lib/python3.12?
Because you have not pip installed anything for this particular version.
When using python/3.12.0 it is looking under ~/.local/lib/python3.12 for pip installed packages.
Since we have not pip installed numpy using version 3.12.0, no package exists under this folder.
Pip Install Numpy - Python 3.12.0
[salexan5@mblog2 ~]$ module purge [salexan5@mblog2 ~]$ module load gcc/13.2.0 python/3.12.0 [salexan5@mblog2 ~]$ pip install numpy==1.26.4 [salexan5@mblog2 ~]$ python py_test.py Python: 3.12.0 (main, Apr 30 2024, 11:30:37) [GCC 13.2.0] Numpy: 1.26.4 [salexan5@mblog2 ~]$ ls .local/lib/ python3.10 python3.12 [salexan5@mblog2 ~]$ ls .local/lib/python3.12/site-packages/ numpy numpy-1.26.4.dist-info numpy.libs
What has changed under the .local/bin folder?
# Previously when installing numpy for python/3.10.6 [salexan5@mblog2 ~]$ ls -al .local/bin total 1 -rwxr-xr-x 1 salexan5 salexan5 257 Jun 24 11:16 f2py -rwxr-xr-x 1 salexan5 salexan5 257 Jun 24 11:16 numpy-config [salexan5@mblog2 ~]$ cat .local/bin/f2py #!/apps/u/spack/gcc/13.2.0/python/3.10.6-jh3qs5v/bin/python ... # Now after installing numpy for python/3.12.0 [salexan5@mblog2 ~]$ ls -al .local/bin/ -rwxr-xr-x 1 salexan5 salexan5 257 Jun 24 11:45 f2py -rwxr-xr-x 1 salexan5 salexan5 257 Jun 24 11:16 numpy-config [salexan5@mblog2 ~]$ cat .local/bin/f2py #!/apps/u/spack/gcc/13.2.0/python/3.12.0-ovfqpv2/bin/python ...
The python/3.10.6 numpy related files have been overridden.