Goal: Understand where pip installs packages within a user’s home folder with respect to different versions of Python.
Python on the System
Python is available on the System (but not necessarily on the compute nodes).
[salexan5@mblog2 ~]$ python --version Python 3.9.18 [salexan5@mblog1 ~]$ which python /usr/bin/python
Note:
Version might update during cluster maintenance.
Pip may/may note be installed.
Recommended to not use to due potential updates.
Python as a Module
Different versions are available via loading a module
[salexan5@mblog2 ~]$ module spider python
----------------------------------------------------------------------------
python:
----------------------------------------------------------------------------
Versions:
python/3.10.6
python/3.12.0
Other possible modules matches:
python2
[salexan5@mblog2 ~]$ module load gcc/13.2.0 python/3.10.6
[salexan5@mblog2 ~]$ python --version
Python 3.10.6
[salexan5@mblog2 ~]$ which pip
/apps/u/spack/gcc/13.2.0/python/3.10.6-jh3qs5v/bin/pip
Note:
Recommended to use a module version for reproducibility.
Pip will be available.
Pip Install Numpy - Python 3.10.6
# Using python/3.10.6
[salexan5@mblog2 ~]$ pip install numpy
Defaulting to user installation because normal site-packages is not writeable
Collecting numpy
Downloading numpy-2.0.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (60 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 60.9/60.9 kB 2.0 MB/s eta 0:00:00
Downloading numpy-2.0.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (19.3 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 19.3/19.3 MB 131.3 MB/s eta 0:00:00
Installing collected packages: numpy
Successfully installed numpy-2.0.0
[notice] A new release of pip is available: 24.0 -> 24.1
[notice] To update, run: pip install --upgrade pip
Use Python 3.12.0 - Is numpy available?
[salexan5@mblog2 ~]$ module purge
[salexan5@mblog2 ~]$ module load gcc/13.2.0 python/3.12.0
[salexan5@mblog2 ~]$ python --version
Python 3.12.0
[salexan5@mblog2 ~]$ which python
/apps/u/spack/gcc/13.2.0/python/3.12.0-ovfqpv2/bin/python
[salexan5@mblog2 ~]$ python py_test.py
Traceback (most recent call last):
File "/cluster/medbow/home/salexan5/py_test.py", line 2, in <module>
import numpy
ModuleNotFoundError: No module named 'numpy
But didn’t we previously install the numpy package?
Python System Configuration
sysconfig: The sysconfig module provides access to Python’s configuration information like the list of installation paths and the configuration variables relevant for the current platform.
[salexan5@mblog2 ~]$ module purge
[salexan5@mblog2 ~]$ module load gcc/13.2.0 python/3.10.6
[salexan5@mblog2 ~]$ python -m sysconfig
Platform: "linux-x86_64"
Python version: "3.10"
Current installation scheme: "posix_prefix"
Paths:
...
scripts = "/apps/u/spack/gcc/13.2.0/python/3.10.6-jh3qs5v/bin"
stdlib = "/apps/u/spack/gcc/13.2.0/python/3.10.6-jh3qs5v/lib/python3.10"
...
Variables:
...
py_version = "3.10.6"
...
py_version_short = "3.10"
...
userbase = "/home/salexan5/.local"
# Full list has over 700 lines!
What is the Installation Scheme?
Prefix scheme: The “prefix scheme” is useful when you wish to use one Python installation to perform the build/install (i.e., to run the setup script), but install modules into the third-party module directory of a different Python installation (or something that looks like a different Python installation). If this sounds a trifle unusual, it is—that’s why the user and home schemes come before. However, there are at least two known cases where the prefix scheme will be useful.
posix_prefix:
Path | Installation directory |
|---|---|
stdlib |
|
scripts |
|
data |
|
The prefix will be the userbase - so in this case: /home/<username>/.local
What is under the userbase?
[salexan5@mblog2 ~]$ ls .local bin lib share [salexan5@mblog2 ~]$ ls -al .local/bin total 1 drwxr-xr-x 2 salexan5 salexan5 4096 Jun 24 11:16 . drwxr-xr-x 2 salexan5 salexan5 4096 Jun 24 11:16 .. -rwxr-xr-x 1 salexan5 salexan5 257 Jun 24 11:16 f2py -rwxr-xr-x 1 salexan5 salexan5 257 Jun 24 11:16 numpy-config [salexan5@mblog2 ~]$ ls .local/lib/ python3.10 [salexan5@mblog2 ~]$ ls .local/lib/python3.10/site-packages/ numpy numpy-2.0.0.dist-info numpy.libs # ~/.local/lib/python3.10
Why is they no lib/python3.12?
Because you have not pip installed anything for this particular version.
When using python/3.12.0 it is looking under ~/.local/lib/python3.12 for pip installed packages.
Since we have not pip installed numpy using version 3.12.0, no package exists under this folder.
Pip Install Numpy - Python 3.12.0
[salexan5@mblog2 ~]$ module purge [salexan5@mblog2 ~]$ module load gcc/13.2.0 python/3.12.0 [salexan5@mblog2 ~]$ pip install numpy==1.26.4 [salexan5@mblog2 ~]$ python py_test.py Python: 3.12.0 (main, Apr 30 2024, 11:30:37) [GCC 13.2.0] Numpy: 1.26.4 [salexan5@mblog2 ~]$ ls .local/lib/ python3.10 python3.12 [salexan5@mblog2 ~]$ ls .local/lib/python3.12/site-packages/ numpy numpy-1.26.4.dist-info numpy.libs
What has changed under the .local/bin folder?
# Previously when installing numpy for python/3.10.6 [salexan5@mblog2 ~]$ ls -al .local/bin total 1 -rwxr-xr-x 1 salexan5 salexan5 257 Jun 24 11:16 f2py -rwxr-xr-x 1 salexan5 salexan5 257 Jun 24 11:16 numpy-config [salexan5@mblog2 ~]$ cat .local/bin/f2py #!/apps/u/spack/gcc/13.2.0/python/3.10.6-jh3qs5v/bin/python ... # Now after installing numpy for python/3.12.0 [salexan5@mblog2 ~]$ ls -al .local/bin/ -rwxr-xr-x 1 salexan5 salexan5 257 Jun 24 11:45 f2py -rwxr-xr-x 1 salexan5 salexan5 257 Jun 24 11:16 numpy-config [salexan5@mblog2 ~]$ cat .local/bin/f2py #!/apps/u/spack/gcc/13.2.0/python/3.12.0-ovfqpv2/bin/python ...
The python/3.10.6 numpy related files have been overridden.