Python Pip Installs on the Cluster
Goal: Understand where pip installs packages within a user’s home folder with respect to different versions of Python.
- 1 Python on the System
- 2 Python as a Module
- 3 Pip Install Numpy - Python 3.10.6
- 4 Use Python 3.12.0 - Is numpy available?
- 5 Python System Configuration
- 6 What is the Installation Scheme?
- 7 What is under the userbase?
- 8 Why is they no lib/python3.12?
- 9 Pip Install Numpy - Python 3.12.0
- 10 What has changed under the .local/bin folder?
- 11 Remember: Pip Installs and Package Locations
Python on the System
Python is available on the System (but not necessarily on the compute nodes due to keeping node images lite).
[salexan5@mblog2 ~]$ python --version
Python 3.9.18
[salexan5@mblog1 ~]$ which python
/usr/bin/python
Note:
Version might update during cluster maintenance.
Pip may/may note be installed.
Recommended to not use to due potential updates.
Python as a Module
Different versions are available via loading a module
[]$ module spider python
----------------------------------------------------------------------------
python:
----------------------------------------------------------------------------
Versions:
python/3.10.6
python/3.12.0
Other possible modules matches:
python2
[]$ module purge
[]$ module load gcc/14.2.0 python/3.10.6
[]$ python --version
Python 3.10.6
[]$ which pip
/apps/u/spack/gcc/14.2.0/python/3.10.6-6lvrsdd/bin/pip
Note:
Recommended to use a module version for reproducibility.
Pip will be available.
Pip Install Numpy - Python 3.10.6
Let’s pip install numpy
and then run a test to verify that it works.
# Using python/3.10.6
[]$ pip install numpy
Defaulting to user installation because normal site-packages is not writeable
Collecting numpy
Downloading numpy-2.0.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (60 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 60.9/60.9 kB 2.0 MB/s eta 0:00:00
Downloading numpy-2.0.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (19.3 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 19.3/19.3 MB 131.3 MB/s eta 0:00:00
Installing collected packages: numpy
Successfully installed numpy-2.0.0
[notice] A new release of pip is available: 24.0 -> 24.1
[notice] To update, run: pip install --upgrade pip
Use Python 3.12.0 - Is numpy available?
Python System Configuration
sysconfig: The sysconfig
module provides access to Python’s configuration information like the list of installation paths and the configuration variables relevant for the current platform.
What is the Installation Scheme?
Prefix scheme: The “prefix scheme” is useful when you wish to use one Python installation to perform the build/install (i.e., to run the setup script), but install modules into the third-party module directory of a different Python installation (or something that looks like a different Python installation). If this sounds a trifle unusual, it is—that’s why the user and home schemes come before. However, there are at least two known cases where the prefix scheme will be useful.
posix_prefix:
Path | Installation directory |
---|---|
stdlib |
|
scripts |
|
data |
|
What is under the userbase?
Why is they no lib/python3.12?
Pip Install Numpy - Python 3.12.0
What has changed under the .local/bin folder?
Remember: Pip Installs and Package Locations