Python Pip Installs on the Cluster

Goal: Understand where pip installs packages within a user’s home folder with respect to different versions of Python.



Python on the System

Python is available on the System (but not necessarily on the compute nodes due to keeping node images lite).

[salexan5@mblog2 ~]$ python --version Python 3.9.18 [salexan5@mblog1 ~]$ which python /usr/bin/python

Note:

  • Version might update during cluster maintenance.

  • Pip may/may note be installed.

  • Recommended to not use to due potential updates.


Python as a Module

Different versions are available via loading a module

[]$ module spider python ---------------------------------------------------------------------------- python: ---------------------------------------------------------------------------- Versions: python/3.10.6 python/3.12.0 Other possible modules matches: python2 []$ module purge []$ module load gcc/14.2.0 python/3.10.6 []$ python --version Python 3.10.6 []$ which pip /apps/u/spack/gcc/14.2.0/python/3.10.6-6lvrsdd/bin/pip

Note:

  • Recommended to use a module version for reproducibility.

  • Pip will be available.


Pip Install Numpy - Python 3.10.6

Let’s pip install numpy and then run a test to verify that it works.

# Using python/3.10.6 []$ pip install numpy Defaulting to user installation because normal site-packages is not writeable Collecting numpy Downloading numpy-2.0.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (60 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 60.9/60.9 kB 2.0 MB/s eta 0:00:00 Downloading numpy-2.0.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (19.3 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 19.3/19.3 MB 131.3 MB/s eta 0:00:00 Installing collected packages: numpy Successfully installed numpy-2.0.0 [notice] A new release of pip is available: 24.0 -> 24.1 [notice] To update, run: pip install --upgrade pip

Use Python 3.12.0 - Is numpy available?


Python System Configuration

sysconfig: The sysconfig module provides access to Python’s configuration information like the list of installation paths and the configuration variables relevant for the current platform.


What is the Installation Scheme?

Prefix scheme: The “prefix scheme” is useful when you wish to use one Python installation to perform the build/install (i.e., to run the setup script), but install modules into the third-party module directory of a different Python installation (or something that looks like a different Python installation). If this sounds a trifle unusual, it is—that’s why the user and home schemes come before. However, there are at least two known cases where the prefix scheme will be useful.

posix_prefix:

Path

Installation directory

Path

Installation directory

stdlib

prefix/lib/pythonX.Y

scripts

prefix/bin

data

prefix


What is under the userbase?


Why is they no lib/python3.12?


Pip Install Numpy - Python 3.12.0


What has changed under the .local/bin folder?


Remember: Pip Installs and Package Locations

image-20240823-142848.png