Goal: Understand where pip installs packages within a user’s home folder with respect to different versions of Python.

1 Python on the System
2 Python as a Module
3 Pip Install Numpy - Python 3.10.6
4 Use Python 3.12.0 - Is numpy available?
5 Python System Configuration
6 What is the Installation Scheme?
7 What is under the userbase?
8 Why is there no lib/python3.12?
9 Pip Install Numpy - Python 3.12.0
10 What has changed under the .local/bin folder?
11 Remember: Pip Installs and Package Locations

Python on the System

Python is available on the System (but not necessarily on the compute nodes due to keeping node images lite).

[salexan5@mblog2 ~]$ python --version
Python 3.9.18

[salexan5@mblog1 ~]$ which python
/usr/bin/python

Note:

Version might update during cluster maintenance.
Pip may/may not be installed.
Recommended to not use to due potential updates.

Python as a Module

Different versions are available via loading a module

[]$ module spider python
----------------------------------------------------------------------------
  python:
----------------------------------------------------------------------------
     Versions:
        python/3.10.6
        python/3.12.0
     Other possible modules matches:
        python2

[]$ module purge        
[]$ module load gcc/14.2.0 python/3.10.6
[]$ python --version
Python 3.10.6
[]$ which pip
/apps/u/spack/gcc/14.2.0/python/3.10.6-6lvrsdd/bin/pip

Note:

Recommended to use a module version for reproducibility.
Pip will be available.

Pip Install Numpy - Python 3.10.6

Let’s pip install numpy and then run a test to verify that it works.

# Using python/3.10.6
[]$ pip install numpy
Defaulting to user installation because normal site-packages is not writeable
Collecting numpy
  Downloading numpy-2.0.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (60 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 60.9/60.9 kB 2.0 MB/s eta 0:00:00
Downloading numpy-2.0.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (19.3 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 19.3/19.3 MB 131.3 MB/s eta 0:00:00
Installing collected packages: numpy
Successfully installed numpy-2.0.0

[notice] A new release of pip is available: 24.0 -> 24.1
[notice] To update, run: pip install --upgrade pip

Use Python 3.12.0 - Is numpy available?

Python System Configuration

sysconfig: The sysconfig module provides access to Python’s configuration information like the list of installation paths and the configuration variables relevant for the current platform.

What is the Installation Scheme?

Prefix scheme: The “prefix scheme” is useful when you wish to use one Python installation to perform the build/install (i.e., to run the setup script), but install modules into the third-party module directory of a different Python installation (or something that looks like a different Python installation). If this sounds a trifle unusual, it is—that’s why the user and home schemes come before. However, there are at least two known cases where the prefix scheme will be useful.

posix_prefix:

Path	Installation directory

Path	Installation directory
stdlib	`prefix/lib/pythonX.Y`
scripts	`prefix/bin`
data	`prefix`

What is under the userbase?

Why is there no lib/python3.12?

Pip Install Numpy - Python 3.12.0

What has changed under the .local/bin folder?

Remember: Pip Installs and Package Locations

Workshop Home

Using Python, Conda and Pip on the Cluster

Next

Pip Install within a Conda Environment

ARCC Wiki

Python Pip Installs on the Cluster