Goal: Understand where pip installs packages within a user’s home folder with respect to different versions of Python.
...
Table of Contents | ||
---|---|---|
|
...
Python on the System
Python is available on the System (but not necessarily on the compute nodes due to keeping node images lite).
Code Block |
---|
[salexan5@mblog2 ~]$ python --version
Python 3.9.18
[salexan5@mblog1 ~]$ which python
/usr/bin/python |
Note |
---|
Note:
|
...
Python as a Module
Different versions are available via loading a module
Code Block |
---|
[]$ module spider python
----------------------------------------------------------------------------
python:
----------------------------------------------------------------------------
Versions:
python/3.10.6
python/3.12.0
Other possible modules matches:
python2
[]$ module purge
[]$ module load gcc/13.2.0 python/3.10.6
[]$ python --version
Python 3.10.6
[]$ which pip
/apps/u/spack/gcc/13.2.0/python/3.10.6-jh3qs5v/bin/pip |
Info |
---|
Note:
|
...
Pip Install Numpy - Python 3.10.6
Info |
---|
Let’s |
Code Block |
---|
# Using python/3.10.6
[]$ pip install numpy
Defaulting to user installation because normal site-packages is not writeable
Collecting numpy
Downloading numpy-2.0.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (60 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 60.9/60.9 kB 2.0 MB/s eta 0:00:00
Downloading numpy-2.0.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (19.3 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 19.3/19.3 MB 131.3 MB/s eta 0:00:00
Installing collected packages: numpy
Successfully installed numpy-2.0.0
[notice] A new release of pip is available: 24.0 -> 24.1
[notice] To update, run: pip install --upgrade pip |
Expand | ||
---|---|---|
| ||
|
Code Block |
---|
[]$ python py_test.py
Python: 3.10.6 (main, Apr 30 2024, 11:23:04) [GCC 13.2.0]
Numpy: 2.0.0 |
...
Use Python 3.12.0 - Is numpy available?
Code Block |
---|
[]$ module purge
[]$ module load gcc/13.2.0 python/3.12.0
[]$ python --version
Python 3.12.0
[]$ which python
/apps/u/spack/gcc/13.2.0/python/3.12.0-ovfqpv2/bin/python
[]$ python py_test.py
Traceback (most recent call last):
File "/cluster/medbow/home/<username>/py_test.py", line 2, in <module>
import numpy
ModuleNotFoundError: No module named 'numpy |
Note |
---|
But didn’t we previously install the |
...
Python System Configuration
sysconfig: The sysconfig
module provides access to Python’s configuration information like the list of installation paths and the configuration variables relevant for the current platform.
Code Block |
---|
[]$ module purge
[]$ module load gcc/13.2.0 python/3.10.6
[]$ python -m sysconfig
Platform: "linux-x86_64"
Python version: "3.10"
Current installation scheme: "posix_prefix"
Paths:
...
scripts = "/apps/u/spack/gcc/13.2.0/python/3.10.6-jh3qs5v/bin"
stdlib = "/apps/u/spack/gcc/13.2.0/python/3.10.6-jh3qs5v/lib/python3.10"
...
Variables:
...
py_version = "3.10.6"
...
py_version_short = "3.10"
...
userbase = "/home/<username>/.local"
# Full list has over 700 lines! |
View file | ||
---|---|---|
|
...
What is the Installation Scheme?
Prefix scheme: The “prefix scheme” is useful when you wish to use one Python installation to perform the build/install (i.e., to run the setup script), but install modules into the third-party module directory of a different Python installation (or something that looks like a different Python installation). If this sounds a trifle unusual, it is—that’s why the user and home schemes come before. However, there are at least two known cases where the prefix scheme will be useful.
posix_prefix:
Path | Installation directory |
---|---|
stdlib |
|
scripts |
|
data |
|
Info |
---|
The prefix will be the |
...
What is under the userbase?
Code Block |
---|
[~]$ ls .local
bin lib share
[]$ ls -al .local/bin
total 1
drwxr-xr-x 2 <username> <username> 4096 Jun 24 11:16 .
drwxr-xr-x 2 <username> <username> 4096 Jun 24 11:16 ..
-rwxr-xr-x 1 <username> <username> 257 Jun 24 11:16 f2py
-rwxr-xr-x 1 <username> <username> 257 Jun 24 11:16 numpy-config
[]$ ls .local/lib/
python3.10
[]$ ls .local/lib/python3.10/site-packages/
numpy numpy-2.0.0.dist-info numpy.libs
# ~/.local/lib/python3.10 |
...
Why is they no lib/python3.12?
Note |
---|
Because you have not pip installed anything for this particular version. |
Info |
---|
When using Since we have not pip installed |
...
Pip Install Numpy - Python 3.12.0
Code Block |
---|
[]$ module purge
[]$ module load gcc/13.2.0 python/3.12.0
[]$ pip install numpy==1.26.4
[]$ python py_test.py
Python: 3.12.0 (main, Apr 30 2024, 11:30:37) [GCC 13.2.0]
Numpy: 1.26.4
[~]$ ls .local/lib/
python3.10 python3.12
[~]$ ls .local/lib/python3.12/site-packages/
numpy numpy-1.26.4.dist-info numpy.libs |
...
What has changed under the .local/bin folder?
Code Block |
---|
# Previously when installing numpy for python/3.10.6
[~]$ ls -al .local/bin
total 1
-rwxr-xr-x 1 salexan5 salexan5 257 Jun 24 11:16 f2py
-rwxr-xr-x 1 salexan5 salexan5 257 Jun 24 11:16 numpy-config
[~]$ cat .local/bin/f2py
#!/apps/u/spack/gcc/13.2.0/python/3.10.6-jh3qs5v/bin/python
...
# Now after installing numpy for python/3.12.0
[~]$ ls -al .local/bin/
-rwxr-xr-x 1 salexan5 salexan5 257 Jun 24 11:45 f2py
-rwxr-xr-x 1 salexan5 salexan5 257 Jun 24 11:16 numpy-config
[~]$ cat .local/bin/f2py
#!/apps/u/spack/gcc/13.2.0/python/3.12.0-ovfqpv2/bin/python
... |
Note |
---|
The |
...
Remember: Pip Installs and Package Locations
Info |
---|
In general: Using |
...