Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 2 Next »

Goal: When using Jupyter how are python packages managed?

Where are Python Packages Installed During a Jupyter Session?

You can use pip list -v and conda list to inspect what is in your environment.

Example using the default Python kernel:

# Cell
import sys
print(sys.version)
3.12.3 | packaged by Anaconda, Inc. | (main, May  6 2024, 19:46:43) [GCC 11.2.0]
 pip list -v
Package                   Version     Location                                                   Installer
------------------------- ----------- ---------------------------------------------------------- ---------
anyio                     4.2.0       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
argon2-cffi               21.3.0      /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
argon2-cffi-bindings      21.2.0      /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
asttokens                 2.0.5       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
async-lru                 2.0.4       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
attrs                     23.1.0      /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
Babel                     2.11.0      /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
beautifulsoup4            4.12.2      /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
bleach                    4.1.0       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
Brotli                    1.0.9       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
certifi                   2024.2.2    /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
cffi                      1.16.0      /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
charset-normalizer        2.0.4       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
comm                      0.2.1       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
debugpy                   1.6.7       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
decorator                 5.1.1       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
defusedxml                0.7.1       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
executing                 0.8.3       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
fastjsonschema            2.16.2      /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
idna                      3.7         /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
ipykernel                 6.28.0      /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
ipython                   8.20.0      /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
ipywidgets                8.1.2       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
jedi                      0.18.1      /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
Jinja2                    3.1.3       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
json5                     0.9.6       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
jsonschema                4.19.2      /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
jsonschema-specifications 2023.7.1    /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
jupyter                   1.0.0       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
jupyter_client            8.6.0       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
jupyter-console           6.6.3       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
jupyter_core              5.5.0       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
jupyter-events            0.8.0       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
jupyter-lsp               2.2.0       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
jupyter_server            2.10.0      /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
jupyter_server_terminals  0.4.4       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
jupyterlab                4.0.11      /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
jupyterlab-pygments       0.1.2       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
jupyterlab_server         2.25.1      /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
jupyterlab-widgets        3.0.10      /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
MarkupSafe                2.1.3       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
matplotlib-inline         0.1.6       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
mistune                   2.0.4       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
nbclient                  0.8.0       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
nbconvert                 7.10.0      /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
nbformat                  5.9.2       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
nest-asyncio              1.6.0       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
notebook                  7.0.8       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
notebook_shim             0.2.3       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
numpy                     2.0.0       /home/salexan5/.local/lib/python3.12/site-packages         pip
overrides                 7.4.0       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
packaging                 23.2        /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
pandocfilters             1.5.0       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
parso                     0.8.3       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
pexpect                   4.8.0       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
pip                       24.0        /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages
platformdirs              3.10.0      /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
ply                       3.11        /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
prometheus-client         0.14.1      /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
prompt-toolkit            3.0.43      /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
psutil                    5.9.0       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
ptyprocess                0.7.0       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
pure-eval                 0.2.2       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
pycparser                 2.21        /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
Pygments                  2.15.1      /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
PyQt5                     5.15.10     /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
PyQt5-sip                 12.13.0     /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
PySocks                   1.7.1       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
python-dateutil           2.9.0.post0 /home/salexan5/.local/lib/python3.12/site-packages         pip
python-json-logger        2.0.7       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
pytz                      2024.1      /home/salexan5/.local/lib/python3.12/site-packages         pip
PyYAML                    6.0.1       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
pyzmq                     25.1.2      /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
qtconsole                 5.5.1       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
QtPy                      2.4.1       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
referencing               0.30.2      /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
requests                  2.31.0      /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
rfc3339-validator         0.1.4       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
rfc3986-validator         0.1.1       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
rpds-py                   0.10.6      /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
Send2Trash                1.8.2       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
setuptools                69.5.1      /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages
sip                       6.7.12      /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
six                       1.16.0      /home/salexan5/.local/lib/python3.12/site-packages         pip
sniffio                   1.3.0       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
soupsieve                 2.5         /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
stack-data                0.2.0       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
terminado                 0.17.1      /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
tinycss2                  1.2.1       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
tornado                   6.3.3       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
traitlets                 5.7.1       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
typing_extensions         4.11.0      /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
tzdata                    2024.1      /home/salexan5/.local/lib/python3.12/site-packages         pip
urllib3                   2.2.1       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
wcwidth                   0.2.5       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
webencodings              0.5.1       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
websocket-client          0.58.0      /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
wheel                     0.43.0      /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages
widgetsnbextension        4.0.10      /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
Note: you may need to restart the kernel to use updated packages.

Notice there are two paths:

  1. The library path of the conda environment that this default kernel is running from: /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages

  2. Your local home library path related to the version of Python within this kernel: /home/salexan5/.local/lib/python3.12/site-packages

Can I pip install?

Yes. And this will installed into your local home under the appropriate library/version folder.

# Before pip install:
[salexan5@mblog2 ~]$ ls .local/lib/python3.12/site-packages/
dateutil  numpy-2.0.0.dist-info  __pycache__                            pytz                   six-1.16.0.dist-info  tzdata
numpy     numpy.libs             python_dateutil-2.9.0.post0.dist-info  pytz-2024.1.dist-info  six.py                tzdata-2024.1.dist-info
# Within Cell:
pip install nltk
 Cell Result
Defaulting to user installation because normal site-packages is not writeable
Collecting nltk
  Using cached nltk-3.8.1-py3-none-any.whl.metadata (2.8 kB)
Collecting click (from nltk)
  Using cached click-8.1.7-py3-none-any.whl.metadata (3.0 kB)
Collecting joblib (from nltk)
  Using cached joblib-1.4.2-py3-none-any.whl.metadata (5.4 kB)
Collecting regex>=2021.8.3 (from nltk)
  Using cached regex-2024.7.24-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (40 kB)
Collecting tqdm (from nltk)
  Downloading tqdm-4.66.5-py3-none-any.whl.metadata (57 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 57.6/57.6 kB 1.5 MB/s eta 0:00:00
Using cached nltk-3.8.1-py3-none-any.whl (1.5 MB)
Using cached regex-2024.7.24-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (790 kB)
Using cached click-8.1.7-py3-none-any.whl (97 kB)
Using cached joblib-1.4.2-py3-none-any.whl (301 kB)
Downloading tqdm-4.66.5-py3-none-any.whl (78 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 78.4/78.4 kB 2.9 MB/s eta 0:00:00
Installing collected packages: tqdm, regex, joblib, click, nltk
Successfully installed click-8.1.7 joblib-1.4.2 nltk-3.8.1 regex-2024.7.24 tqdm-4.66.5
Note: you may need to restart the kernel to use updated packages.
# After Installation:
# Notice the new packages installed.
[salexan5@mblog2 ~]$ ls .local/lib/python3.12/site-packages/
click                  joblib-1.4.2.dist-info  numpy-2.0.0.dist-info                  pytz                       six-1.16.0.dist-info   tzdata
click-8.1.7.dist-info  nltk                    numpy.libs                             pytz-2024.1.dist-info      six.py                 tzdata-2024.1.dist-info
dateutil               nltk-3.8.1.dist-info    __pycache__                            regex                      tqdm
joblib                 numpy                   python_dateutil-2.9.0.post0.dist-info  regex-2024.7.24.dist-info  tqdm-4.66.5.dist-info

Remember: Any previously install Python packages under this location will be available to your notebook.

Any installs/updates within the notebook will thus also affect any other Python environments you have that use this location and packages.

This could cause issues with dependencies and versions.

Can I Use Conda Within My Notebook?

Yes and no…

 conda list
# packages in environment at /apps/s/jupyterlab/miniconda3:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                        main  
_openmp_mutex             5.1                       1_gnu  
anyio                     4.2.0           py312h06a4308_0  
... 
zlib                      1.2.13               h5eee18b_1  
zstd                      1.5.5                hc292b87_2  
Note: you may need to restart the kernel to use updated packages.
 conda search nltk
Loading channels: done
# Name                       Version           Build  Channel             
nltk                           3.2.1          py27_0  conda-forge         
nltk                           3.2.1          py34_0  conda-forge         
...      
nltk                           3.8.1  py39h06a4308_0  pkgs/main           
nltk                           3.8.1    pyhd8ed1ab_0  conda-forge         
Note: you may need to restart the kernel to use updated packages.
 conda install nltk
Retrieving notices: ...working... done
Channels:
 - conda-forge
 - defaults
Platform: linux-64
Collecting package metadata (repodata.json): done
Solving environment: done

## Package Plan ##
  environment location: /apps/s/jupyterlab/miniconda3
  added / updated specs:
    - nltk

The following packages will be downloaded:
    package                    |            build
    ---------------------------|-----------------
    certifi-2024.7.4           |     pyhd8ed1ab_0         156 KB  conda-forge
    nltk-3.8.1                 |     pyhd8ed1ab_0         1.0 MB  conda-forge
    regex-2023.10.3            |  py312h5eee18b_0         386 KB
    tqdm-4.66.5                |     pyhd8ed1ab_0          87 KB  conda-forge
    ------------------------------------------------------------
                                           Total:         1.6 MB

The following NEW packages will be INSTALLED:
  click              conda-forge/noarch::click-8.1.7-unix_pyh707e725_0 
  colorama           conda-forge/noarch::colorama-0.4.6-pyhd8ed1ab_0 
  joblib             conda-forge/noarch::joblib-1.4.2-pyhd8ed1ab_0 
  nltk               conda-forge/noarch::nltk-3.8.1-pyhd8ed1ab_0 
  regex              pkgs/main/linux-64::regex-2023.10.3-py312h5eee18b_0 
  tqdm               conda-forge/noarch::tqdm-4.66.5-pyhd8ed1ab_0 

The following packages will be UPDATED:
  ca-certificates    pkgs/main::ca-certificates-2024.3.11-~ --> conda-forge::ca-certificates-2024.7.4-hbcca054_0 
  certifi            pkgs/main/linux-64::certifi-2024.2.2-~ --> conda-forge/noarch::certifi-2024.7.4-pyhd8ed1ab_0 
  libgcc-ng          pkgs/main::libgcc-ng-11.2.0-h1234567_1 --> conda-forge::libgcc-ng-14.1.0-h77fa898_0 
  libgomp              pkgs/main::libgomp-11.2.0-h1234567_1 --> conda-forge::libgomp-14.1.0-h77fa898_0 
  openssl              pkgs/main::openssl-3.0.13-h7f8727e_1 --> conda-forge::openssl-3.3.1-h4bc722e_2 

The following packages will be SUPERSEDED by a higher-priority channel:
  _libgcc_mutex           pkgs/main::_libgcc_mutex-0.1-main --> conda-forge::_libgcc_mutex-0.1-conda_forge 
  _openmp_mutex          pkgs/main::_openmp_mutex-5.1-1_gnu --> conda-forge::_openmp_mutex-4.5-2_gnu 
...                                                                           
Preparing transaction: done
Verifying transaction: failed

EnvironmentNotWritableError: The current user does not have write permissions to the target environment.
  environment location: /apps/s/jupyterlab/miniconda3
  uid: 10275911
  gid: 10275911

Note: you may need to restart the kernel to use updated packages.

You can use conda commands within cells to inspect and search.

You will not have permissions to conda install since you do not own this folder, so do not have permissions to install/update anything within it.

But, you should have permissions if you have created the kernel, own it, and thus have permissions.

Jupyter Sessions

Issue: Jupyter Doesn’t Start?

Prev

Create a Module File to Load Your Conda Environment

Workshop Home

Using Python, Conda and Pip on the Cluster

Next

Conda and Pip Environments and Reproducibility

 

  • No labels