Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 9 Current »

Goal: When using Jupyter how are python packages managed?

Where are Python Packages Installed During a Jupyter Session?

You can use pip list -v and conda list to inspect what is in your environment.

Example using the default Python kernel:

# Cell
import sys
print(sys.version)
3.12.3 | packaged by Anaconda, Inc. | (main, May  6 2024, 19:46:43) [GCC 11.2.0]
 pip list -v
Package                   Version     Location                                                   Installer
------------------------- ----------- ---------------------------------------------------------- ---------
anyio                     4.2.0       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
argon2-cffi               21.3.0      /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
argon2-cffi-bindings      21.2.0      /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
asttokens                 2.0.5       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
async-lru                 2.0.4       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
attrs                     23.1.0      /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
Babel                     2.11.0      /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
beautifulsoup4            4.12.2      /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
bleach                    4.1.0       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
Brotli                    1.0.9       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
certifi                   2024.2.2    /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
cffi                      1.16.0      /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
charset-normalizer        2.0.4       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
comm                      0.2.1       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
debugpy                   1.6.7       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
decorator                 5.1.1       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
defusedxml                0.7.1       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
executing                 0.8.3       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
fastjsonschema            2.16.2      /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
idna                      3.7         /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
ipykernel                 6.28.0      /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
ipython                   8.20.0      /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
ipywidgets                8.1.2       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
jedi                      0.18.1      /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
Jinja2                    3.1.3       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
json5                     0.9.6       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
jsonschema                4.19.2      /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
jsonschema-specifications 2023.7.1    /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
jupyter                   1.0.0       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
jupyter_client            8.6.0       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
jupyter-console           6.6.3       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
jupyter_core              5.5.0       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
jupyter-events            0.8.0       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
jupyter-lsp               2.2.0       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
jupyter_server            2.10.0      /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
jupyter_server_terminals  0.4.4       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
jupyterlab                4.0.11      /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
jupyterlab-pygments       0.1.2       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
jupyterlab_server         2.25.1      /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
jupyterlab-widgets        3.0.10      /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
MarkupSafe                2.1.3       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
matplotlib-inline         0.1.6       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
mistune                   2.0.4       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
nbclient                  0.8.0       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
nbconvert                 7.10.0      /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
nbformat                  5.9.2       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
nest-asyncio              1.6.0       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
notebook                  7.0.8       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
notebook_shim             0.2.3       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
numpy                     2.0.0       /home/<username>/.local/lib/python3.12/site-packages         pip
overrides                 7.4.0       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
packaging                 23.2        /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
pandocfilters             1.5.0       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
parso                     0.8.3       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
pexpect                   4.8.0       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
pip                       24.0        /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages
platformdirs              3.10.0      /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
ply                       3.11        /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
prometheus-client         0.14.1      /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
prompt-toolkit            3.0.43      /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
psutil                    5.9.0       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
ptyprocess                0.7.0       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
pure-eval                 0.2.2       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
pycparser                 2.21        /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
Pygments                  2.15.1      /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
PyQt5                     5.15.10     /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
PyQt5-sip                 12.13.0     /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
PySocks                   1.7.1       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
python-dateutil           2.9.0.post0 /home/<username>/.local/lib/python3.12/site-packages         pip
python-json-logger        2.0.7       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
pytz                      2024.1      /home/<username>/.local/lib/python3.12/site-packages         pip
PyYAML                    6.0.1       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
pyzmq                     25.1.2      /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
qtconsole                 5.5.1       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
QtPy                      2.4.1       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
referencing               0.30.2      /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
requests                  2.31.0      /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
rfc3339-validator         0.1.4       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
rfc3986-validator         0.1.1       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
rpds-py                   0.10.6      /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
Send2Trash                1.8.2       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
setuptools                69.5.1      /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages
sip                       6.7.12      /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
six                       1.16.0      /home/<username>/.local/lib/python3.12/site-packages         pip
sniffio                   1.3.0       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
soupsieve                 2.5         /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
stack-data                0.2.0       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
terminado                 0.17.1      /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
tinycss2                  1.2.1       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
tornado                   6.3.3       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
traitlets                 5.7.1       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
typing_extensions         4.11.0      /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
tzdata                    2024.1      /home/<username>/.local/lib/python3.12/site-packages         pip
urllib3                   2.2.1       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
wcwidth                   0.2.5       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
webencodings              0.5.1       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
websocket-client          0.58.0      /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
wheel                     0.43.0      /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages
widgetsnbextension        4.0.10      /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
Note: you may need to restart the kernel to use updated packages.

Notice there are two paths:

  1. The library path of the conda environment that this default kernel is running from: /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages

  2. Your local home library path related to the version of Python within this kernel: /home/salexan5/.local/lib/python3.12/site-packages

Can I pip install?

Yes. And this will installed into your local home under the appropriate library/version folder.

# Before pip install:
[~]$ ls .local/lib/python3.12/site-packages/
dateutil  numpy-2.0.0.dist-info  __pycache__                            pytz                   six-1.16.0.dist-info  tzdata
numpy     numpy.libs             python_dateutil-2.9.0.post0.dist-info  pytz-2024.1.dist-info  six.py                tzdata-2024.1.dist-info
# Within Cell:
pip install nltk
 Cell Result
Defaulting to user installation because normal site-packages is not writeable
Collecting nltk
  Using cached nltk-3.8.1-py3-none-any.whl.metadata (2.8 kB)
Collecting click (from nltk)
  Using cached click-8.1.7-py3-none-any.whl.metadata (3.0 kB)
Collecting joblib (from nltk)
  Using cached joblib-1.4.2-py3-none-any.whl.metadata (5.4 kB)
Collecting regex>=2021.8.3 (from nltk)
  Using cached regex-2024.7.24-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (40 kB)
Collecting tqdm (from nltk)
  Downloading tqdm-4.66.5-py3-none-any.whl.metadata (57 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 57.6/57.6 kB 1.5 MB/s eta 0:00:00
Using cached nltk-3.8.1-py3-none-any.whl (1.5 MB)
Using cached regex-2024.7.24-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (790 kB)
Using cached click-8.1.7-py3-none-any.whl (97 kB)
Using cached joblib-1.4.2-py3-none-any.whl (301 kB)
Downloading tqdm-4.66.5-py3-none-any.whl (78 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 78.4/78.4 kB 2.9 MB/s eta 0:00:00
Installing collected packages: tqdm, regex, joblib, click, nltk
Successfully installed click-8.1.7 joblib-1.4.2 nltk-3.8.1 regex-2024.7.24 tqdm-4.66.5
Note: you may need to restart the kernel to use updated packages.
# After Installation:
# Notice the new packages installed.
[~]$ ls .local/lib/python3.12/site-packages/
click                  joblib-1.4.2.dist-info  numpy-2.0.0.dist-info                  pytz                       six-1.16.0.dist-info   tzdata
click-8.1.7.dist-info  nltk                    numpy.libs                             pytz-2024.1.dist-info      six.py                 tzdata-2024.1.dist-info
dateutil               nltk-3.8.1.dist-info    __pycache__                            regex                      tqdm
joblib                 numpy                   python_dateutil-2.9.0.post0.dist-info  regex-2024.7.24.dist-info  tqdm-4.66.5.dist-info

Remember: Any previously install Python packages under this location will be available to your notebook.

Any installs/updates within the notebook will thus also affect any other Python environments you have that use this location and packages.

This could cause issues with dependencies and versions.

Can I Use Conda Within My Notebook?

Yes and no…

 conda list
# packages in environment at /apps/s/jupyterlab/miniconda3:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                        main  
_openmp_mutex             5.1                       1_gnu  
anyio                     4.2.0           py312h06a4308_0  
... 
zlib                      1.2.13               h5eee18b_1  
zstd                      1.5.5                hc292b87_2  
Note: you may need to restart the kernel to use updated packages.
 conda search nltk
Loading channels: done
# Name                       Version           Build  Channel             
nltk                           3.2.1          py27_0  conda-forge         
nltk                           3.2.1          py34_0  conda-forge         
...      
nltk                           3.8.1  py39h06a4308_0  pkgs/main           
nltk                           3.8.1    pyhd8ed1ab_0  conda-forge         
Note: you may need to restart the kernel to use updated packages.
 conda install nltk
Retrieving notices: ...working... done
Channels:
 - conda-forge
 - defaults
Platform: linux-64
Collecting package metadata (repodata.json): done
Solving environment: done

## Package Plan ##
  environment location: /apps/s/jupyterlab/miniconda3
  added / updated specs:
    - nltk

The following packages will be downloaded:
    package                    |            build
    ---------------------------|-----------------
    certifi-2024.7.4           |     pyhd8ed1ab_0         156 KB  conda-forge
    nltk-3.8.1                 |     pyhd8ed1ab_0         1.0 MB  conda-forge
    regex-2023.10.3            |  py312h5eee18b_0         386 KB
    tqdm-4.66.5                |     pyhd8ed1ab_0          87 KB  conda-forge
    ------------------------------------------------------------
                                           Total:         1.6 MB

The following NEW packages will be INSTALLED:
  click              conda-forge/noarch::click-8.1.7-unix_pyh707e725_0 
  colorama           conda-forge/noarch::colorama-0.4.6-pyhd8ed1ab_0 
  joblib             conda-forge/noarch::joblib-1.4.2-pyhd8ed1ab_0 
  nltk               conda-forge/noarch::nltk-3.8.1-pyhd8ed1ab_0 
  regex              pkgs/main/linux-64::regex-2023.10.3-py312h5eee18b_0 
  tqdm               conda-forge/noarch::tqdm-4.66.5-pyhd8ed1ab_0 

The following packages will be UPDATED:
  ca-certificates    pkgs/main::ca-certificates-2024.3.11-~ --> conda-forge::ca-certificates-2024.7.4-hbcca054_0 
  certifi            pkgs/main/linux-64::certifi-2024.2.2-~ --> conda-forge/noarch::certifi-2024.7.4-pyhd8ed1ab_0 
  libgcc-ng          pkgs/main::libgcc-ng-11.2.0-h1234567_1 --> conda-forge::libgcc-ng-14.1.0-h77fa898_0 
  libgomp              pkgs/main::libgomp-11.2.0-h1234567_1 --> conda-forge::libgomp-14.1.0-h77fa898_0 
  openssl              pkgs/main::openssl-3.0.13-h7f8727e_1 --> conda-forge::openssl-3.3.1-h4bc722e_2 

The following packages will be SUPERSEDED by a higher-priority channel:
  _libgcc_mutex           pkgs/main::_libgcc_mutex-0.1-main --> conda-forge::_libgcc_mutex-0.1-conda_forge 
  _openmp_mutex          pkgs/main::_openmp_mutex-5.1-1_gnu --> conda-forge::_openmp_mutex-4.5-2_gnu 
...                                                                           
Preparing transaction: done
Verifying transaction: failed

EnvironmentNotWritableError: The current user does not have write permissions to the target environment.
  environment location: /apps/s/jupyterlab/miniconda3
  uid: 10275911
  gid: 10275911

Note: you may need to restart the kernel to use updated packages.

You can use conda commands within cells to inspect and search.

You will not have permissions to conda install under this default Python kernel since you do not own this folder, so do not have permissions to install/update anything within it.

But, you should be able to if you have created the kernel, own it, and thus have permissions.

Jupyter Sessions

When you start a Jupyter session via OnDemand, you get a Session ID, which also creates a corresponding folder within your home:

image-20240806-144730.png

Clicking on the Session ID link will open up a new tab of the form:

image-20240806-144836.png

Or, you can navigate to it via the command-line:

[]$ ls ondemand/data/sys/dashboard/batch_connect/sys/jupyter/form/output/8c0766ae-b302-47d7-8f40-007460cc3ad1/
after.sh  before.sh  config.py  connection.yml  job_script_content.sh  job_script_options.json  output.log  script.sh  user_defined_context.json

You can view the output.log file under this folder to track the session.

Issue: Jupyter Doesn’t Start?

Sometimes when you try and start a Jupyter session it immediately fails to open.

This is typically caused by something that you have done/run/installed that has installed a python package under the library/version folder that is causing a dependency issue with a Python package installed under the kernel..

If you do not have the Session ID, then list the child session folders, in date order, under ~/ondemand/data/sys/dashboard/batch_connect/sys/jupyter/form/output/ and look at the output.log file under the last child folder created:

[]$ ls -altr ~/ondemand/data/sys/dashboard/batch_connect/sys/jupyter/form/output/
...
drwxr-xr-x 2 <username> <username> 4096 Aug  6 08:07 8c0766ae-b302-47d7-8f40-007460cc3ad1

Issue: Jupyter Doesn’t Start: Example

This example is from an older cluster/version of OnDemand, but the principle still applies.

The fakeusername user had the following error:

 Full Trace:
[fakeusername@blog2 d248fc5c-162c-42c6-b96d-e2ea46cbf3c1]$ cat output.log
...
TIMING - Starting jupyter at: Fri Jul 26 13:20:12 MDT 2024
Traceback (most recent call last):
  File "/apps/s/jupyterlab/miniconda3/bin/jupyter-lab", line 6, in <module>
    from jupyterlab.labapp import main
  File "/apps/s/jupyterlab/miniconda3/lib/python3.9/site-packages/jupyterlab/__init__.py", line 7, in <module>
    from .labapp import LabApp
  File "/apps/s/jupyterlab/miniconda3/lib/python3.9/site-packages/jupyterlab/labapp.py", line 18, in <module>
    from jupyterlab_server import WORKSPACE_EXTENSION, LabServerApp, slugify
  File "/apps/s/jupyterlab/miniconda3/lib/python3.9/site-packages/jupyterlab_server/__init__.py", line 4, in <module>
    from .app import LabServerApp
  File "/apps/s/jupyterlab/miniconda3/lib/python3.9/site-packages/jupyterlab_server/app.py", line 11, in <module>
    from .handlers import LabConfig, add_handlers
  File "/apps/s/jupyterlab/miniconda3/lib/python3.9/site-packages/jupyterlab_server/handlers.py", line 19, in <module>
    from .translations_handler import TranslationsHandler
  File "/apps/s/jupyterlab/miniconda3/lib/python3.9/site-packages/jupyterlab_server/translations_handler.py", line 16, in <module>
    from .translation_utils import get_language_pack, get_language_packs, is_valid_locale, translator
  File "/apps/s/jupyterlab/miniconda3/lib/python3.9/site-packages/jupyterlab_server/translation_utils.py", line 15, in <module>
    import pkg_resources
  File "/home/fakeusername/.local/lib/python3.9/site-packages/pkg_resources/__init__.py", line 96, in <module>
    from jaraco.text import (
  File "/home/fakeusername/.local/lib/python3.9/site-packages/setuptools/_vendor/jaraco/text/__init__.py", line 12, in <module>
    from jaraco.context import ExceptionTrap
  File "/home/fakeusername/.local/lib/python3.9/site-packages/setuptools/_vendor/jaraco/context.py", line 17, in <module>
    from backports import tarfile
ImportError: cannot import name 'tarfile' from 'backports' (/apps/s/jupyterlab/miniconda3/lib/python3.9/site-packages/backports/__init__.py)
Timed out waiting for Jupyter Notebook server to open port 40276!
TIMING - Wait ended at: Fri Jul 26 13:21:19 MDT 2024
Cleaning up...

Two things to notice.

  1. The ImportError caused by a dependency issue: ImportError: cannot import name 'tarfile' from 'backports'

  2. The location of the package, that has been installed, where this is originating from: /home/fakeusername/.local/lib/python3.9/site-packages/pkg_resources

Unfortunately, the simplest way to resolve is to remove this folder.

How this effects your environment/pipeline you will have to figure out.

Unfortunately, the more complicated you try and make a single environment, the more likely this is too occur.

Suggestion: Maybe consider creating a separate Conda environment/kernel.

Prev

Extend Conda Environment to Jupyter Kernel

Workshop Home

Using Python, Conda and Pip on the Cluster

Next

Conda and Pip Environments and Reproducibility

 

  • No labels