Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Info

You can use pip list -v and conda list to inspect what is in your environment.

Info

Example using the default Python kernel:

Code Block
# Cell
import sys
print(sys.version)

...

Expand
titlepip list -v
Code Block
Package                   Version     Location                                                   Installer
------------------------- ----------- ---------------------------------------------------------- ---------
anyio                     4.2.0       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
argon2-cffi               21.3.0      /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
argon2-cffi-bindings      21.2.0      /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
asttokens                 2.0.5       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
async-lru                 2.0.4       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
attrs                     23.1.0      /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
Babel                     2.11.0      /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
beautifulsoup4            4.12.2      /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
bleach                    4.1.0       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
Brotli                    1.0.9       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
certifi                   2024.2.2    /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
cffi                      1.16.0      /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
charset-normalizer        2.0.4       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
comm                      0.2.1       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
debugpy                   1.6.7       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
decorator                 5.1.1       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
defusedxml                0.7.1       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
executing                 0.8.3       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
fastjsonschema            2.16.2      /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
idna                      3.7         /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
ipykernel                 6.28.0      /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
ipython                   8.20.0      /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
ipywidgets                8.1.2       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
jedi                      0.18.1      /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
Jinja2                    3.1.3       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
json5                     0.9.6       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
jsonschema                4.19.2      /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
jsonschema-specifications 2023.7.1    /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
jupyter                   1.0.0       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
jupyter_client            8.6.0       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
jupyter-console           6.6.3       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
jupyter_core              5.5.0       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
jupyter-events            0.8.0       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
jupyter-lsp               2.2.0       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
jupyter_server            2.10.0      /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
jupyter_server_terminals  0.4.4       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
jupyterlab                4.0.11      /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
jupyterlab-pygments       0.1.2       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
jupyterlab_server         2.25.1      /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
jupyterlab-widgets        3.0.10      /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
MarkupSafe                2.1.3       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
matplotlib-inline         0.1.6       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
mistune                   2.0.4       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
nbclient                  0.8.0       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
nbconvert                 7.10.0      /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
nbformat                  5.9.2       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
nest-asyncio              1.6.0       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
notebook                  7.0.8       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
notebook_shim             0.2.3       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
numpy                     2.0.0       /home/salexan5<username>/.local/lib/python3.12/site-packages         pip
overrides                 7.4.0       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
packaging                 23.2        /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
pandocfilters             1.5.0       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
parso                     0.8.3       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
pexpect                   4.8.0       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
pip                       24.0        /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages
platformdirs              3.10.0      /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
ply                       3.11        /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
prometheus-client         0.14.1      /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
prompt-toolkit            3.0.43      /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
psutil                    5.9.0       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
ptyprocess                0.7.0       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
pure-eval                 0.2.2       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
pycparser                 2.21        /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
Pygments                  2.15.1      /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
PyQt5                     5.15.10     /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
PyQt5-sip                 12.13.0     /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
PySocks                   1.7.1       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
python-dateutil           2.9.0.post0 /home/salexan5<username>/.local/lib/python3.12/site-packages         pip
python-json-logger        2.0.7       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
pytz                      2024.1      /home/salexan5<username>/.local/lib/python3.12/site-packages         pip
PyYAML                    6.0.1       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
pyzmq                     25.1.2      /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
qtconsole                 5.5.1       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
QtPy                      2.4.1       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
referencing               0.30.2      /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
requests                  2.31.0      /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
rfc3339-validator         0.1.4       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
rfc3986-validator         0.1.1       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
rpds-py                   0.10.6      /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
Send2Trash                1.8.2       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
setuptools                69.5.1      /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages
sip                       6.7.12      /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
six                       1.16.0      /home/salexan5<username>/.local/lib/python3.12/site-packages         pip
sniffio                   1.3.0       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
soupsieve                 2.5         /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
stack-data                0.2.0       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
terminado                 0.17.1      /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
tinycss2                  1.2.1       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
tornado                   6.3.3       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
traitlets                 5.7.1       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
typing_extensions         4.11.0      /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
tzdata                    2024.1      /home/salexan5<username>/.local/lib/python3.12/site-packages         pip
urllib3                   2.2.1       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
wcwidth                   0.2.5       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
webencodings              0.5.1       /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
websocket-client          0.58.0      /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
wheel                     0.43.0      /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages
widgetsnbextension        4.0.10      /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
Note: you may need to restart the kernel to use updated packages.
Info

Notice there are two paths:

  1. The library path of the conda environment that this default kernel is running from: /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages

  2. Your local home library path related to the version of Python within this kernel: /home/salexan5<username>/.local/lib/python3.12/site-packages

...

Code Block
# Before pip install:
[salexan5@mblog2 ~]$ ls .local/lib/python3.12/site-packages/
dateutil  numpy-2.0.0.dist-info  __pycache__                            pytz                   six-1.16.0.dist-info  tzdata
numpy     numpy.libs             python_dateutil-2.9.0.post0.dist-info  pytz-2024.1.dist-info  six.py                tzdata-2024.1.dist-info

...

Code Block
# After Installation:
# Notice the new packages installed.
[salexan5@mblog2 ~]$ ls .local/lib/python3.12/site-packages/
click                  joblib-1.4.2.dist-info  numpy-2.0.0.dist-info                  pytz                       six-1.16.0.dist-info   tzdata
click-8.1.7.dist-info  nltk                    numpy.libs                             pytz-2024.1.dist-info      six.py                 tzdata-2024.1.dist-info
dateutil               nltk-3.8.1.dist-info    __pycache__                            regex                      tqdm
joblib                 numpy                   python_dateutil-2.9.0.post0.dist-info  regex-2024.7.24.dist-info  tqdm-4.66.5.dist-info
Note

Remember: Any previously install Python packages under this location will be available to your notebook.

Any installs/updates within the notebook will thus also affect any other Python environments you have that use this location and packages.

This could cause issues with dependencies and versions.

...

Note

You can use conda commands within cells to inspect and search.

You will not have permissions to conda install under this default Python kernel since you do not own this folder, so do not have permissions to install/update anything within it.

Info

But, you should have permissions be able to if you have created the kernel, own it, and thus have permissions.

...

Jupyter Sessions

Info

Issue: Jupyter Doesn’t Start?

Prev

...

When you start a Jupyter session via OnDemand, you get a Session ID, which also creates a corresponding folder within your home:

...

Info

Clicking on the Session ID link will open up a new tab of the form:

...

Info

Or, you can navigate to it via the command-line:

Code Block
[]$ ls ~/ondemand/data/sys/dashboard/batch_connect/sys/jupyter/form/output/8c0766ae-b302-47d7-8f40-007460cc3ad1/
after.sh  before.sh  config.py  connection.yml  job_script_content.sh  job_script_options.json  output.log  script.sh  user_defined_context.json
Info

You can view the output.log file under this folder to track the session.

...

Issue: Jupyter Doesn’t Start?

Note

Sometimes when you try and start a Jupyter session it immediately fails to open.

This is typically caused by something that you have done/run/installed that has installed a python package under the library/version folder that is causing a dependency issue with a Python package installed under the kernel..

Info

If you do not have the Session ID, then list the child session folders, in date order, under ~/ondemand/data/sys/dashboard/batch_connect/sys/jupyter/form/output/ and look at the output.log file under the last child folder created:

Code Block
[]$ ls -altr ~/ondemand/data/sys/dashboard/batch_connect/sys/jupyter/form/output/
...
drwxr-xr-x 2 <username> <username> 4096 Aug  6 08:07 8c0766ae-b302-47d7-8f40-007460cc3ad1

...

Issue: Jupyter Doesn’t Start: Example

Info

This example is from an older cluster/version of OnDemand, but the principle still applies.

Info

The fakeusername user had the following error:

Expand
titleFull Trace:
Code Block
[fakeusername@blog2 d248fc5c-162c-42c6-b96d-e2ea46cbf3c1]$ cat output.log
...
TIMING - Starting jupyter at: Fri Jul 26 13:20:12 MDT 2024
Traceback (most recent call last):
  File "/apps/s/jupyterlab/miniconda3/bin/jupyter-lab", line 6, in <module>
    from jupyterlab.labapp import main
  File "/apps/s/jupyterlab/miniconda3/lib/python3.9/site-packages/jupyterlab/__init__.py", line 7, in <module>
    from .labapp import LabApp
  File "/apps/s/jupyterlab/miniconda3/lib/python3.9/site-packages/jupyterlab/labapp.py", line 18, in <module>
    from jupyterlab_server import WORKSPACE_EXTENSION, LabServerApp, slugify
  File "/apps/s/jupyterlab/miniconda3/lib/python3.9/site-packages/jupyterlab_server/__init__.py", line 4, in <module>
    from .app import LabServerApp
  File "/apps/s/jupyterlab/miniconda3/lib/python3.9/site-packages/jupyterlab_server/app.py", line 11, in <module>
    from .handlers import LabConfig, add_handlers
  File "/apps/s/jupyterlab/miniconda3/lib/python3.9/site-packages/jupyterlab_server/handlers.py", line 19, in <module>
    from .translations_handler import TranslationsHandler
  File "/apps/s/jupyterlab/miniconda3/lib/python3.9/site-packages/jupyterlab_server/translations_handler.py", line 16, in <module>
    from .translation_utils import get_language_pack, get_language_packs, is_valid_locale, translator
  File "/apps/s/jupyterlab/miniconda3/lib/python3.9/site-packages/jupyterlab_server/translation_utils.py", line 15, in <module>
    import pkg_resources
  File "/home/fakeusername/.local/lib/python3.9/site-packages/pkg_resources/__init__.py", line 96, in <module>
    from jaraco.text import (
  File "/home/fakeusername/.local/lib/python3.9/site-packages/setuptools/_vendor/jaraco/text/__init__.py", line 12, in <module>
    from jaraco.context import ExceptionTrap
  File "/home/fakeusername/.local/lib/python3.9/site-packages/setuptools/_vendor/jaraco/context.py", line 17, in <module>
    from backports import tarfile
ImportError: cannot import name 'tarfile' from 'backports' (/apps/s/jupyterlab/miniconda3/lib/python3.9/site-packages/backports/__init__.py)
Timed out waiting for Jupyter Notebook server to open port 40276!
TIMING - Wait ended at: Fri Jul 26 13:21:19 MDT 2024
Cleaning up...
Note

Two things to notice.

  1. The ImportError caused by a dependency issue: ImportError: cannot import name 'tarfile' from 'backports'

  2. The location of the package, that has been installed, where this is originating from: /home/fakeusername/.local/lib/python3.9/site-packages/pkg_resources

Info

Unfortunately, the simplest way to resolve is to remove this folder.

How this effects your environment/pipeline you will have to figure out.

Unfortunately, the more complicated you try and make a single environment, the more likely this is too occur.

Suggestion: Maybe consider creating a separate Conda environment/kernel.

...

...