Goal: When using Jupyter how are python packages managed?
Where are Python Packages Installed During a Jupyter Session?
You can use pip list -v
and conda list
to inspect what is in your environment.
Example using the default Python kernel:
# Cell import sys print(sys.version)
3.12.3 | packaged by Anaconda, Inc. | (main, May 6 2024, 19:46:43) [GCC 11.2.0]
Notice there are two paths:
The library path of the conda environment that this default kernel is running from:
/apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages
Your local home library path related to the version of Python within this kernel:
/home/salexan5/.local/lib/python3.12/site-packages
Can I pip install
?
Yes. And this will installed into your local home under the appropriate library/version folder.
# Before pip install: [salexan5@mblog2 ~]$ ls .local/lib/python3.12/site-packages/ dateutil numpy-2.0.0.dist-info __pycache__ pytz six-1.16.0.dist-info tzdata numpy numpy.libs python_dateutil-2.9.0.post0.dist-info pytz-2024.1.dist-info six.py tzdata-2024.1.dist-info
# Within Cell: pip install nltk
# After Installation: # Notice the new packages installed. [salexan5@mblog2 ~]$ ls .local/lib/python3.12/site-packages/ click joblib-1.4.2.dist-info numpy-2.0.0.dist-info pytz six-1.16.0.dist-info tzdata click-8.1.7.dist-info nltk numpy.libs pytz-2024.1.dist-info six.py tzdata-2024.1.dist-info dateutil nltk-3.8.1.dist-info __pycache__ regex tqdm joblib numpy python_dateutil-2.9.0.post0.dist-info regex-2024.7.24.dist-info tqdm-4.66.5.dist-info
Remember: Any previously install Python packages under this location will be available to your notebook.
Any installs/updates within the notebook will thus also affect any other Python environments you have that use this location and packages.
This could cause issues with dependencies and versions.
Can I Use Conda Within My Notebook?
Yes and no…
You can use conda
commands within cells to inspect and search.
You will not have permissions to conda install
since you do not own this folder, so do not have permissions to install/update anything within it.
But, you should have permissions if you have created the kernel, own it, and thus have permissions.
Jupyter Sessions
When you start a Jupyter session via OnDemand, you get a Session ID, which also creates a corresponding folder within your home:
Clicking on the Session ID link will open up a new tab:
Or, you can navigate to it via the command-line:
[salexan5@mblog2 ~]$ ls ondemand/data/sys/dashboard/batch_connect/sys/jupyter/form/output/8c0766ae-b302-47d7-8f40-007460cc3ad1/ after.sh before.sh config.py connection.yml job_script_content.sh job_script_options.json output.log script.sh user_defined_context.json
You can view the output.log
file under this folder to track the session.
Issue: Jupyter Doesn’t Start?
Sometimes when you try and start a Jupyter session it immediately fails to open.
This is typically caused by something that you have done/run/installed that have installed a python package under the library/version folder that is causing a dependency issue with a Python package installed under the kernel..
If you do not have the Session ID, then list the child session folders under ~/ondemand/data/sys/dashboard/batch_connect/sys/jupyter/form/output/
and look at the output.log file under the last child folder created
[salexan5@mblog2 ~]$ ls -altr ~/ondemand/data/sys/dashboard/batch_connect/sys/jupyter/form/output/ ... drwxr-xr-x 2 salexan5 salexan5 4096 Aug 6 08:07 8c0766ae-b302-47d7-8f40-007460cc3ad1
Issue: Jupyter Doesn’t Start: Example
This example is from an older cluster/version of OnDemand, but the principle still applies.
A user had the following error:
[nvlastos@blog2 d248fc5c-162c-42c6-b96d-e2ea46cbf3c1]$ cat output.log ... TIMING - Starting jupyter at: Fri Jul 26 13:20:12 MDT 2024 Traceback (most recent call last): File "/apps/s/jupyterlab/miniconda3/bin/jupyter-lab", line 6, in <module> from jupyterlab.labapp import main File "/apps/s/jupyterlab/miniconda3/lib/python3.9/site-packages/jupyterlab/__init__.py", line 7, in <module> from .labapp import LabApp File "/apps/s/jupyterlab/miniconda3/lib/python3.9/site-packages/jupyterlab/labapp.py", line 18, in <module> from jupyterlab_server import WORKSPACE_EXTENSION, LabServerApp, slugify File "/apps/s/jupyterlab/miniconda3/lib/python3.9/site-packages/jupyterlab_server/__init__.py", line 4, in <module> from .app import LabServerApp File "/apps/s/jupyterlab/miniconda3/lib/python3.9/site-packages/jupyterlab_server/app.py", line 11, in <module> from .handlers import LabConfig, add_handlers File "/apps/s/jupyterlab/miniconda3/lib/python3.9/site-packages/jupyterlab_server/handlers.py", line 19, in <module> from .translations_handler import TranslationsHandler File "/apps/s/jupyterlab/miniconda3/lib/python3.9/site-packages/jupyterlab_server/translations_handler.py", line 16, in <module> from .translation_utils import get_language_pack, get_language_packs, is_valid_locale, translator File "/apps/s/jupyterlab/miniconda3/lib/python3.9/site-packages/jupyterlab_server/translation_utils.py", line 15, in <module> import pkg_resources File "/home/nvlastos/.local/lib/python3.9/site-packages/pkg_resources/__init__.py", line 96, in <module> from jaraco.text import ( File "/home/fakeusername/.local/lib/python3.9/site-packages/setuptools/_vendor/jaraco/text/__init__.py", line 12, in <module> from jaraco.context import ExceptionTrap File "/home/fakeusername/.local/lib/python3.9/site-packages/setuptools/_vendor/jaraco/context.py", line 17, in <module> from backports import tarfile ImportError: cannot import name 'tarfile' from 'backports' (/apps/s/jupyterlab/miniconda3/lib/python3.9/site-packages/backports/__init__.py) Timed out waiting for Jupyter Notebook server to open port 40276! TIMING - Wait ended at: Fri Jul 26 13:21:19 MDT 2024 Cleaning up...
Prev | Workshop Home | Next |