Goal: When using Jupyter how are python packages managed?
...
Table of Contents |
---|
minLevel | 1 |
---|
maxLevel | 1 |
---|
outline | false |
---|
style | none |
---|
type | list |
---|
printable | true |
---|
|
...
Where are Python Packages Installed During a Jupyter Session?
Info |
---|
You can use pip list -v and conda list to inspect what is in your environment. |
Info |
---|
Example using the default Python kernel: |
Code Block |
---|
# Cell
import sys
print(sys.version) |
Code Block |
---|
3.12.3 | packaged by Anaconda, Inc. | (main, May 6 2024, 19:46:43) [GCC 11.2.0] |
Expand |
---|
|
Code Block |
---|
Package Version Location Installer
------------------------- ----------- ---------------------------------------------------------- ---------
anyio 4.2.0 /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
argon2-cffi 21.3.0 /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
argon2-cffi-bindings 21.2.0 /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
asttokens 2.0.5 /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
async-lru 2.0.4 /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
attrs 23.1.0 /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
Babel 2.11.0 /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
beautifulsoup4 4.12.2 /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
bleach 4.1.0 /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
Brotli 1.0.9 /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
certifi 2024.2.2 /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
cffi 1.16.0 /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
charset-normalizer 2.0.4 /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
comm 0.2.1 /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
debugpy 1.6.7 /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
decorator 5.1.1 /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
defusedxml 0.7.1 /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
executing 0.8.3 /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
fastjsonschema 2.16.2 /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
idna 3.7 /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
ipykernel 6.28.0 /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
ipython 8.20.0 /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
ipywidgets 8.1.2 /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
jedi 0.18.1 /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
Jinja2 3.1.3 /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
json5 0.9.6 /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
jsonschema 4.19.2 /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
jsonschema-specifications 2023.7.1 /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
jupyter 1.0.0 /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
jupyter_client 8.6.0 /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
jupyter-console 6.6.3 /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
jupyter_core 5.5.0 /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
jupyter-events 0.8.0 /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
jupyter-lsp 2.2.0 /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
jupyter_server 2.10.0 /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
jupyter_server_terminals 0.4.4 /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
jupyterlab 4.0.11 /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
jupyterlab-pygments 0.1.2 /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
jupyterlab_server 2.25.1 /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
jupyterlab-widgets 3.0.10 /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
MarkupSafe 2.1.3 /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
matplotlib-inline 0.1.6 /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
mistune 2.0.4 /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
nbclient 0.8.0 /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
nbconvert 7.10.0 /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
nbformat 5.9.2 /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
nest-asyncio 1.6.0 /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
notebook 7.0.8 /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
notebook_shim 0.2.3 /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
numpy 2.0.0 /home/salexan5/.local/lib/python3.12/site-packages pip
overrides 7.4.0 /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
packaging 23.2 /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
pandocfilters 1.5.0 /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
parso 0.8.3 /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
pexpect 4.8.0 /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
pip 24.0 /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages
platformdirs 3.10.0 /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
ply 3.11 /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
prometheus-client 0.14.1 /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
prompt-toolkit 3.0.43 /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
psutil 5.9.0 /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
ptyprocess 0.7.0 /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
pure-eval 0.2.2 /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
pycparser 2.21 /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
Pygments 2.15.1 /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
PyQt5 5.15.10 /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
PyQt5-sip 12.13.0 /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
PySocks 1.7.1 /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
python-dateutil 2.9.0.post0 /home/salexan5/.local/lib/python3.12/site-packages pip
python-json-logger 2.0.7 /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
pytz 2024.1 /home/salexan5/.local/lib/python3.12/site-packages pip
PyYAML 6.0.1 /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
pyzmq 25.1.2 /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
qtconsole 5.5.1 /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
QtPy 2.4.1 /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
referencing 0.30.2 /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
requests 2.31.0 /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
rfc3339-validator 0.1.4 /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
rfc3986-validator 0.1.1 /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
rpds-py 0.10.6 /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
Send2Trash 1.8.2 /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
setuptools 69.5.1 /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages
sip 6.7.12 /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
six 1.16.0 /home/salexan5/.local/lib/python3.12/site-packages pip
sniffio 1.3.0 /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
soupsieve 2.5 /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
stack-data 0.2.0 /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
terminado 0.17.1 /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
tinycss2 1.2.1 /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
tornado 6.3.3 /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
traitlets 5.7.1 /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
typing_extensions 4.11.0 /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
tzdata 2024.1 /home/salexan5/.local/lib/python3.12/site-packages pip
urllib3 2.2.1 /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
wcwidth 0.2.5 /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
webencodings 0.5.1 /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
websocket-client 0.58.0 /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
wheel 0.43.0 /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages
widgetsnbextension 4.0.10 /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages conda
Note: you may need to restart the kernel to use updated packages. |
|
Info |
---|
Notice there are two paths: The library path of the conda environment that this default kernel is running from: /apps/s/jupyterlab/miniconda3/lib/python3.12/site-packages Your local home library path related to the version of Python within this kernel: /home/salexan5/.local/lib/python3.12/site-packages
|
...
Can I pip install
?
Info |
---|
Yes. And this will installed into your local home under the appropriate library/version folder. |
Code Block |
---|
# Before pip install:
[salexan5@mblog2 ~]$ ls .local/lib/python3.12/site-packages/
dateutil numpy-2.0.0.dist-info __pycache__ pytz six-1.16.0.dist-info tzdata
numpy numpy.libs python_dateutil-2.9.0.post0.dist-info pytz-2024.1.dist-info six.py tzdata-2024.1.dist-info |
Code Block |
---|
# Within Cell:
pip install nltk |
Expand |
---|
|
Code Block |
---|
Defaulting to user installation because normal site-packages is not writeable
Collecting nltk
Using cached nltk-3.8.1-py3-none-any.whl.metadata (2.8 kB)
Collecting click (from nltk)
Using cached click-8.1.7-py3-none-any.whl.metadata (3.0 kB)
Collecting joblib (from nltk)
Using cached joblib-1.4.2-py3-none-any.whl.metadata (5.4 kB)
Collecting regex>=2021.8.3 (from nltk)
Using cached regex-2024.7.24-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (40 kB)
Collecting tqdm (from nltk)
Downloading tqdm-4.66.5-py3-none-any.whl.metadata (57 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 57.6/57.6 kB 1.5 MB/s eta 0:00:00
Using cached nltk-3.8.1-py3-none-any.whl (1.5 MB)
Using cached regex-2024.7.24-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (790 kB)
Using cached click-8.1.7-py3-none-any.whl (97 kB)
Using cached joblib-1.4.2-py3-none-any.whl (301 kB)
Downloading tqdm-4.66.5-py3-none-any.whl (78 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 78.4/78.4 kB 2.9 MB/s eta 0:00:00
Installing collected packages: tqdm, regex, joblib, click, nltk
Successfully installed click-8.1.7 joblib-1.4.2 nltk-3.8.1 regex-2024.7.24 tqdm-4.66.5
Note: you may need to restart the kernel to use updated packages. |
|
Code Block |
---|
# After Installation:
# Notice the new packages installed.
[salexan5@mblog2 ~]$ ls .local/lib/python3.12/site-packages/
click joblib-1.4.2.dist-info numpy-2.0.0.dist-info pytz six-1.16.0.dist-info tzdata
click-8.1.7.dist-info nltk numpy.libs pytz-2024.1.dist-info six.py tzdata-2024.1.dist-info
dateutil nltk-3.8.1.dist-info __pycache__ regex tqdm
joblib numpy python_dateutil-2.9.0.post0.dist-info regex-2024.7.24.dist-info tqdm-4.66.5.dist-info |
Note |
---|
Remember: Any previously install Python packages under this location will be available to your notebook. Any installs/updates within the notebook will thus also affect any other Python environments you have that use this location and packages. This could cause issues with dependencies and versions. |
...
Can I Use Conda Within My Notebook?
Expand |
---|
|
Code Block |
---|
# packages in environment at /apps/s/jupyterlab/miniconda3:
#
# Name Version Build Channel
_libgcc_mutex 0.1 main
_openmp_mutex 5.1 1_gnu
anyio 4.2.0 py312h06a4308_0
...
zlib 1.2.13 h5eee18b_1
zstd 1.5.5 hc292b87_2
Note: you may need to restart the kernel to use updated packages. |
|
Expand |
---|
|
Code Block |
---|
Loading channels: done
# Name Version Build Channel
nltk 3.2.1 py27_0 conda-forge
nltk 3.2.1 py34_0 conda-forge
...
nltk 3.8.1 py39h06a4308_0 pkgs/main
nltk 3.8.1 pyhd8ed1ab_0 conda-forge
Note: you may need to restart the kernel to use updated packages. |
|
Expand |
---|
|
Code Block |
---|
Retrieving notices: ...working... done
Channels:
- conda-forge
- defaults
Platform: linux-64
Collecting package metadata (repodata.json): done
Solving environment: done
## Package Plan ##
environment location: /apps/s/jupyterlab/miniconda3
added / updated specs:
- nltk
The following packages will be downloaded:
package | build
---------------------------|-----------------
certifi-2024.7.4 | pyhd8ed1ab_0 156 KB conda-forge
nltk-3.8.1 | pyhd8ed1ab_0 1.0 MB conda-forge
regex-2023.10.3 | py312h5eee18b_0 386 KB
tqdm-4.66.5 | pyhd8ed1ab_0 87 KB conda-forge
------------------------------------------------------------
Total: 1.6 MB
The following NEW packages will be INSTALLED:
click conda-forge/noarch::click-8.1.7-unix_pyh707e725_0
colorama conda-forge/noarch::colorama-0.4.6-pyhd8ed1ab_0
joblib conda-forge/noarch::joblib-1.4.2-pyhd8ed1ab_0
nltk conda-forge/noarch::nltk-3.8.1-pyhd8ed1ab_0
regex pkgs/main/linux-64::regex-2023.10.3-py312h5eee18b_0
tqdm conda-forge/noarch::tqdm-4.66.5-pyhd8ed1ab_0
The following packages will be UPDATED:
ca-certificates pkgs/main::ca-certificates-2024.3.11-~ --> conda-forge::ca-certificates-2024.7.4-hbcca054_0
certifi pkgs/main/linux-64::certifi-2024.2.2-~ --> conda-forge/noarch::certifi-2024.7.4-pyhd8ed1ab_0
libgcc-ng pkgs/main::libgcc-ng-11.2.0-h1234567_1 --> conda-forge::libgcc-ng-14.1.0-h77fa898_0
libgomp pkgs/main::libgomp-11.2.0-h1234567_1 --> conda-forge::libgomp-14.1.0-h77fa898_0
openssl pkgs/main::openssl-3.0.13-h7f8727e_1 --> conda-forge::openssl-3.3.1-h4bc722e_2
The following packages will be SUPERSEDED by a higher-priority channel:
_libgcc_mutex pkgs/main::_libgcc_mutex-0.1-main --> conda-forge::_libgcc_mutex-0.1-conda_forge
_openmp_mutex pkgs/main::_openmp_mutex-5.1-1_gnu --> conda-forge::_openmp_mutex-4.5-2_gnu
...
Preparing transaction: done
Verifying transaction: failed
EnvironmentNotWritableError: The current user does not have write permissions to the target environment.
environment location: /apps/s/jupyterlab/miniconda3
uid: 10275911
gid: 10275911
Note: you may need to restart the kernel to use updated packages. |
|
Note |
---|
You can use conda commands within cells to inspect and search. You will not have permissions to conda install since you do not own this folder, so do not have permissions to install/update anything within it. |
Info |
---|
But, you should have permissions if you have created the kernel, own it, and thus have permissions. |
...
Jupyter Sessions
Issue: Jupyter Doesn’t Start?
...
...
...