Versions Compared

Old Version 6

changes.mady.by.user Simon Alexander

Saved on

compared with

New Version Current

changes.mady.by.user Simon Alexander

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Table of Contents
minLevel1
maxLevel1
outlinefalse
stylenone
typelist
printabletrue

...

What is Jupyter and What is a Kernel?

Note

  • Jupyter is a large umbrella project that covers many different software offerings and tools, including the popular Jupyter Notebook and JupyterLab web-based notebook authoring and editing applications.

  • Kernels are programming language specific processes that run independently and interact with the Jupyter Applications and their user interfaces.

    • They can be considered as the computational engine for notebooks.

    • The kernel process is a standalone process that runs in the background and executes the code that you write in your notebooks. The kernel process is responsible for running the code and returning the results to the frontend.

  • ARCC provides the Jupyter with OnDemand workshop.

...

General Process

Info

The general process involves updating the conda environment to include kernel related packages, and then configuring the kernel spec to allow it to be picked up by the Jupyter service.

...

Install the ipykernel package

Code Block
[salexan5@mblog1 ~]$ module purge
[]$ module load miniconda3/24.3.0
[salexan5@mblog1 ~]$ conda activate /cluster/medbow/project/arcc<project-name>/software/tensorflow/2.16
(/cluster/medbow/project/arcc<project-name>/software/tensorflow/2.16) [salexan5@mblog1 ~]$ export PYTHONUSERBASE=$CONDA_PREFIX
(/cluster/medbow/project/arcc<project-name>/software/tensorflow/2.16) [salexan5@mblog1 ~]$ conda install ipykernel
...
(/cluster/medbow/project/arcc<project-name>/software/tensorflow/2.16) [salexan5@mblog1 ~]$ conda deactivate

Created kernel related folder

Info

Installing the ipykernel package will create a kernel spec related folder that we can use.

This can be found under the Conda environment location, under: share/jupyter/kernels/ named python3

Code Block
[salexan5@mblog1 ~]$ ls /project/arccpython -m ipykernel install --user --name=TF2.16-local

(/cluster/medbow/project/<project-name>/software/tensorflow/2.16/share/jupyter/kernels/python3/

[salexan5@mblog1 ~]$ cd /project/arcc) []$ conda deactivate
[]$

...

The Related Kernel Details

Info

Using the python -m ipykernel install --user --name=TF2.16-local above will result in the following output of the form:

Code Block
() (/project/<project-name>/software/tensorflow/2.16) [salexan5@mblog1 2.16]$ cd share/jupyter/kernels/
[salexan5@mblog1 kernels]$ ls
python3

[salexan5@mblog1 kernels]$ ls python3
kernel.json  logo-32x32.png  logo-64x64.png  logo-svg.svg
Info

Rename the folder to something more appropriate:

Code Block
[salexan5@mblog1 kernels]$ mv python3 TF2.16

Configure Your Jupyter Environment

Note

If you haven’t used the Jupyter service, then you might not have, and thus will need to create the following folders:

python -m ipykernel install --user --name=TF2.16-local
Installed kernelspec TF2.16-local in /cluster/medbow/home/<username>/.local/share/jupyter/kernels/tf2.16-local
Info

Copy the created TF2.16 folder into your homeNotice: This file is created in the user's home folder, and can be accessed and viewed by the user:

Code Block
[salexan5@mblog1 kernels]$ cp -r TF2.16/ ~ls /cluster/medbow/home/<username>/.local/share/jupyter/kernels/
Info

Update the kernel.json file to:

  • give this kernel a unique display name.

Code Block
[salexan5@mblog1 kernels]$ ~tf2.16-local/
kernel.json  logo-32x32.png  logo-64x64.png  logo-svg.svg

[]$ cat /cluster/medbow/home/<username>/.local/share/jupyter/kernels/TF2tf2.16/
[salexan5@mblog1 tf2.16]$ cat -local/kernel.json
{
 "argv": [
  "/cluster/medbow/project/arcc<project-name>/software/tensorflow/2.16/bin/python",
  "-Xfrozen_modules=off",
  "-m",
  "ipykernel_launcher",
  "-f",
  "{connection_file}"
 ],
 "display_name": "TF2.16 (-local)",
 "language": "python",
 "metadata": {
  "debugger": true
 }
}
Info

Suggestion: Using “-local” will help to distinguish that this is a local kernel, based on a Conda environment, that the user has created, and not something provided by ARCC.

Info

Side Note: Installing the ipykernel package creates a kernel spec related folder within the Conda environment, under: share/jupyter/kernels/ named python3.

Running python -m ipykernel will copy and update this folder.

Note

Kernel names can only contain ASCII letters and numbers and these separators: - . _ (hyphen, period, and underscore)

Names can not use whitespace such as --name="TF2.16 (local)"

...

Start Jupyter

Info

From OnDemand start a Jupyter session.

Notice how the newly configured kernel is now available.

...

Within a Notebook

Within a Jupyter notebook cell try:

Code Block
import tensorflow as tf; 
print("TensorFlow Version: " + str( tf.__version__))

...

Prev

Create a Module File to Load Your Conda Environment

Workshop Home

Using Python, Conda and Pip on the Cluster

NextConda and Pip Environments and Reproducibility

Jupyter Python Packages and Issues

 

Add label