Goals:

• Provide logic and reasoning for using Conda environments, versus other available options

• Walk through the process for creating personalized kernels from a Conda environment

This training is available for download as a PDF here.

Why use Conda?

Also: Python dependency Hell: r/learnpython Dependency hell in Python has caused myself and every… [Rant] After going through literal HELL with Python’s package… Dependency HELL: r/learnpython - Reddit really getting frustrated with this new pip dependency hell Stack Overflow: Python dependency hell - compromise between… How can I escape from python environment HELL!? python - How to resolve the dependency hell to install ONE… dependency hell with python-apt - Stack Overflow Python’s dependency hell due to changes between versions… The Nine Circles of Python Dependency Hell

pip installs everything to one central consolidated location. Changes for one set of packages or software can unintentionally affect others.

Why make your own kernels? Why have multiple kernels?

Individual researchers on ARCC systems often need access to specific software and modules - specific to their focus of study or research.

While ARCC provides general “base” kernels, they typically lack all software packages specific to any one researcher’s needs. To solve this, researchers may perform installs within the jupyter notebook (with !pip for Python or install.packages() for R).

ARCC recommends against this practice because by default this will install the packages within the user’s $HOME creating a new set of issues in the future.

Issue 1: Storage Quotas

1. $HOME directories on HPC are usually relatively small compared to storage in /project or /gscratch. ARCC provides a default quotas on HPC for each user’s $HOME, but this can fill up quickly when performing frequent python pip or R install.package() installations, which leads to exceeding the storage quota in your $HOME directory.

   1. While installation locations can be redirected, several other configuration changes should be made to make them available and work appropriately.

   2. It’s far more straightforward to create Conda environments and redirect Conda installations with the -p flag (to specify installation path).

Issue 2: Software Conflicts

2. By default, local installs will install to one central location within your home directory (pip under ~/.local, install.packages() under ~/R)

   1. Package installations are separated by kernel software versions, but if conflicts exist within the overall install location, packages are overwritten to make the software and dependencies “fit” with your most recently requested installation. This can change installation and available packages for your user profile, breaking older installations and software that you still want to use.

   2. Installing in your $HOME directory makes packages and versions of software in your /home available to you regardless of whether you want them at the time, or not.

   3. This causes software conflicts between versions native to the HPC system, those you may want to load off and on with modules, and those you’ve installed in $HOME, meaning that loaded modules or native HPC software may not work properly or crash due to underlying dependencies can be superseded by packages in $HOME.

Dependency Hell is not Limited to Python: Examples in R

Also: R Dependency Hell: r/rprogramming Dependency Hell is NP-hard my R dependency hell… :r/ProgrammerHumor Stack Overflow - How to get out of the R dependency hell? R package dependency hell. How can I break free? R-bloggers - Reduce Dependency Hell! R - Dependency Management: I’m in HELL Dependency Hell in R. Worst of all the Hells How to get out of the R dependency hell? Updating R and avoiding dependency hell of updates to…

Kernels from Conda Environments

In the Conda module, we learned that conda allows software environments to be contained, meaning they do not conflict with one another, and can be loaded and unloaded so that they are exposed to you only when you want them to be.

In OnDemand Jupyter sessions you can launch a Jupyter session utilizing the software packages from a Conda environment you’ve set up yourself.

• This becomes very useful if you are using a package that requires extensions be installed in the environment that is launching the Jupyter session.

• To configure your environment so that it launches as you want, you should ensure that the appropriate packages are installed within the Conda environment.

• In this module, we will go through steps needed to correctly create your environment and configure it as a kernel available to you within your Jupyter sessions.

Exporting your Conda Environment to a Kernel (Python)

In these steps, we assume you’ve already created your desired Conda environment. To learn how to create a Conda environment, please see our Conda module.

You should also return to the documentation on exporting conda environments to a python kernel if you have not completed this training.

#open command line interface
module load miniconda3
conda activate <insert_environment_name_or_path_here>
conda install ipykernel
python -m ipykernel install --user --name=mypythonkernel

Exporting your Conda Environment to a Kernel (R)

In these steps, we assume you’ve already created your desired Conda environment. To learn how to create a Conda environment, please see our Conda module.

Note: A more in depth description with background for creating an R kernel is available at in our R and RStudio Training Materials in the section re: Create an R Kernel for Jupyter

# Step (1) open command line interface:
# Step (2) Load miniconda:
module load miniconda3
# Step (3) activate conda environment:
conda activate <insert_environment_name_or_path_here>
# Step (4) install kernel in conda environment:
conda install r-irkernel
# Step (5a) Note R location in conda:
which R
/<conda_env_path>/bin/R
# Step (5b) Launch R:
R
# Step (5c) Install kernel within R prompt:
> install.packages('IRkernel') 
> q()
#Step (5d) make a directory for your kernel:
mkdir ~/.local/share/jupyter/kernels/r4.4.0
#Step (5e) Copy kernelspec from conda to .local:
cp /<conda_env_name_or_path>/lib/R/library/IRkernel/kernelspec/* ~/.local/share/jupyter/kernels/r4.4.0/.

Running a Console Kernel

Open a new Jupyter Session

Select your new kernel from the dropdown list

Next Steps

Use the following link to provide feedback on this training: https://forms.gle/48Cy6mmY1gniN9Jy5 or use the QR code below.

Image Removed

Image Added