Versions Compared

Old Version 21

changes.mady.by.user Lisa Stafford

Saved on

compared with

New Version Current

changes.mady.by.user Lisa Stafford

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

If we select the default Python 3 (ipykernel), we are presented with the file explorer showing our home directory as it’s rooted location. This means we can’t go up any further in system’s directory structure.

  • With our local root location for the notebook set to our $HOME, we are unable to see our /project and /gscratch directories on the cluster.

files.png

  • To expose these folders to the jupyter environment, create a symbolic link (aka shortcut) within our /home.

  • Instructions for creating a symbolic link may be found here or expanded in the cell to the right

Expand
titleSteps to create a symbolic link

  1. Open an ssh connection to the HPC cluster with:
    ssh your_username@clustername.arcc.uwyo.edu
    or open a shell through OnDemand:

  2. In the shell/terminal interface, create a symbolic link to your project (replacing project_name with the name of your project) with:

[~] ln -s /project/project_name/ project

  1. In the shell/terminal interface, create a symbolic link to your gscratch (replacing username with the your username on the HPC) with:

[~] ln -s /gscratch/username/ gscratch

...

Or, lets get clever:

Alternatively, we can simplify things by create a symbolic link from within our notebook using ! functionality (if we’re running an ipython kernel):

Image Modified

What Packages are Available in our Kernel?

...

Can we get outside of home now?

We can see new links to our external directories:

updatedfiles.pngImage Added

And now we can get to them:

Image Added

...

Getting information about packages?

What’s Installed - How to find out:

In our notebook, we can see which modules are available by opening a new cell with the + button.

In our cell box, set as “code” use the python import command, followed by a space, then hit tab to get a list of options.

Hitting tab after import runs autocomplete options for the import command. This list of options has populated all modules available to us in our jupyter notebook:

Image Modified

New Cell in our Notebook

...

Since we appear to have a large number of packages available in this environment, we’ll import one we expect to be there.

In our bottom-most cell, add to the import command by typing an import for a common package used in mathematic and multi dimensional matrix computations - numpy.

...

What’s Installed - Can we get a list in Python?

Yes. By running help('modules')
Note: the numpy library isn’t available

Image Added

...

What’s installed and how to use it: Python - help()

  • Generally, help ('modules <module_name>') will give us information on how to use the specific python library we’re importing as long as that library is installed.

  • Similar in functionality to the --help and man commands for shell.

...

What’s Installed - Can we get a list in R?

...

What’s Installed - Query a specific package in R?

...

What’s Installed and how to use it: In R - help()

...

We’ve confirmed the package we need is unavailable:

Our output results in an error:

Image Modified

  • The error means this particular module is not available in the kernel we have loaded, despite being a commonly used software package for researchers and computations.

  • While many packages were listed when we autocompleted an import command, most of them were installed as part of the jupyter installation and underlying OS environment.

  • Most software we’d need to perform even more simple and common activities for our research would still need to be installed or made available somehow. What are our options?

...

Option 1: Load a different kernel

Depending on the HPC’s native environment, you may have other kernels available.

Image Removed

To load a different kernel, we go to the Kernel option in our drop down menu then navigate to “Change kernel”.

To load a different kernel, we go to the

Image Added

Or not --->

MedBow currently has a minimal number of global kernels (purposefully).

Image Added

...

If this were an option, we’d see it in our dropdown list of kernels and could select a different one:

  • Kernel option in our drop down menu then navigate to “Change kernel”.

  • Select a different kernel

than the current one

  • , based on your own preference

  • Example shows others available, but on MB they may not be.

Image Modified

...

The new kernel is loaded as shown in the top right of our notebook.

  • If we rerun our 2 cells again, what happens this time?

Image Added

  • Depends on the kernel we loaded:

Image Modified

...

No available kernels have all the software I need - Now what?

Partially covered in python and conda materials, but short answer:

Best practice - Do NOT install the software directly from your jupyter kernel

...

...


Doing so can and frequently does eventually result in:

...

To be continued…

...

Next Steps

...