...
In many ways, HPC computations and Jupyter notebooks don’t suit each other’s strengths. Their use cases and original intentions are very different. Jupyter Notebooks can be powerful development and collaboration tools, but they often aren’t suitable for long-running, computationally intensive workflows. Classic HPC runs in batches, with long running jobs through terminal access.
You can however use them together, and tools are available if you want to do this:
OpenOndemand - makes it easier to launch from HPC with requested resources.
ipython parallel (designed to integrate with MPI libraries)
dask
spark
In some cases the tools end up being more of a “workaround” and don’t really allow your computation to be run as one job inside the notebook. Instead what you may have is two separate jobs running simultaneously with information In these cases, you usually have your classic hpc jobs spawned from a jupyter session. These jobs run simultaneously with jupyter and information gets communicated between them.
...