Conda and Pip Environments and Reproducibility

Owned by Simon Alexander
Last updated: Aug 21, 2024
10 min read

Goal: Introduce ideas and practices to assist in managing the reproducibility of environments created using Conda and Pip.

Create an Example Environment

To help demonstrate the various concepts, lets create the following environment:

[] cd /project/<project-name>/<username>/conda []$ module purge []$ module load miniconda3/24.3.0 []$ conda create -p py_conda_pip_env python=3.12 []$ conda activate /cluster/medbow/project/<project-name>/<username>/conda/py_conda_pip_env (/cluster/medbow/project/<project-name>/<username>/conda/py_conda_pip_env) []$ python --version Python 3.12.4 (/cluster/medbow/project/<project-name>/<username>/conda/py_conda_pip_env) []$ which python /cluster/medbow/project/<project-name>/<username>/conda/py_conda_pip_env/bin/python # Create a self contined environment i.e don't install anything into the local home folder. (/cluster/medbow/project/<project-name>/<username>/conda/py_conda_pip_env) []$ export PYTHONUSERBASE=$CONDA_PREFIX # Use conda to install: numpy (/cluster/medbow/project/<project-name>/<username>/conda/py_conda_pip_env) []$ conda install numpy # Use pip to install: nltk # has dependencies: click, joblib, regex and tqdm (/cluster/medbow/project/<project-name>/<username>/conda/py_conda_pip_env) []$ pip install nltk

What is Installed? conda list

The conda list command will provide details of all packages as well as conda environment libraries:

(/cluster/medbow/project/<project-name>/<username>/conda/py_conda_pip_env) []$ conda list ... click 8.1.7 pypi_0 pypi joblib 1.4.2 pypi_0 pypi nltk 3.8.1 pypi_0 pypi numpy 2.0.1 py312h1103770_0 conda-forge regex 2024.7.24 pypi_0 pypi tqdm 4.66.4 pypi_0 pypi
(/cluster/medbow/project/<project-name>/<username>/conda/py_conda_pip_env) []$ conda list # packages in environment at /cluster/medbow/project/<project-name>/<username>/conda/py_conda_pip_env: # # Name Version Build Channel _libgcc_mutex 0.1 conda_forge conda-forge _openmp_mutex 4.5 2_gnu conda-forge bzip2 1.0.8 h4bc722e_7 conda-forge ca-certificates 2024.7.4 hbcca054_0 conda-forge click 8.1.7 pypi_0 pypi joblib 1.4.2 pypi_0 pypi ld_impl_linux-64 2.40 hf3520f5_7 conda-forge libblas 3.9.0 23_linux64_openblas conda-forge libcblas 3.9.0 23_linux64_openblas conda-forge libexpat 2.6.2 h59595ed_0 conda-forge libffi 3.4.2 h7f98852_5 conda-forge libgcc-ng 14.1.0 h77fa898_0 conda-forge libgfortran-ng 14.1.0 h69a702a_0 conda-forge libgfortran5 14.1.0 hc5f4f2c_0 conda-forge libgomp 14.1.0 h77fa898_0 conda-forge liblapack 3.9.0 23_linux64_openblas conda-forge libnsl 2.0.1 hd590300_0 conda-forge libopenblas 0.3.27 pthreads_hac2b453_1 conda-forge libsqlite 3.46.0 hde9e2c9_0 conda-forge libstdcxx-ng 14.1.0 hc0a3c3a_0 conda-forge libuuid 2.38.1 h0b41bf4_0 conda-forge libxcrypt 4.4.36 hd590300_1 conda-forge libzlib 1.3.1 h4ab18f5_1 conda-forge ncurses 6.5 h59595ed_0 conda-forge nltk 3.8.1 pypi_0 pypi numpy 2.0.1 py312h1103770_0 conda-forge openssl 3.3.1 h4bc722e_2 conda-forge pip 24.0 pyhd8ed1ab_0 conda-forge python 3.12.4 h194c7f8_0_cpython conda-forge python_abi 3.12 4_cp312 conda-forge readline 8.2 h8228510_1 conda-forge regex 2024.7.24 pypi_0 pypi setuptools 71.0.4 pyhd8ed1ab_0 conda-forge tk 8.6.13 noxft_h4845f30_101 conda-forge tqdm 4.66.4 pypi_0 pypi tzdata 2024a h0c530f3_0 conda-forge wheel 0.43.0 pyhd8ed1ab_1 conda-forge xz 5.2.6 h166bdaf_0 conda-forge

What is Installed? pip list -v

The pip list command will only list the explicitly installed pip python packages.

Look at the Installer column to identify what was installed via conda install, pip install, by default.

What is Installed? conda env export

Prev

Jupyter Python Packages and Issues

Workshop Home

Using Python, Conda and Pip on the Cluster

Next

Python, Conda and Pip: Suggested Best Practices