Conda and Pip Environments and Reproducibility
Goal: Introduce ideas and practices to assist in managing the reproducibility of environments created using Conda and Pip.
Create an Example Environment
To help demonstrate the various concepts, lets create the following environment:
[] cd /project/<project-name>/<username>/conda
[]$ module purge
[]$ module load miniconda3/24.3.0
[]$ conda create -p py_conda_pip_env python=3.12
[]$ conda activate /cluster/medbow/project/<project-name>/<username>/conda/py_conda_pip_env
(/cluster/medbow/project/<project-name>/<username>/conda/py_conda_pip_env) []$ python --version
Python 3.12.4
(/cluster/medbow/project/<project-name>/<username>/conda/py_conda_pip_env) []$ which python
/cluster/medbow/project/<project-name>/<username>/conda/py_conda_pip_env/bin/python
# Create a self contined environment i.e don't install anything into the local home folder.
(/cluster/medbow/project/<project-name>/<username>/conda/py_conda_pip_env) []$ export PYTHONUSERBASE=$CONDA_PREFIX
# Use conda to install: numpy
(/cluster/medbow/project/<project-name>/<username>/conda/py_conda_pip_env) []$ conda install numpy
# Use pip to install: nltk
# has dependencies: click, joblib, regex and tqdm
(/cluster/medbow/project/<project-name>/<username>/conda/py_conda_pip_env) []$ pip install nltk
What is Installed? conda list
The conda list
command will provide details of all packages as well as conda environment libraries:
(/cluster/medbow/project/<project-name>/<username>/conda/py_conda_pip_env) []$ conda list
...
click 8.1.7 pypi_0 pypi
joblib 1.4.2 pypi_0 pypi
nltk 3.8.1 pypi_0 pypi
numpy 2.0.1 py312h1103770_0 conda-forge
regex 2024.7.24 pypi_0 pypi
tqdm 4.66.4 pypi_0 pypi
What is Installed? pip list -v
The pip list
command will only list the explicitly installed pip python packages.
Look at the Installer column to identify what was installed via conda install
, pip install
, by default.
What is Installed? conda env export