Goal: Introduce ideas and practices to assist in managing the reproducibility of environments created using Conda and Pip.
Create an Example Environment
To help demonstrate the various concepts, lets create the following environment:
[] cd /project/<project-name>/<username>/conda []$ module load miniconda3/24.3.0 []$ conda create -p py_conda_pip_env python=3.12 []$ conda activate /cluster/medbow/project/<project-name>/<username>/conda/py_conda_pip_env (/cluster/medbow/project/<project-name>/<username>/conda/py_conda_pip_env) []$ python --version Python 3.12.4 (/cluster/medbow/project/<project-name>/<username>/conda/py_conda_pip_env) []$ which python /cluster/medbow/project/<project-name>/<username>/conda/py_conda_pip_env/bin/python # Create a self contined environment i.e don't install anything into the local home folder. (/cluster/medbow/project/<project-name>/<username>/conda/py_conda_pip_env) []$ export PYTHONUSERBASE=$CONDA_PREFIX # Use conda to install: numpy (/cluster/medbow/project/<project-name>/<username>/conda/py_conda_pip_env) []$ conda install numpy # Use pip to install: nltk # has dependencies: click, joblib, regex and tqdm (/cluster/medbow/project/<project-name>/<username>/conda/py_conda_pip_env) []$ pip install nltk
What is Installed? conda list
The conda list
command will provide details of all packages as well as conda environment libraries:
(/cluster/medbow/project/<project-name>/<username>/conda/py_conda_pip_env) []$ conda list ... click 8.1.7 pypi_0 pypi joblib 1.4.2 pypi_0 pypi nltk 3.8.1 pypi_0 pypi numpy 2.0.1 py312h1103770_0 conda-forge regex 2024.7.24 pypi_0 pypi tqdm 4.66.4 pypi_0 pypi
What is Installed? pip list -v
The pip list
command will only list the explicitly installed pip python packages.
Look at the Installer column to identify what was installed via conda install
, pip install
, by default.
(/cluster/medbow/project/<project-name>/<username>/conda/py_conda_pip_env) []$ pip list -v Package Version Location Installer ---------- --------- ----------------------------------------------------------------------------------------- --------- click 8.1.7 /cluster/medbow/project/<project-name>/<username>/conda/py_conda_pip_env/lib/python3.12/site-packages pip joblib 1.4.2 /cluster/medbow/project/<project-name>/<username>/conda/py_conda_pip_env/lib/python3.12/site-packages pip nltk 3.8.1 /cluster/medbow/project/<project-name>/<username>/conda/py_conda_pip_env/lib/python3.12/site-packages pip numpy 2.0.1 /cluster/medbow/project/<project-name>/<username>/conda/py_conda_pip_env/lib/python3.12/site-packages conda pip 24.0 /cluster/medbow/project/<project-name>/<username>/conda/py_conda_pip_env/lib/python3.12/site-packages regex 2024.7.24 /cluster/medbow/project/<project-name>/<username>/conda/py_conda_pip_env/lib/python3.12/site-packages pip setuptools 71.0.4 /cluster/medbow/project/<project-name>/<username>/conda/py_conda_pip_env/lib/python3.12/site-packages tqdm 4.66.4 /cluster/medbow/project/<project-name>/<username>/conda/py_conda_pip_env/lib/python3.12/site-packages pip wheel 0.43.0 /cluster/medbow/project/<project-name>/<username>/conda/py_conda_pip_env/lib/python3.12/site-packages
Check under the python related lib
folder.
What is Installed? conda env export
The output of the conda env export
command you can use to recreate.
Identify what was install via conda install
vs pip install
:
(/cluster/medbow/project/<project-name>/<username>/conda/py_conda_pip_env) []$ conda env export > env.yml (/cluster/medbow/project/<project-name>/<username>/conda/py_conda_pip_env) []$ cat env.yml name: /cluster/medbow/project/<project-name>/<username>/conda/py_conda_pip_env ... dependencies: ... - numpy=2.0.1=py312h1103770_0 ... - python=3.12.4=h194c7f8_0_cpython ... - pip: - click==8.1.7 - joblib==1.4.2 - nltk==3.8.1 - regex==2024.7.24 - tqdm==4.66.4 prefix: /cluster/medbow/project/<project-name>/<username>/conda/py_conda_pip_env
Prev | Workshop Home | Next |