Reproducibility and Sharing
- Simon Alexander
Goal: Introduce how conda environments can be reproduced and shared.
Based on Managing Environments.
In the following examples, we assume you already have a miniconda3 module already loaded.
Clone an Environment
Existing Conda environments can be cloned (essentially copied).
[]$ conda create --help
...
options:
...
--clone ENV Create a new environment as a copy of an existing local environment.
...Example
[]$ conda create -p py_env2 --clone py_env
Retrieving notices: ...working... done
Source: /home/<username>/.conda/envs/py_env
Destination: /cluster/medbow/project/<project-name>/<username>/software/py_env2
Packages: 39
Files: 1
...
#
# To activate this environment, use
# $ conda activate /cluster/medbow/project/<project-name>/<username>/software/py_env2Notice: The source (environment being cloned) and the destination (new cloned environment) are listed.
In the above, we are using the -p option so the destination is within the current working directory, and cloning from an existing environment that was created within the default environment location - so only its name (not absolute path) is required.
Using Cloned Environment
Notice how you need to activate/use this cloned environment:
[]$ cat ~/.conda/environments.txt
/home/<username>/.conda/envs/py_env
/project/<project-name>/<username>/conda/envs/r_env
/cluster/medbow/project/<project-name>/<username>/software/slim_env_4.2.2
/cluster/medbow/project/<project-name>/<username>/software/py_env2
[]$ conda info --env
# conda environments:
#
base /apps/u/opt/linux/miniconda3/24.3.0
/cluster/medbow/project/<project-name>/<username>/software/py_env2
/cluster/medbow/project/<project-name>/<username>/software/slim_env_4.2.2
py_env /home/<username>/.conda/envs/py_env
r_env /project/<project-name>/<username>/conda/envs/r_envNeed to define the full path to activate.
[]$ conda activate /cluster/medbow/project/<project-name>/<username>/software/py_env2
(/cluster/medbow/project/<project-name>/<username>/software/py_env2) []$ Conda env command
The env sub-command provides the a process that can be used to export into a readable text file a description of the entire Conda environment that can then used to create (essentially import) to replicate.
[]$ conda env --help
usage: conda env [-h] command ...
positional arguments:
command
config Configure a conda environment.
create Create an environment based on an environment definition file.
export Export a given environment
list List the Conda environments.
remove Remove an environment.
update Update the current environment based on environment file.
options:
-h, --help Show this help message and exit.Export an Environment
Within an active environment we can export into a readable file a description of the entire Conda environment. This describes all the system related libraries as well as anything that that was explicitly conda install-ed.
[]$ conda activate r_env
(r_env) []$ conda env export > r_env.yml
(r_env) []$ conda deactivate
[]$ cat r_env.yml
name: r_env
channels:
- conda-forge
- defaults
dependencies:
- _libgcc_mutex=0.1=main
...
- zstd=1.5.5=hc292b87_2
prefix: /project/<project-name>/<username>/conda/envs/r_envImport an Environment
Using the env create sub-command we can then define a file to read/import that describes the environment we want to create - essentially cloning it.
[]$ cd /project/<project-name>/<username>/software/
[]$ conda env create -p r_env2 --file r_env.yml
...
#
# To activate this environment, use
#
# $ conda activate /cluster/medbow/project/<project-name>/<username>/software/r_env2
...