Create a Shared TensorFlow Conda Environment
Goal: General process for creating and sharing a conda environment under a project.
Use Case
We want to create a software installation of TensorFlow (an end-to-end platform for machine learning) that can be shared across a project by all users.
From the documentation (as of time of writing):
The install documentation only details using a
pip install
.TensorFlow is tested and supported on the following 64-bit systems: Python 3.8–3.11
General Process
[]$ module purge
[]$ module load miniconda3/24.3.0
[]$ conda create -p 2.16 python=3.11
[]$ conda activate /cluster/medbow/project/<project-name>/software/tensorflow/2.16
(/cluster/medbow/project/<project-name>/software/tensorflow/2.16) []$ export PYTHONUSERBASE=$CONDA_PREFIX
(/cluster/medbow/project/<project-name>/software/tensorflow/2.16) []$ echo $PYTHONUSERBASE
/cluster/medbow/project/<project-name>/software/tensorflow/2.16
(/cluster/medbow/project/<project-name>/software/tensorflow/2.16) []$ pip install tensorflow
(/cluster/medbow/project/<project-name>/software/tensorflow/2.16) []$ python --version
Python 3.11.9
(/cluster/medbow/project/<project-name>/software/tensorflow/2.16) []$ python ~/tf_test.py
...
TensorFlow Version: 2.16.1
tf.Tensor(997.42975, shape=(), dtype=float32)
(/cluster/medbow/project/<project-name>/software/tensorflow/2.16) []$ conda deactivate
[]$
Notice Where Related Packages Were Installed
Since we set export PYTHONUSERBASE=$CONDA_PREFIX
there is nothing within the home folder under: .local/lib/
i.e. no Python3.11
folder:
Everything is self contained within the conda environment itself:
How is this Shared?
Since the environment was created under a project, any one who is part of this project can also access/activate this environment.
Assume the user user02
is part of the <project-name>
project.
Exercise: TensorFlow CPU vs GPU