Create a Shared TensorFlow Conda Environment

Owned by Simon Alexander
Last updated: Aug 23, 2024
9 min read

Goal: General process for creating and sharing a conda environment under a project.

Use Case

We want to create a software installation of TensorFlow (an end-to-end platform for machine learning) that can be shared across a project by all users.

From the documentation (as of time of writing):

  • The install documentation only details using a pip install.

  • TensorFlow is tested and supported on the following 64-bit systems: Python 3.8–3.11

General Process

[]$ module purge []$ module load miniconda3/24.3.0 []$ conda create -p 2.16 python=3.11 []$ conda activate /cluster/medbow/project/<project-name>/software/tensorflow/2.16 (/cluster/medbow/project/<project-name>/software/tensorflow/2.16) []$ export PYTHONUSERBASE=$CONDA_PREFIX (/cluster/medbow/project/<project-name>/software/tensorflow/2.16) []$ echo $PYTHONUSERBASE /cluster/medbow/project/<project-name>/software/tensorflow/2.16 (/cluster/medbow/project/<project-name>/software/tensorflow/2.16) []$ pip install tensorflow (/cluster/medbow/project/<project-name>/software/tensorflow/2.16) []$ python --version Python 3.11.9 (/cluster/medbow/project/<project-name>/software/tensorflow/2.16) []$ python ~/tf_test.py ... TensorFlow Version: 2.16.1 tf.Tensor(997.42975, shape=(), dtype=float32) (/cluster/medbow/project/<project-name>/software/tensorflow/2.16) []$ conda deactivate []$
import tensorflow as tf print("TensorFlow Version: " + str( tf.__version__)) print(tf.reduce_sum(tf.random.normal([1000, 1000])))
(/cluster/medbow/project/<project-name>/software/tensorflow/2.16) []$ pip install tensorflow Collecting tensorflow Downloading tensorflow-2.16.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (4.3 kB) Collecting absl-py>=1.0.0 (from tensorflow) Downloading absl_py-2.1.0-py3-none-any.whl.metadata (2.3 kB) Collecting astunparse>=1.6.0 (from tensorflow) Downloading astunparse-1.6.3-py2.py3-none-any.whl.metadata (4.4 kB) Collecting flatbuffers>=23.5.26 (from tensorflow) Downloading flatbuffers-24.3.25-py2.py3-none-any.whl.metadata (850 bytes) Collecting gast!=0.5.0,!=0.5.1,!=0.5.2,>=0.2.1 (from tensorflow) Downloading gast-0.5.4-py3-none-any.whl.metadata (1.3 kB) Collecting google-pasta>=0.1.1 (from tensorflow) Downloading google_pasta-0.2.0-py3-none-any.whl.metadata (814 bytes) Collecting h5py>=3.10.0 (from tensorflow) Downloading h5py-3.11.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (2.5 kB) Collecting libclang>=13.0.0 (from tensorflow) Downloading libclang-18.1.1-py2.py3-none-manylinux2010_x86_64.whl.metadata (5.2 kB) Collecting ml-dtypes~=0.3.1 (from tensorflow) Downloading ml_dtypes-0.3.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (20 kB) Collecting opt-einsum>=2.3.2 (from tensorflow) Downloading opt_einsum-3.3.0-py3-none-any.whl.metadata (6.5 kB) Collecting packaging (from tensorflow) Downloading packaging-24.1-py3-none-any.whl.metadata (3.2 kB) Collecting protobuf!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.20.3 (from tensorflow) Downloading protobuf-4.25.3-cp37-abi3-manylinux2014_x86_64.whl.metadata (541 bytes) Collecting requests<3,>=2.21.0 (from tensorflow) Downloading requests-2.32.3-py3-none-any.whl.metadata (4.6 kB) Requirement already satisfied: setuptools in ./2.16/lib/python3.11/site-packages (from tensorflow) (70.1.0) Collecting six>=1.12.0 (from tensorflow) Using cached six-1.16.0-py2.py3-none-any.whl.metadata (1.8 kB) Collecting termcolor>=1.1.0 (from tensorflow) Downloading termcolor-2.4.0-py3-none-any.whl.metadata (6.1 kB) Collecting typing-extensions>=3.6.6 (from tensorflow) Downloading typing_extensions-4.12.2-py3-none-any.whl.metadata (3.0 kB) Collecting wrapt>=1.11.0 (from tensorflow) Downloading wrapt-1.16.0-cp311-cp311-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (6.6 kB) Collecting grpcio<2.0,>=1.24.3 (from tensorflow) Downloading grpcio-1.64.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (3.3 kB) Collecting tensorboard<2.17,>=2.16 (from tensorflow) Downloading tensorboard-2.16.2-py3-none-any.whl.metadata (1.6 kB) Collecting keras>=3.0.0 (from tensorflow) Downloading keras-3.3.3-py3-none-any.whl.metadata (5.7 kB) Collecting tensorflow-io-gcs-filesystem>=0.23.1 (from tensorflow) Downloading tensorflow_io_gcs_filesystem-0.37.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (14 kB) Collecting numpy<2.0.0,>=1.23.5 (from tensorflow) Downloading numpy-1.26.4-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (61 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 61.0/61.0 kB 2.1 MB/s eta 0:00:00 Requirement already satisfied: wheel<1.0,>=0.23.0 in ./2.16/lib/python3.11/site-packages (from astunparse>=1.6.0->tensorflow) (0.43.0) Collecting rich (from keras>=3.0.0->tensorflow) Downloading rich-13.7.1-py3-none-any.whl.metadata (18 kB) Collecting namex (from keras>=3.0.0->tensorflow) Downloading namex-0.0.8-py3-none-any.whl.metadata (246 bytes) Collecting optree (from keras>=3.0.0->tensorflow) Downloading optree-0.11.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (45 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 45.4/45.4 kB 1.6 MB/s eta 0:00:00 Collecting charset-normalizer<4,>=2 (from requests<3,>=2.21.0->tensorflow) Downloading charset_normalizer-3.3.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (33 kB) Collecting idna<4,>=2.5 (from requests<3,>=2.21.0->tensorflow) Downloading idna-3.7-py3-none-any.whl.metadata (9.9 kB) Collecting urllib3<3,>=1.21.1 (from requests<3,>=2.21.0->tensorflow) Downloading urllib3-2.2.2-py3-none-any.whl.metadata (6.4 kB) Collecting certifi>=2017.4.17 (from requests<3,>=2.21.0->tensorflow) Downloading certifi-2024.6.2-py3-none-any.whl.metadata (2.2 kB) Collecting markdown>=2.6.8 (from tensorboard<2.17,>=2.16->tensorflow) Downloading Markdown-3.6-py3-none-any.whl.metadata (7.0 kB) Collecting tensorboard-data-server<0.8.0,>=0.7.0 (from tensorboard<2.17,>=2.16->tensorflow) Downloading tensorboard_data_server-0.7.2-py3-none-manylinux_2_31_x86_64.whl.metadata (1.1 kB) Collecting werkzeug>=1.0.1 (from tensorboard<2.17,>=2.16->tensorflow) Downloading werkzeug-3.0.3-py3-none-any.whl.metadata (3.7 kB) Collecting MarkupSafe>=2.1.1 (from werkzeug>=1.0.1->tensorboard<2.17,>=2.16->tensorflow) Downloading MarkupSafe-2.1.5-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (3.0 kB) Collecting markdown-it-py>=2.2.0 (from rich->keras>=3.0.0->tensorflow) Downloading markdown_it_py-3.0.0-py3-none-any.whl.metadata (6.9 kB) Collecting pygments<3.0.0,>=2.13.0 (from rich->keras>=3.0.0->tensorflow) Downloading pygments-2.18.0-py3-none-any.whl.metadata (2.5 kB) Collecting mdurl~=0.1 (from markdown-it-py>=2.2.0->rich->keras>=3.0.0->tensorflow) Downloading mdurl-0.1.2-py3-none-any.whl.metadata (1.6 kB) Downloading tensorflow-2.16.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (589.8 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 589.8/589.8 MB 12.5 MB/s eta 0:00:00 Downloading absl_py-2.1.0-py3-none-any.whl (133 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 133.7/133.7 kB 6.2 MB/s eta 0:00:00 Downloading astunparse-1.6.3-py2.py3-none-any.whl (12 kB) Downloading flatbuffers-24.3.25-py2.py3-none-any.whl (26 kB) Downloading gast-0.5.4-py3-none-any.whl (19 kB) Downloading google_pasta-0.2.0-py3-none-any.whl (57 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 57.5/57.5 kB 2.0 MB/s eta 0:00:00 Downloading grpcio-1.64.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (5.6 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 5.6/5.6 MB 107.4 MB/s eta 0:00:00 Downloading h5py-3.11.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (5.4 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 5.4/5.4 MB 104.4 MB/s eta 0:00:00 Downloading keras-3.3.3-py3-none-any.whl (1.1 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.1/1.1 MB 37.0 MB/s eta 0:00:00 Downloading libclang-18.1.1-py2.py3-none-manylinux2010_x86_64.whl (24.5 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 24.5/24.5 MB 89.4 MB/s eta 0:00:00 Downloading ml_dtypes-0.3.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.2 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.2/2.2 MB 60.9 MB/s eta 0:00:00 Downloading numpy-1.26.4-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (18.3 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 18.3/18.3 MB 91.6 MB/s eta 0:00:00 Downloading opt_einsum-3.3.0-py3-none-any.whl (65 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 65.5/65.5 kB 2.9 MB/s eta 0:00:00 Downloading protobuf-4.25.3-cp37-abi3-manylinux2014_x86_64.whl (294 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 294.6/294.6 kB 15.6 MB/s eta 0:00:00 Downloading requests-2.32.3-py3-none-any.whl (64 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 64.9/64.9 kB 2.6 MB/s eta 0:00:00 Using cached six-1.16.0-py2.py3-none-any.whl (11 kB) Downloading tensorboard-2.16.2-py3-none-any.whl (5.5 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 5.5/5.5 MB 86.2 MB/s eta 0:00:00 Downloading tensorflow_io_gcs_filesystem-0.37.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (5.1 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 5.1/5.1 MB 85.7 MB/s eta 0:00:00 Downloading termcolor-2.4.0-py3-none-any.whl (7.7 kB) Downloading typing_extensions-4.12.2-py3-none-any.whl (37 kB) Downloading wrapt-1.16.0-cp311-cp311-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (80 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 80.7/80.7 kB 3.2 MB/s eta 0:00:00 Downloading packaging-24.1-py3-none-any.whl (53 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 54.0/54.0 kB 2.1 MB/s eta 0:00:00 Downloading certifi-2024.6.2-py3-none-any.whl (164 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 164.4/164.4 kB 7.5 MB/s eta 0:00:00 Downloading charset_normalizer-3.3.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (140 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 140.3/140.3 kB 6.1 MB/s eta 0:00:00 Downloading idna-3.7-py3-none-any.whl (66 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 66.8/66.8 kB 2.7 MB/s eta 0:00:00 Downloading Markdown-3.6-py3-none-any.whl (105 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 105.4/105.4 kB 4.5 MB/s eta 0:00:00 Downloading tensorboard_data_server-0.7.2-py3-none-manylinux_2_31_x86_64.whl (6.6 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 6.6/6.6 MB 102.1 MB/s eta 0:00:00 Downloading urllib3-2.2.2-py3-none-any.whl (121 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 121.4/121.4 kB 4.7 MB/s eta 0:00:00 Downloading werkzeug-3.0.3-py3-none-any.whl (227 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 227.3/227.3 kB 10.1 MB/s eta 0:00:00 Downloading namex-0.0.8-py3-none-any.whl (5.8 kB) Downloading optree-0.11.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (312 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 312.0/312.0 kB 15.7 MB/s eta 0:00:00 Downloading rich-13.7.1-py3-none-any.whl (240 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 240.7/240.7 kB 12.1 MB/s eta 0:00:00 Downloading markdown_it_py-3.0.0-py3-none-any.whl (87 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 87.5/87.5 kB 3.5 MB/s eta 0:00:00 Downloading MarkupSafe-2.1.5-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (28 kB) Downloading pygments-2.18.0-py3-none-any.whl (1.2 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.2/1.2 MB 47.0 MB/s eta 0:00:00 Downloading mdurl-0.1.2-py3-none-any.whl (10.0 kB) Installing collected packages: namex, libclang, flatbuffers, wrapt, urllib3, typing-extensions, termcolor, tensorflow-io-gcs-filesystem, tensorboard-data-server, six, pygments, protobuf, packaging, numpy, mdurl, MarkupSafe, markdown, idna, grpcio, gast, charset-normalizer, certifi, absl-py, werkzeug, requests, optree, opt-einsum, ml-dtypes, markdown-it-py, h5py, google-pasta, astunparse, tensorboard, rich, keras, tensorflow Successfully installed MarkupSafe-2.1.5 absl-py-2.1.0 astunparse-1.6.3 certifi-2024.6.2 charset-normalizer-3.3.2 flatbuffers-24.3.25 gast-0.5.4 google-pasta-0.2.0 grpcio-1.64.1 h5py-3.11.0 idna-3.7 keras-3.3.3 libclang-18.1.1 markdown-3.6 markdown-it-py-3.0.0 mdurl-0.1.2 ml-dtypes-0.3.2 namex-0.0.8 numpy-1.26.4 opt-einsum-3.3.0 optree-0.11.0 packaging-24.1 protobuf-4.25.3 pygments-2.18.0 requests-2.32.3 rich-13.7.1 six-1.16.0 tensorboard-2.16.2 tensorboard-data-server-0.7.2 tensorflow-2.16.1 tensorflow-io-gcs-filesystem-0.37.0 termcolor-2.4.0 typing-extensions-4.12.2 urllib3-2.2.2 werkzeug-3.0.3 wrapt-1.16.0

Notice Where Related Packages Were Installed

Since we set export PYTHONUSERBASE=$CONDA_PREFIX there is nothing within the home folder under: .local/lib/ i.e. no Python3.11 folder:

Everything is self contained within the conda environment itself:

How is this Shared?

Since the environment was created under a project, any one who is part of this project can also access/activate this environment.

Assume the user user02 is part of the <project-name> project.

Exercise: TensorFlow CPU vs GPU

Prev

Pip Install within a Conda Environment

Workshop Home

Using Python, Conda and Pip on the Cluster

Next

Create a Module File to Load Your Conda Environment