Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Goal: Create Demonstrate a basic conda environment creation workflow by creating a Python environment that contains the numpy and pandas packages.

Table of Contents
stylenone

...

General Process

[salexan5@mblog1 ~]$ module load miniconda3/24.3.0 [salexan5@mblog1 ~
Code Block
Info

Below demonstrates the general process for:

  • Setting up your environment to use Conda via a module load.

  • How to create a conda environment.

  • How to activate/start your environment and install packages.

  • How to then use what you’ve installed.

  • And then how to deactivate/finish using your environment.

Code Block
[]$ module purge
[]$ module load miniconda3/24.3.0
[]$ conda search python
[salexan5@mblog1 ~]$ conda create -n py_env
[salexan5@mblog1 ~]$ conda activate py_env
(py_env) [salexan5@mblog1 ~]$ conda install python=3.12.4
(py_env) [salexan5@mblog1 ~]$ python --version
Python 3.12.4
(py_env) [salexan5@mblog1 ~]$ conda search numpy
(py_env) [salexan5@mblog1 ~]$ conda install numpy
(py_env) [salexan5@mblog1 ~]$ python -c "import numpy; print(numpy.__version__)"
1.26.4
(py_env) [salexan5@mblog1 ~]$ conda deactivate
[salexan5@mblog1 ~]$

...

Search for Packages

[salexan5@mblog1 ~
Code Block
Info

Conda enables you look up and search for packages and what versions are available.

Code Block
[]$ conda search python
Loading channels: done
# Name                       Version           Build  Channel
python                        2.7.13     hac47a24_15  pkgs/main
...
python                        3.12.3      h996f2a0_1  pkgs/main
python                        3.12.4      h5148396_1  pkgs/main
Info

Notice: Although Python version 2 has been deprecated, it is still used for old packages/modules/scripts. You can create a conda environment that provides this old, no longer supported version.

This is how ARCC provides this version on the cluster.

Code Block
(py_env) [salexan5@mblog1 ~]$ conda search numpy
Loading channels: done
# Name                       Version           Build  Channel
numpy                          1.9.3 py27_nomklhbee5d10_3  pkgs/main
...
numpy                         1.26.4  py39heeff2f4_0  pkgs/main

Create an Environment

Code Block
[salexan5@mblog1 ~]$ conda create -n py_env
Channels:
 - defaults
Platform: linux-64
Collecting package metadata (repodata.json): done
Solving environment: done

## Package Plan ##
  environment location: /home/salexan5/.conda/envs/py_env

Proceed ([y]/n)? y
Info

Note the location that the environment will be saved to: environment location: /home/salexan5/.conda/envs/py_env

Create an Environment: Proceed

Code Block
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
#
# To activate this environment, use
#
#     $ conda activate py_env
#
# To deactivate an active environment, use
#
#     $ conda deactivate
Info

Note the command required to activate your environment: conda activate py_env

This will be required every time you wish to use it.

Activate an Environment

Code Block
[salexan5@mblog1 ~]$ conda activate py_env
(py_env) [salexan5@mblog1 ~]$ 
Info

Note how the command line prompt has changed once the environment has been activated: (py_env) [...]

This indicates the name of the conda environment that is currently active.

Conda Install a Version of Python

Code Block
(py_env) [salexan5@mblog1 ~]$ conda install python=3.12.4

...

titleOutput of what is happening:

...

Create an Environment

Info

Use the create sub command to create your environment.

The -n option will create your environment (which is just a folder) in the default configured location.

Out the box, this will be under: /home/<username>/.conda/envs/

Code Block
[]$ conda create -n py_env
Channels:
 - defaults
Platform: linux-64
Collecting package metadata (repodata.json): done
Solving environment: done

## Package Plan ##
  environment location: /home/salexan5/.conda/envs/py_env

Proceed ([y]/n)? y
Info

Note the location that the environment will be saved to: environment location: /home/salexan5/.conda/envs/py_env

...

Create an Environment: Proceed

Code Block
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
#
# To activate this environment, use
#
#     $ conda activate py_env
#
# To deactivate an active environment, use
#
#     $ conda deactivate
Info

Note the command required to activate your environment: conda activate py_env

This will be required every time you wish to use it.

...

Activate an Environment

Info

Before you can use a conda environment it must be activated.

Every time you want to use it, you must remember to activate it.

Code Block
[]$ conda activate py_env
(py_env) []$ 
Info

Note: See how the command line prompt has changed once the environment has been activated: (py_env) [...]

This indicates the name of the conda environment that is currently active.

...

Conda Install a Version of Python

Info

Once a conda environment is active, you can start to use it, and one aspect of this is to install conda packages.

Code Block
(py_env) []$ conda install python=3.12.4
Expand
titleOutput of what is happening:
Code Block
(py_env) []$ conda install python=3.12.4
Channels:
 - defaults
Platform: linux-64
Collecting package metadata (repodata.json): done
Solving environment: done

## Package Plan ##

  environment location: /home/salexan5/.conda/envs/py_env

  added / updated specs:
    - python=3.12.4

The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    pip-24.0                   |  py312h06a4308_0         3.3 MB
    setuptools-69.5.1          |  py312h06a4308_0         1.3 MB
    wheel-0.43.0               |  py312h06a4308_0         142 buildKB
    ---------------------------|-----------------
    pip-24.0----------------
                  |  py312h06a4308_0         3.3 MB     setuptools-69.5.1        Total:  |  py312h06a4308_0         1.34.7 MB

The following  wheel-0.43.0   NEW packages will be INSTALLED:

  _libgcc_mutex        |  py312h06a4308_0         142 KB
    ------------------------------------------------------------
  pkgs/main/linux-64::_libgcc_mutex-0.1-main
  _openmp_mutex      pkgs/main/linux-64::_openmp_mutex-5.1-1_gnu
  bzip2              pkgs/main/linux-64::bzip2-1.0.8-h5eee18b_6
  ca-certificates     pkgs/main/linux-64::ca-certificates-2024.3.11-h06a4308_0
  expat       Total:         4.7 MB

The following NEW packages will be INSTALLED:

  _libgcc_mutex      pkgs/main/linux-64::_libgcc_mutex-0.1-main
  _openmp_mutexpkgs/main/linux-64::expat-2.6.2-h6a678d5_0
  ld_impl_linux-64   pkgs/main/linux-64::ld_impl_linux-64-2.38-h1181459_1
  libffi             pkgs/main/linux-64::_openmp_mutex-5.1-1_gnu
  bzip2 libffi-3.4.4-h6a678d5_1
  libgcc-ng             pkgs/main/linux-64::bzip2-1libgcc-ng-11.2.0.8-h5eee18b_6-h1234567_1
  libgomp        ca-certificates    pkgs/main/linux-64::calibgomp-certificates-202411.32.110-h06a4308h1234567_01
  expat    libstdcxx-ng          pkgs/main/linux-64::expatlibstdcxx-ng-11.2.6.20-h6a678d5_0h1234567_1
  libuuid         ld_impl_linux-64   pkgs/main/linux-64::ld_impl_linux-64-2.38-h1181459_1::libuuid-1.41.5-h5eee18b_0
  libffi ncurses            pkgs/main/linux-64::libffincurses-36.4.4-h6a678d5_10
  openssl  libgcc-ng          pkgs/main/linux-64::libgccopenssl-ng-113.20.014-h1234567_1h5eee18b_0
  pip    libgomp            pkgs/main/linux-64::libgomppip-1124.2.0-h1234567_1py312h06a4308_0
  python      libstdcxx-ng       pkgs/main/linux-64::libstdcxxpython-ng-113.212.04-h1234567h5148396_1
  libuuidreadline            pkgs/main/linux-64::libuuidreadline-18.41.52-h5eee18b_0
  ncurses   setuptools         pkgs/main/linux-64::ncursessetuptools-669.5.41-h6a678d5py312h06a4308_0
  opensslsqlite             pkgs/main/linux-64::opensslsqlite-3.045.143-h5eee18b_0
  piptk                 pkgs/main/linux-64::piptk-248.6.014-py312h06a4308h39e8969_0
  pythontzdata             pkgs/main/linux-64noarch::python-3.12.4-h5148396_1tzdata-2024a-h04d1e81_0
  wheel   readline           pkgs/main/linux-64::readlinewheel-80.43.20-h5eee18bpy312h06a4308_0
  xz      setuptools           pkgs/main/linux-64::setuptoolsxz-695.54.16-py312h06a4308h5eee18b_01
  zlib  sqlite             pkgs/main/linux-64::sqlitezlib-31.452.313-h5eee18b_01

 tk Proceed ([y]/n)? y

Downloading  and Extracting Packages:

Preparing transaction: done
Verifying transaction: done
Executing  pkgs/main/linux-64::tk-8.6.14-h39e8969_0
  tzdata             pkgs/main/noarch::tzdata-2024a-h04d1e81_0
  wheeltransaction: done
Info

Note all the addition dependencies/libraries that are being installed into the environment

Lets check the version installed, within are active environment.

Code Block
(py_env) []$ python --version
Python 3.12.4

...

Conda Install the numpy Package

Info

Let’s first check if numpy is available as a package, and if it is, which versions are available.

Code Block
(py_env) []$ conda search numpy
Loading channels: done
# Name                    
pkgs/main/linux-64::wheel-0.43.0-py312h06a4308_0
   
xz
Version           Build  Channel
numpy   
pkgs/main/linux-64::xz-5.4.6-h5eee18b_1
   
zlib
               
pkgs/main/linux-64::zlib-1.2.13-h5eee18b_1
  
Proceed
 
([y]/n)?
 
y
 
Downloading and Extracting Packages: Preparing transaction: done Verifying transaction: done Executing transaction: done
Info

Note all the addition dependencies/libraries that are being installed into the environment

Lets check the version installed, within are active environment.

Code Block
(py_env) [salexan5@mblog1 ~]$ python --version
Python 3.12.4

Conda Install the numpy Package

1.9.3 py27_nomklhbee5d10_3  pkgs/main
...
numpy                         1.26.4  py39heeff2f4_0  pkgs/main

Code Block
(py_env) [

...

]$ conda install numpy
Expand
titleOutput of what is happening:
Code Block
(py_env) [

...

]$ conda install numpy
Channels:
 - defaults
Platform: linux-64
Collecting package metadata (repodata.json): done
Solving environment: done

## Package Plan ##

  environment location: /home/salexan5/.conda/envs/py_env

  added / updated specs:
    - numpy

The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    mkl-service-2.4.0          |  py312h5eee18b_1          66 KB
    mkl_fft-1.3.8              |  py312h5eee18b_0         204 KB
    mkl_random-1.2.4           |  py312hdb19cb5_0         284 KB
    numpy-1.26.4               |  py312hc5e2394_0          11 KB
    numpy-base-1.26.4          |  py312h0da6c21_0         7.7 MB
    ------------------------------------------------------------
                                           Total:         8.2 MB

The following NEW packages will be INSTALLED:

  blas               pkgs/main/linux-64::blas-1.0-mkl
  intel-openmp       pkgs/main/linux-64::intel-openmp-2023.1.0-hdb19cb5_46306
  mkl                pkgs/main/linux-64::mkl-2023.1.0-h213fc3f_46344
  mkl-service        pkgs/main/linux-64::mkl-service-2.4.0-py312h5eee18b_1
  mkl_fft            pkgs/main/linux-64::mkl_fft-1.3.8-py312h5eee18b_0
  mkl_random         pkgs/main/linux-64::mkl_random-1.2.4-py312hdb19cb5_0
  numpy              pkgs/main/linux-64::numpy-1.26.4-py312hc5e2394_0
  numpy-base         pkgs/main/linux-64::numpy-base-1.26.4-py312h0da6c21_0
  tbb                pkgs/main/linux-64::tbb-2021.8.0-hdb19cb5_0

Proceed ([y]/n)? y

Downloading and Extracting 

...

Packages:

Preparing transaction: done
Verifying transaction: done

...

Executing transaction: done

...

Info

Again, note all the addition dependencies/libraries that are being installed into the environment

If no package version is defined, typically it’ll install the latest version.

Lets check the version installed, within are active environment.

...

Info

(Again) Note: See all the addition dependencies/libraries that are being installed into the environment

If no package version is defined, typically it’ll install the latest version.

Lets check the version installed, within are active environment.

Code Block
(py_env) []$ python -c "import numpy; print(numpy.__version__)"
1.26.4
Note
  • Packages are being updated on a frequent basis.

  • When this workshop was created numpy/1.26.4 was the latest version.

  • As of 20240815, version 2.0.1 is available, and in a few weeks a newer version will likely be available.

  • If you do not define a version, then the latest will be installed.

...

Conda Deactivate your Environment

Code Block
(py_env) [salexan5@mblog1 ~]$ conda deactivate
[salexan5@mblog1 ~]$
Info

Note how the command line prompt has changed, reverting back to before being activated: [...]

The conda environment is no longer activate and can not be used.

Code Block
[salexan5@mblog1 ~]$ python --version
Python 3.12.2

[salexan5@mblog1 ~]$ python -c "import numpy; print(numpy.__version__)"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'numpy'

...

(Re-)Using the Environment

Info

To re-use a conda environment, you must activate it again.

But, you do not have to re-create it. Creation only happens once.

Code Block
[salexan5@mblog1 ~[]$ module purge
[]$ module load miniconda3/24.3.0
[salexan5@mblog1 ~]$ conda activate py_env
(py_env) [salexan5@mblog1 ~]$ python --version
Python 3.12.4
(py_env) [salexan5@mblog1 ~]$ python -c "import numpy; print(numpy.__version__)"
1.26.4
(py_env) [salexan5@mblog1 ~]$ conda deactivate
[salexan5@mblog1 ~]$
Info

Note how the conda environment must be activated to use it.

Adding to an Existing Environment

Code Block
[salexan5@mblog1 ~

: The conda environment must be activated to use it.

...

Adding to an Existing Environment

Info

Although we only need to create a conda environment once, we can add/update it every time we use it.

Let’s go back into are existing conda environment and add the pandas package.

Code Block
[]$ module purge
[]$ module load miniconda3/24.3.0
[salexan5@mblog1 ~]$ conda activate py_env
(py_env) [salexan5@mblog1 ~]$ conda install pandas
(py_env) [salexan5@mblog1 ~]$ python py_test.py
Python: 3.12.4 | packaged by Anaconda, Inc. | (main, Jun 18 2024, 15:12:24) [GCC 11.2.0]
Numpy: 1.26.4
Pandas: 2.2.2
(py_env) [salexan5@mblog1 ~]$ conda deactivate
Expand
titlepy_test.py
Code Block
import sys
import numpy
import pandas

print("Python: " + str(sys.version))
print("Numpy: " + str(numpy.__version__))
print("Pandas: " + str(pandas.__version__))
Expand
titleInstallation Output:
Code Block
(py_env) [salexan5@mblog1 ~]$ conda install pandas
Channels:
 - defaults
Platform: linux-64
Collecting package metadata (repodata.json): done
Solving environment: done

## Package Plan ##

  environment location: /home/salexan5/.conda/envs/py_env

  added / updated specs:
    - pandas


The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    bottleneck-1.3.7           |  py312ha883a20_0         140 KB
    numexpr-2.8.7              |  py312hf827012_0         149 KB
    pandas-2.2.2               |  py312h526ad5a_0        15.4 MB
    python-dateutil-2.9.0post0 |  py312h06a4308_2         318 KB
    pytz-2024.1                |  py312h06a4308_0         220 KB
    ------------------------------------------------------------
                                           Total:        16.2 MB

The following NEW packages will be INSTALLED:

  bottleneck         pkgs/main/linux-64::bottleneck-1.3.7-py312ha883a20_0
  numexpr            pkgs/main/linux-64::numexpr-2.8.7-py312hf827012_0
  pandas             pkgs/main/linux-64::pandas-2.2.2-py312h526ad5a_0
  python-dateutil    pkgs/main/linux-64::python-dateutil-2.9.0post0-py312h06a4308_2
  python-tzdata      pkgs/main/noarch::python-tzdata-2023.3-pyhd3eb1b0_0
  pytz               pkgs/main/linux-64::pytz-2024.1-py312h06a4308_0
  six                pkgs/main/noarch::six-1.16.0-pyhd3eb1b0_1


Proceed ([y]/n)? y


Downloading and Extracting Packages:

Preparing transaction: done
Verifying transaction: done
Executing transaction: done

...