Pip Install within a Conda Environment
- Simon Alexander
Goal: Understand how Conda’s pip works with a User’s Python pip package Installs.
- 1 Quick Note: Default Python Version with a Conda Environment
- 2 Conda Environments and pip Installs
- 3 Try Installing numpy into our Conda environment
- 4 Confirm what is Currently Installed/Available
- 5 Force a numpy update
- 6 Create a Self-Contained Conda Environment
- 7 Recommendation: Always set the PYTHONUSERBASE environment variable
- 8 Remember: Conda Environments, Pip Installs and Package Locations
Quick Note: Default Python Version with a Conda Environment
Miniconda does ship with a default version of Python.
[]$ cd /project/<project-name>/<username>/conda
[]$ module purge
[]$ module load miniconda3/24.3.0
[]$ conda create -p py_env
## Package Plan ##
environment location: /cluster/medbow/project/<project-name>/<username>/conda/py_env
# To activate this environment, use
# $ conda activate /cluster/medbow/project/<project-name>/<username>/conda/py_env
[]$ conda activate /cluster/medbow/project/<project-name>/<username>/conda/py_env
(/cluster/medbow/project/<project-name>/<username>/conda/py_env) []$ python --version
Python 3.12.2
(/cluster/medbow/project/<project-name>/<username>/conda/py_env) []$ which python
/apps/u/opt/linux/miniconda3/24.3.0/bin/pythonConda Environments and pip Installs
Installing non-conda packages:
If a package is not available from conda or Anaconda.org, you may be able to find and install the package via conda-forge or with another package manager like
pip.Pip packages do not have all the features of conda packages and we recommend first trying to install any package with conda. If the package is unavailable through conda, try finding and installing it with conda-forge.
If you still cannot install the package, you can try installing it with pip. The differences between pip and conda packages cause certain unavoidable limits in compatibility but conda works hard to be as compatible with pip as possible.
Note:
Both pip and conda are included in Anaconda and Miniconda, so you do not need to install them separately.
It is possible to have pip installed outside a conda environment or inside a conda environment.
Not every Python package can be conda install-ed. Tensorflow is only available via pip install.
Try Installing numpy into our Conda environment
[]$ module purge
[]$ module load miniconda3/24.3.0
[]$ conda activate /cluster/medbow/project/<project-name>/<username>/conda/py_env
(/cluster/medbow/project/<project-name>/<username>/conda/py_env) []$ pip install numpy
Defaulting to user installation because normal site-packages is not writeable
Requirement already satisfied: numpy in /home/<username>/.local/lib/python3.12/site-packages (1.26.4)
(/cluster/medbow/project/<project-name>/<username>/conda/py_env) [salexan5@mblog1 conda]$ python ~/py_test.py
Python: 3.12.2 | packaged by Anaconda, Inc. | (main, Feb 27 2024, 17:35:02) [GCC 11.2.0]
Numpy: 1.26.4What’s happened?
Conda’s pip will look under the userbase for existing packages.
In this case ~/.local/lib/python3.12/site-packages/ where it will find the previously installed version of numpy/1.26.4
Confirm what is Currently Installed/Available
Check for conda installed packages:
# This list should be empty since we have essentially a blank conda environment
# since we have not conda installed anything.
(/cluster/medbow/project/<project-name>/<username>/conda/py_env) [salexan5@mblog1 conda]$ conda list
# packages in environment at /cluster/medbow/project/<project-name>/<username>/conda/py_env:
#
# Name Version Build ChannelCheck for pip installed packages:
(/cluster/medbow/project/<project-name>/<username>/conda/py_env) []$ pip list -v
Package Version Location Installer
----------------------- ----------- ---------------------------------------------------------------- ---------
anaconda-anon-usage 0.4.4 /apps/u/opt/linux/miniconda3/24.3.0/lib/python3.12/site-packages conda
archspec 0.2.3 /apps/u/opt/linux/miniconda3/24.3.0/lib/python3.12/site-packages conda
...
numpy 1.26.4 /home/<username>/.local/lib/python3.12/site-packages pip
...
pip 23.3.1 /apps/u/opt/linux/miniconda3/24.3.0/lib/python3.12/site-packages
...Conda’s pip will check for package installs under the ~/.local/lib/PythonX.Y folder.
If the python package already exists it will be used.
Force a numpy update
(/cluster/medbow/project/<project-name>/<username>/conda/py_env) []$ pip install numpy==2.0.0
Defaulting to user installation because normal site-packages is not writeable
...
Installing collected packages: numpy
Attempting uninstall: numpy
Found existing installation: numpy 1.26.4
Uninstalling numpy-1.26.4:
Successfully uninstalled numpy-1.26.4
Successfully installed numpy-2.0.0
(/cluster/medbow/project/<project-name>/<username>/conda/py_env) []$ pip list -v
Package Version Location Installer
----------------------- ----------- ---------------------------------------------------------------- ---------
...
numpy 2.0.0 /home/<username>/.local/lib/python3.12/site-packages pip
...The numpy package has been updated, but still lives under: ~/.local/lib/python3.12/
Create a Self-Contained Conda Environment
We can force our conda environment to only use what is installed within it, by setting the PYTHONUSERBASE environment variable:
(/cluster/medbow/project/<project-name>/<username>/conda/py_env) []$ echo $CONDA_PREFIX
/cluster/medbow/project/<project-name>/<username>/conda/py_env
(/cluster/medbow/project/<project-name>/<username>/conda/py_env_3.12.4) []$ export PYTHONUSERBASE=$CONDA_PREFIX
(/cluster/medbow/project/<project-name>/<username>/conda/py_env) []$ pip list -v
Package Version Location Installer
----------------------- -------- ------------------------------------------------------------------------------- ---------
...
numpy 2.0.0 /cluster/medbow/project/<project-name>/<username>/conda/py_env/lib/python3.12/site-packages pip
...Conda’s pip now only looks under, and installs under, it’s own site-packages folder:
(/cluster/medbow/project/<project-name>/<username>/conda/py_env) []$ pwd
/project/<project-name>/<username>/conda
(/cluster/medbow/project/<project-name>/<username>/conda/py_env) []$ ls py_env/lib/python3.12/site-packages/
numpy numpy-2.0.0.dist-info numpy.libsRecommendation: Always set the PYTHONUSERBASE environment variable
Sessions/Environments can get confused if you have previously set environment variables and have forgotten what has/hasn’t been set within a session.
Starting a new session, notice which version of numpy is being used by our conda environment before and after setting our PYTHONUSERBASE environment variable.
[]$ module purge
[]$ module load miniconda3/24.3.0
[]$ conda activate /project/<project-name>/<username>/conda/py_env
(/project/<project-name>/<username>/conda/py_env) []$ pip list -v
Package Version Location Installer
----------------------- ----------- ---------------------------------------------------------------- ---------
...
numpy 2.0.0 /home/<username>/.local/lib/python3.12/site-packages pip
...
(/project/<project-name>/<username>/conda/py_env) []$ export PYTHONUSERBASE=$CONDA_PREFIX
(/project/<project-name>/<username>/conda/py_env) []$ pip list -v
Package Version Location Installer
----------------------- -------- ---------------------------------------------------------------- ---------
...
numpy 2.0.0 /project/<project-name>/<username>/conda/py_env/lib/python3.12/site-packages pip
...Remember: Conda Environments, Pip Installs and Package Locations
Remember: If you do not set export PYTHONUSERBASE=$CONDA_PREFIX (after the Conda environment has been activated), then pip install will place Python packages under ~/.local/lib/pythonX.Y/site-packages.
