Software Environments

Python Environments

---

Python

It is possible to install modules or create your own Python environment in your home directory if the python environment on the HPC machine is not suitable to run certain programs: Either because there are modules missing or their version are too old or too new.

This allows you to install extra modules or different versions thereof, or even an entirely different Python version.

Please note that Python 2 is now obsolete and end-of-life. Everybody should consider using Python 3 or migrating to it.

There are several ways to create an environment:

---

pip/pip3

Pip and its Python 3 equivalent pip3 are installation tools for the Python packages index, abbreviated to PyPI. This allows you to install new modules or programs which are not installed (yet). You can search through the package index on https://pypi.org/

For instance, if you want to install tensorflow, do:

 module load devtoolset/8
 pip3 install --user tensorflow

Pip3 will then download tensorflow and compile and install its dependent modules. When finished, you can check tensorflow's version by:

 [feverdij@hpc12:~]$ python3
 Python 3.6.8 (default, Apr  2 2020, 13:34:55) 
 [GCC 4.8.5 20150623 (Red Hat 4.8.5-39)] on linux
 Type "help", "copyright", "credits" or "license" for more information.
 >>> import tensorflow as tf
 >>> tf.__version__
 '1.14.0'

pip3 list gives a list of locally installed modules.

uninstalling pip modules can be done with :

 pip3 uninstall <name of module>

Note that sometimes pip will install different versions of system modules like numpy/scipy. Since the locally pip-installed modules takes precedence over the system ones, one may get into problems with code developed with the native system modules.

Also, if you need to install multiple programs and modules, pip can cause conflicts between programs if there are dependency conflicts. sometimes these are not easily resolvable, which means you need to up- or downgrade your modules.

---

virtualenv/venv

Virtualenv and venv (for python3) are a solution to pip dependency problems by creating a separate environment for a python program. It creates a directory where the virtual environment is installed. If you want to use it, you can activate that environment.

Lets try to install pytorch. First install a virtual environment:

 virtualenv pytorch

Then activte it:

 source pytorch/bin/activate

When activated, you see the environment in brackets:

 (pytorch) [feverdij@hpc12:~]$ 

Inside the environment, you can use pip to install pytorch

 pip install future torch torchvision

and check it with

 (pytorch) [feverdij@hpc12:pytorch]$ python
 Python 2.7.5 (default, Aug  7 2019, 00:51:29) 
[GCC 4.8.5 20150623 (Red Hat 4.8.5-39)] on linux2
 Type "help", "copyright", "credits" or "license" for more information.
 >>> import torch
 >>> print torch.__version__
 1.4.0

If you need to return to your normal python environment, do:

 deactivate

---

conda and miniconda

Another virtual environment package is conda. Short for anaconda, it is a full Python environment. A minimal/bare environment is miniconda, where packages and their dependencies can be installed with the conda package manager.

For installing specific python packages, miniconda is preferred because it uses less disk space as conda.

To install miniconda, download the installer from the anaconda website:

 wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh

Then execute it:

 bash Miniconda3-latest-Linux-x86_64.sh

Select a directory-name to install. The default should be fine.

 Do you wish the installer to initialize Miniconda3
 by running conda init? [yes|no] yes

If 'yes', the script will modify .bashrc to start the conda environment every time you log into the cluster. You can defer this choice by selecting 'no' and do

 conda init

later...

After a logout/login the conda command should be available. If this is not the case, there may be a problem sourcing ~/.bashrc. As a workaround, you can copy ~/.bashrc to ~/.profile and logout/login again.

If you see '(base)' in your prompt, then conda is installed and ready to use.

To create a new conda environment with for instance the pylops package, do:

 conda create -n pylops -c conda-forge pylops

and activate the environment with:

 conda activate pylops

After a while verify that your package is installed and your environment is ready to use:

 (base) [feverdij@hpc06:~]$ conda activate pylops
 (pylops) [feverdij@hpc06:~]$ python3
 Python 3.8.10 | packaged by conda-forge | (default, May 11 2021, 07:01:05) 
 [GCC 9.3.0] on linux
 Type "help", "copyright", "credits" or "license" for more information.
 >>> import pylops
 >>> pylops.__version__
 '1.13.0'
 >>> 

and leave the environment with:

 conda deactivate

A specific python version can also be installed using conda:

 conda create -n mypython python=3.6.5
 conda activate mypython
 (mypython) [feverdij@hpc06:~]$ python3
 Python 3.6.5 |Anaconda, Inc.| (default, Apr 29 2018, 16:14:56) 
 [GCC 7.2.0] on linux
 Type "help", "copyright", "credits" or "license" for more information.
 >>>