== Software Environments ==
-----

=== Python ===

It is possible to install modules or create your own Python environment in your home directory if the python environment on the HPC machine is not suitable to run certain programs: Either because there are modules missing or their version are too old or too new.

This allows you to install extra modules or different versions thereof, or even an entirely different Python version.

Please note that Python 2 is now obsolete and end-of-life. Everybody should consider using Python 3 or migrating to it.

There are several ways to create an environment:
-----
==== pip/pip3 ====

Pip and its Python 3 equivalent pip3 are installation tools for the Python packages index, abbreviated to PyPI.
This allows you to install new modules or programs which are not installed (yet). You can search through the package index on https://pypi.org/

For instance, if you want to install tensorflow, do:

module load devtoolset/8
pip3 install --user tensorflow

Pip3 will then download tensorflow and compile and install its dependent modules. When finished, you can check tensorflow's version by:

[feverdij@hpc12:~]$ python3
Python 3.6.8 (default, Apr 2 2020, 13:34:55)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-39)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import tensorflow as tf
>>> tf.__version__
'1.14.0'

'''pip3 list''' gives a list of locally installed modules.

uninstalling pip modules can be done with :

pip3 uninstall <name of module>

Note that sometimes pip will install different versions of system modules like numpy/scipy. Since the locally pip-installed modules takes precedence over the system ones, one may get into problems with code developed with the native system modules.

Also, if you need to install multiple programs and modules, pip can cause conflicts between programs if there are dependency conflicts. sometimes these are not easily resolvable, which means you need to up- or downgrade your modules.
-----
==== virtualenv/venv ====

Virtualenv and venv (for python3) are a solution to pip dependency problems by creating a separate environment for a python program. It creates a directory where the virtual environment is installed. If you want to use it, you can activate that environment.

Lets try to install pytorch. First install a virtual environment:

virtualenv pytorch

Then activte it:

source pytorch/bin/activate

When activated, you see the environment in brackets:

(pytorch) [feverdij@hpc12:~]$

Inside the environment, you can use pip to install pytorch

pip install future torch torchvision

and check it with

(pytorch) [feverdij@hpc12:pytorch]$ python
Python 2.7.5 (default, Aug 7 2019, 00:51:29)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-39)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> print torch.__version__
1.4.0

If you need to return to your normal python environment, do:

deactivate

-----
==== conda and miniconda ====

Another virtual environment package is conda. Short for anaconda, it is a full Python environment. A minimal/bare environment is miniconda, where packages and their dependencies can be installed with the conda package manager.

For installing specific python packages, miniconda is preferred because it uses less disk space as conda.

To install miniconda, download the installer from the anaconda website:

wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh

Then execute it:

bash Miniconda3-latest-Linux-x86_64.sh

Select a directoryname to instal. Make it a distinct name for your project, for example 'pylops' if you want to install that package. Next, it continues with:

Do you wish the installer to initialize Miniconda3
by running conda init? [yes|no] yes

If 'yes', the script will modify .bashrc to start the conda environment every time you log into the cluster. You can defer this choice by selecting 'no' and do

conda init

later...

Activating a conda environment is similar to virtualenv/venv

source pylops/bin/activate

Next, install in the conda environment your conda package:

conda install -c conda-forge pylops

After a while verify that it has installed your package

(base) [feverdij@hpc12:projects]$ python3
Python 3.8.5 (default, Sep 4 2020, 07:30:14)
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pylops
>>> pylops.__version__
'1.13.0'

and leave the environment with

conda deactivate

Main Page

2021-04-14T09:36:20Z

Frank Everdij: Added new page

<strong>Welcome to the hpcwiki!</strong>

This wiki is intended as a guide for the hpc clusters managed bij the ICT Department. If you have never used one of our clusters before, this is the place to start. Please note that this wiki is a work in progress, it may not yet be complete.

[[Introduction]]

[[How to log in|Access]] (How to log in and how to copy files)

[[The queue system]]

[[More about queues and nodes]]

[[How to run a job]] (Examples)

[[Applications]]

[[Software Environments]]

[[Further reading]] (Where to find more information)