Python

Versions

The H2P cluster supports various versions of Python. For general use, I suggest the following modules:

  • python/anaconda3.5-4.2.0
  • python/anaconda3.6-5.2.0
  • python/3.7.0

Some packages (e.g. Tensorflow) must be dealt with in a more specific way. I will discuss them at the end . All the modules are Anaconda distributions and support pip and conda installation commands. The tricky part is that no user has privelages to install their own python packages. You have a few options you will need to install them:

  1. pip install --user <package>: This will install into a local directory that you have permission to access.
  2. Manage your own environment using virtualenvwrapper.

If you are unsure of what version of the package you want to install you can run pip install <package>== and it will print the available versions.

Virtual Environments

I strongly encourage using virtual environments, which gives you complete control over which versions of packages are installed. Here is a little example of installing PyTorch (CPU Version) into a Virtual Environment using virtualenvwrapper.

$ module load python/3.7 venv/wrap # Available for the versions of Python listed above
$ mkvirtualenv pytorch
$ workon pytorch
$ pip install numpy torch torchvision
$ python
Python 3.7.0 (default, Jun 28 2018, 13:15:42) 
[GCC 7.2.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
@>>> import torch
@>>> x = torch.rand(5, 3)
@>>> print(x)
tensor([[0.6022, 0.5194, 0.3726],
        [0.0010, 0.7181, 0.7031],
        [0.7442, 0.5017, 0.2003],
        [0.1068, 0.4622, 0.2478],
        [0.8989, 0.8953, 0.0129]])
@>>> 

To list your environments, use workon with no arguments. The nice thing about this tool is that you can "hot-swap" Python environments. If you want to swap pytorch for tensorflow you can do workon tensorflow.

Machine Learning

At CRC, we are currently running CUDA 10.1 and the corresponding drivers. The recommended version of PyTorch is 1.4.0 and Torch Vision 0.5.0 and can be installed via pip install torch==1.4.0 torchvision==0.5.0. The recommended version of Tensorflow is 2.1.0 and can be installed via pip install tensorflow==2.1.0

Access Virtual Environment from JupyterHub

To access a virtual environment from JupyterHub you need to install ipykernel. In the following code snippet, ray is an arbitrary name you can use any name you like (but make sure to replace all instances of ray). /location/to/python is the version of python you want to use. If you want to use the one from python/3.7.0 you can omit this option. To get the path, use which python.

module purge
module load python/3.7.0 venv/wrap
mkvirtualenv -p /location/to/python ray
workon ray
pip install ipykernel
python -m ipykernel install --user --name=ray