Managing Packages Using Conda

Conda is a package manager, which helps you find and install packages such as numpy or scipy. It also serves as an environment manager, and allows you to have multiple isolated environments for different projects on a single machine. Each environment has its own installation directories, that doesn’t share packages with other environments.

Loading Anaconda Environments

There are various anaconda python environments provided for everyone to use. Run “module spider python” to check the available python environments.  

The anaconda environments can be loaded through provided module files:

module load python/ondemand-jupyter-python3.8

Where anaconda2 represents Python 2 installations and anaconda3 represents Python 3 installations.

Create new conda environment

Create a conda environment named scanpy with python 3.8.

[fmu@login0b ~]$ module purge
[fmu@login0b ~]$ module load python/ondemand-jupyter-python3.8
[fmu@login0b ~]$ conda create -n scanpy python=3.8
Collecting package metadata (current_repodata.json): done
Solving environment: done
...

Usually, the conda environment is installed in your home directory, /ihome/<your group name>/<your username>. The newly created environment <scanpy> will be installed in the directory /ihome/<your group name>/<your username>/.conda/envs/scanpy. You can see all the packages under folder “bin”.

You can also provide a prefix to install conda environment.

module load python/bioconda-3.7-2019.03
conda create --prefix=/bgfs/fmu/envs/samtools python=3.7

This environment will be installed in the directory /bgfs/fmu/envs/samtools

Use the environment you just created

Activate your environment:
    source activate scanpy

It will show your environment name at the beginning of the prompt.

If you are using a prefix to install conda environment, use the prefix to activate your environment.

source activate /bgfs/fmu/envs/samtools

More recent Anaconda distributions will tell you to use the command conda activate instead of source activate to activate your newly created environment. If you use conda activate, you will be prompted to issue the command conda init. Do not do this. When you load one of our Anaconda modules, you have effectively loaded the base conda environment for that Anaconda version. However, conda tries to manage activation of its own base environment by modifying the user's .bashrc file.

Even if you have installed your own local version of Anaconda or miniconda, do not use conda init. When conda init runs, it places commands into your .bashrc file that will stop certain things from working on the system. Many modules use their own python environments. Activating a specific conda environment in your .bashrc file can break these python environments.

Workaround for the conda init command

The conda init command places code in your .bashrc file that modifies, among other things, the PATH environment variable by prepending it to the path of the base conda environment. This occurs before the default system modules are loaded. Other modules may also have libraries that will hide Anaconda libraries and cause errors.

To work around this in local Anaconda or miniconda installations:

    Run conda init, and then immediately open .bashrc with a file editor.
    Remove the code that was added by conda init and place it in another script file (for example, conda_init.sh).
    After the login process completes, run the code in the script file:

    source conda_init.sh

You should now be able to use conda activate.

conda configuration file

The conda configuration file, .condarc, is an optional runtime configuration file that allows advanced users to configure various aspects of conda, such as which channels it searches for packages, proxy settings, and environment directories.

The following displays a sample .condarc file:

[fmu@login0b ~]$ cat .condarc
pkgs_dirs:
  - /ihome/fmu/fmu/.conda/pkgs
channels:
  - r
  - conda-forge
  - bioconda
  - defaults

Install packages in the conda environment

The conda package manager is recommended for maintaining your user Miniconda Python distribution. Take look at and use the Conda cheat sheet,  which lists the most commonly used commands. More detailed documentation is in the Conda User Guide.

To install a new package, run

conda install [packagename]

Conda channels are the remote repository that conda takes to search or download the packages. If you want to install a package that is not in the default Anaconda channel, you can tell conda which channel containing the package, so that conda can find and install. To install a new package from bioconda channel, run

conda install -c bioconda [packagename]

Installing Python modules with pip

When a Python package does not exist as a conda package, one can use the Python pip installer. We recommend using pip only as a last resort since this way one loses the flexibility of the conda packaging environment (automatic conflict resolution and version upgrade/downgrade). See conda's references. To install a module using pip, run:

pip install [packagename]

Custom Miniconda installation and usage

You can use miniconda installer under /ihome/crc/build/python. You can also download the Miniconda installer using the wget command.  Then run the installer, pointing it to the directory where you want to install it.

[fmu@login0b ~]$ cd /ihome/crc/build/python
[fmu@login0b python]$ bash Miniconda3-py38_4.9.2-Linux-x86_64.sh -b -p ~/python_env/myenv -s
PREFIX=/ihome/fmu/fmu/python_env/myenv
...
[fmu@login0b python]$ cd ~/python_env/myenv/bin
[fmu@login0b bin]$ pwd
/ihome/fmu/fmu/python_env/myenv/bin
[fmu@login0b bin]$ # ./conda install -c conda-forge jupyterlab # You can install other conda package to your miniconda environment
...

The flag '-b' forces unattended installation, which among other things does not add Miniconda to your default environment - we will do it in the next step via environment modules. The '-p' marks the installation directory. The '-s' will not automatically set up your environment to use this miniconda - we will do this in the next section using the environment module.

The full path to this local anaconda installs directory is /ihome/fmu/fmu/python_env/myenv

Custom Miniconda environment module

To easily set up the Miniconda environment, create an user environment module. First create a directory where the user environment module hierarchy will reside, and then copy our miniconda module file to this directory.

[fmu@login0b ~]$ mkdir modulefiles
[fmu@login0b ~]$ module use ~/modulefiles
[fmu@login0b ~]$ cd modulefiles
[fmu@login0b modulefiles]$ cp /ihome/crc/modules/Core/python/ondemand-jupyter-python3.8.lua myenv.lua
[fmu@login0b modulefiles]$ vi myenv.lua #edit myenv.lua: local package_root = "/ihome/fmu/fmu/python_env/myenv"

The user module environment must be loaded into the default module environment with the module use command. After that, we can load the user space miniconda module.

[fmu@login0b ~]$ module purge
[fmu@login0b ~]$ module use ~/modulefiles
[fmu@login0b ~]$ module load myenv
[fmu@login0b ~]$ which conda
~/python_env/myenv/bin/conda

Running a Conda installed application

Here is a sample batch script that uses tools that were installed via a Conda environment:

#!/bin/bash
#
#SBATCH --job-name=conda
#SBATCH -N 1 # Ensure that all cores are on one machine
#SBATCH -t 3-00:00 # Runtime in D-HH:MM
#SBATCH --cpus-per-task=1 # Request that ncpus be allocated per process.

module load python/bioconda-3.7-2019.03
source activate /bgfs/fmu/envs/samtools

#your code
samtools --help