Managing Packages Using Conda
Anaconda or 'Conda' is a package manager for the Python programming language. It can be used to find and install packages such as numpy or scipy. It also serves as an environment manager, and allows you to have multiple isolated environments for different projects on a single machine. Each environment has its own installation directories and does not share packages with other environments.
- Loading a Particular Verions of Python
- Creating a New Conda Environment
- Activating Your Conda Environment
- Conda Environment Configuration Files
- Install Packages in your Conda Environment
- Custom Miniconda Installation and Usage
- Custom Miniconda Environment Module
- Running a Conda Installed Application
There are various python versions provided for everyone to use via the module system.
Run “module spider python” to check the available python environments.
These can be loaded through provided module files. For example:
module load python/ondemand-jupyter-python3.8
anaconda2 represents Python 2 installations and anaconda3 represents Python 3 installations.
After loading your desired python version, you can create a virtual environment.
Here is an example of creating a virtual environment through conda named scanpy with python version 3.8.
[fmu@login0b ~]$ module purge [fmu@login0b ~]$ module load python/ondemand-jupyter-python3.8 [fmu@login0b ~]$ conda create -n scanpy python=3.8 Collecting package metadata (current_repodata.json): done Solving environment: done ...
Usually, the conda environment is installed in your home directory
/ihome/<your group name>/<your username>
The newly created environment <scanpy> will be installed in the directory
/ihome/<your group name>/<your username>/.conda/envs/scanpy.
You can see all the packages under folder “bin”.
You can also provide a prefix to install conda environment.
module load python/bioconda-3.7-2019.03 conda create --prefix=/bgfs/fmu/envs/samtools python=3.7
This resulting environment will be installed in the directory `/bgfs/fmu/envs/samtools`
Activate your environment with the source command:
source activate scanpy
When this is done, the environment name should be visible at the beginning of your terminal prompt.
If you are using a prefix to install conda environment, use the prefix to activate your environment.
source activate /bgfs/fmu/envs/samtools
Important Note on `conda activate`:
Do not use `conda activate` to source your environment on the clusters. More recent Anaconda distributions will tell you to use this command instead of source activate to activate your newly created environment. If you use conda activate, you will be prompted to issue the command conda init. Do not do this. When you load one of our Anaconda modules, you have effectively loaded the base conda environment for that Anaconda version. However, conda tries to manage activation of its own base environment by modifying the user's .bashrc file.
Even if you have installed your own local version of Anaconda or miniconda, do not use conda init. When conda init runs, it places commands into your .bashrc file that will stop certain things from working on the system. Many modules use their own python environments. Activating a specific conda environment in your .bashrc file can break these python environments.
Workaround for the conda init command
The conda init command places code in your .bashrc file that modifies, among other things, the PATH environment variable by prepending it to the path of the base conda environment. This occurs before the default system modules are loaded. Other modules may also have libraries that will hide Anaconda libraries and cause errors.
To work around this in local Anaconda or miniconda installations,
Run conda init, and then immediately open .bashrc with a file editor.
Remove the code that was added by conda init and place it in another script file (for example, conda_init.sh).
After the login process completes, run the code in the script file:
You should now be able to use conda activate.
The conda configuration file, .condarc, is an optional runtime configuration file that allows advanced users to configure various aspects of conda, such as which channels it searches for packages, proxy settings, and environment directories.
The following displays a sample .condarc file:
[fmu@login0b ~]$ cat .condarc pkgs_dirs: - /ihome/fmu/fmu/.conda/pkgs channels: - r - conda-forge - bioconda - defaults
Installing Python modules with `conda`
The conda package manager is recommended for maintaining your environment. Take look at and use the Conda cheat sheet, which lists the most commonly used commands. More detailed documentation is in the Conda User Guide.
To install a new package, run
conda install [packagename]
Conda channels are the remote repository that conda takes to search or download the packages. If you want to install a package that is not in the default Anaconda channel, you can tell conda which channel containing the package, so that conda can find and install. To install a new package from bioconda channel, run
conda install -c bioconda [packagename]
Installing Python modules with `pip`
When a Python package does not exist as a conda package, one can use the Python pip installer. We recommend using pip only as a last resort since this way one loses the flexibility of the conda packaging environment (automatic conflict resolution and version upgrade/downgrade). See conda's references. To install a module using pip, run:
pip install [packagename]
You can use miniconda installer under /ihome/crc/build/python. You can also download the Miniconda installer using the wget command. Then run the installer, pointing it to the directory where you want to install it.
[fmu@login0b ~]$ cd /ihome/crc/build/python [fmu@login0b python]$ bash Miniconda3-py38_4.9.2-Linux-x86_64.sh -b -p ~/python_env/myenv -s PREFIX=/ihome/fmu/fmu/python_env/myenv ... [fmu@login0b python]$ cd ~/python_env/myenv/bin [fmu@login0b bin]$ pwd /ihome/fmu/fmu/python_env/myenv/bin [fmu@login0b bin]$ # ./conda install -c conda-forge jupyterlab # You can install other conda package to your miniconda environment ...
The flag '-b' forces unattended installation, which among other things does not add Miniconda to your default environment - we will do it in the next step via environment modules. The '-p' marks the installation directory. The '-s' will not automatically set up your environment to use this miniconda - we will do this in the next section using the environment module.
The full path to this local anaconda installs directory is /ihome/fmu/fmu/python_env/myenv
To easily set up the Miniconda environment, create an user environment module. First create a directory where the user environment module hierarchy will reside, and then copy our miniconda module file to this directory.
[fmu@login0b ~]$ mkdir modulefiles [fmu@login0b ~]$ module use ~/modulefiles [fmu@login0b ~]$ cd modulefiles [fmu@login0b modulefiles]$ cp /ihome/crc/modules/Core/python/ondemand-jupyter-python3.8.lua myenv.lua [fmu@login0b modulefiles]$ vi myenv.lua #edit myenv.lua: local package_root = "/ihome/fmu/fmu/python_env/myenv"
The user module environment must be loaded into the default module environment with the module use command. After that, we can load the user space miniconda module.
[fmu@login0b ~]$ module purge [fmu@login0b ~]$ module use ~/modulefiles [fmu@login0b ~]$ module load myenv [fmu@login0b ~]$ which conda ~/python_env/myenv/bin/conda
Here is a sample batch script that uses tools that were installed via a Conda environment:
#!/bin/bash # #SBATCH --job-name=conda #SBATCH -N 1 # Ensure that all cores are on one machine #SBATCH -t 3-00:00 # Runtime in D-HH:MM #SBATCH --cpus-per-task=1 # Request that ncpus be allocated per process. module load python/bioconda-3.7-2019.03 source activate /bgfs/fmu/envs/samtools #your code samtools --help