The OnDemand interface allows you to conduct your research on HTC cluster through a web browser. With OnDemand, users can upload and download files; create, edit, submit and monitor jobs; and run GUI applications (e.g. RStudio Server and Jupyter Notebook), without logging in to the HTC cluster via traditional interfaces.
OnDemand was created by the Ohio Supercomputer Center (OSC). This document provides an outline of how to use OnDemand on the HTC cluster. For more help, check the extensive documentation for OnDemand created by OSC, including many video tutorials, or submit a help ticket.
If your computer is not connected to the Pitt network (e.g. you are working from home or on a trip, or you are using Pitt wireless network), or you are working from a laptop that is connected to the UPMC network, make sure you use Pitt SSLVPN, so that you can communicate with the Center for Research Computing (CRC) clusters. Note that there are many different VPN roles. Only Firewall-SAM-USERS-Pulse role can connect to CRC clusters.
To connect to the HTC cluster via OnDemand, point your browser to https://ondemand.htc.crc.pitt.edu.
- You will be prompted for a username and password. Enter your Pitt username and password.
- The OnDemand Dashboard will open. From this page, you can use the menus across the top of the page to manage files and submit jobs to the HTC cluster.
To end your OnDemand session, choose Log Out at the top right of the Dashboard window and Close your browser.
Note: We recommend to use Chrome or Firefox. Safari and Internet Explorer might work for some of the apps in the portal, but are not fully supported.
To create, edit or move files, click on the Files menu from the Dashboard window. A dropdown menu will appear, listing your directories on CRC file systems. Your home directory is already listed. If you have folders under /mnt/mobydisk/groupshares, and /zfs1 directories, they are also listed.
Choosing one of the file spaces opens the File Explorer in a new browser tab. The files in the selected directory are listed. No matter which directory you are in, your home directory is displayed in a panel on the left.
There are two sets of buttons in the File Explorer.
Buttons on the top of the window on the right perform these functions:
Navigate to another directory or file system
Open in Terminal
Open a terminal window in a new browser tab
Creates a new empty file
Create a new subdirectory
Copies a file from your local machine to the HTC cluster
Toggles the display of dotfiles
Toggles the display of owner and permission settings
If you have files under other directories, which are not listed in the dropdown menu, use the Go To to navigate to the directory.
Buttons on the top left just below the name of the current directory allow you to View, Edit, Rename, Download, Copy or Paste (after you have moved to a different directory) a file, or you can toggle the file selection with (Un)Select All.
You can get shell access to Bridges by choosing Clusters > >_HTC Shell Access from the top menus in the Dashboard window. In the window that opens, you are logged in to one of HTC login nodes as if you used ssh to connect to the HTC cluster.
Jupyter Notebook and Jupyter Lab
Jupyter notebook and Jupyter Lab allow for interactive code development. You can run Jupyter notebook, and Jupyter Lab, through OnDemand.
Jupyter Notebook inside OnDemand which runs on a cluster compute node using dedicated resources can be launched by choosing menu Interactive Apps -> Jupyter Notebook. A job submission screen appears:
Set the job parameters. Unless using numerical libraries like NumPy, which are thread-parallelized, or your jobs require more memory, it is not adviseable to choose more than one CPU. Then hit Launch to submit the interactive job. The job gets queued up and when it starts and the Jupyter is provisioned. Click on the Connect to Jupyter button to open a new browser tab with the main Jupyter interface.
Click New and choose the Notebook type to open a new Jupyter notebook:
The python version "bioconda 3.7" will load python/bioconda-3.7-2019.03 module on HTC cluster. New -> Python 3 will use this python.
You can install local python packages in virtual environment and generate ipykernel to be used on jupyter notebook (pyscenic and scanpy in the notebook options). Instructions are available on single cell analysis (login required) section Scanpy.
Stopping your Jupyter Notebook session
In the Dashboard window, click the red Delete button.
RStudio Server runs the RStudio interactive development environment inside of a browser. The OnDemand implementation allows to set up and launch the RStudio Server on a cluster compute node for dedicated resources, which allows to run more compute intensive R programs on the RStudio environment. To start RStudio Server job.
- Select Interactive Apps > RStudio Server from the top menu in the Dashboard window.
- In the screen that opens, specify the R version, time limit.
- Choose the appropriate number of cores, keeping in mind that R can internally thread parallelize vector based data processing, for which more than one CPU can be utilized.
- Click the blue Launch button to start your RStudio session. You may have to wait in the queue for resources to be available.
- When your session starts, click the blue Connect to RStudio Server button. A new window opens with the RStudio interface.
The R version 3.5.1 loads gcc/8.2.0 r/3.5.1 module on HTC cluster, and the R version 3.6.0 loads gcc/8.2.0 r/3.6.0 module on HTC cluster. Within each R module, various R packages and bioconductor packages have been installed. Within the R console, load the library to check whether it is already installed. If you need specific R packages, submit a help ticket.
You can also install your own R packages. R searches the user’s path for libraries followed by the root installation. R will stop searching when it finds the first instance of the library within the path hierarchy.Use “.libPaths()” to check the searching path.
To install your own packages, Clusters >_HTC Shell Access
Load the R module, for example, module load gcc/8.2.0 r/3.6.0
Run R, R
Within R environment, install packages: install.packages(“pkg_name”)
For bioconductor packages, use: BiocManager::install("Bioconductor_pkg_name")
If you exceed the time limit you requested when setting up your RStudio session, you will see this error:
Error: Status code 503 returned
To continue using RStudio, go to Interactive Apps > RStudio from the top menu in the Dashboard window and start a new session.
Stopping your RStudio session
To end your RStudio session, either select File > Quit Session or click the red icon in the upper right of your RStudio window. NOTE that this only closes your RStudio session; it does not close your interactive HTC session. You are still consuming CPU hours on the HTC cluster.
To end your interactive HTC session, return to the Dashboard window and click the red Delete button.