Cluster Interactive/remote Computing with VS Code

Attention: You are viewing a page from the old CRC user manual.

This page may not represent the most up to date version of the content.

You can view it in our new user manual here.

This tutorial outlines how to set up VS Code for interactive/remote development/debugging on Pitt CRC computing nodes.

Prerequisits:

• Latest version of VS Code on your local machine
• Latest version of the "Remote Development" extension pack (here) 

Steps performed only once

• Add the following lines to the ssh config file on your local machine (~/.ssh/config) and replace <name> with your username:

Host htc
  ControlMaster auto
  ControlPath ~/.ssh/master-%r@%h:%p
  HostName htc.crc.pitt.edu
  User <name>

Host htcx
  ProxyCommand ssh htc "nc \$(squeue --me --name=tunnel --states=R -h -O NodeList,Comment)"
  StrictHostKeyChecking no
  User <name>

• Connect to the HTC cluster and create the following sbatch file into your home directory (~/tunnel.sbatch):

#!/bin/bash

#SBATCH --output="tunnel.log"
#SBATCH --job-name="tunnel"
#SBATCH --time=4:00:00 # walltime
#SBATCH --cpus-per-task=2 # number of cores
#SBATCH --mem-per-cpu=8G # memory per CPU core

# load the modules
module load python/ondemand-jupyter-python3.8

# find open port
PORT=$(python -c 'import socket; s=socket.socket(); s.bind(("", 0)); print(s.getsockname()[1]); s.close()')
scontrol update JobId="$SLURM_JOB_ID" Comment="$PORT"

# start sshd server on the available port
echo "Starting sshd on port $PORT"
/usr/sbin/sshd -D -p ${PORT} -f /dev/null -h ${HOME}/.ssh/id_rsa

• In the terminal on your local machine, generate an ssh key if you don't have one already using the following:

ssh-keygen -t rsa

• Copy your local public ssh key to the cluster using:

ssh-copy-id htc 

• On htc.crc.pitt.edu, run the following commands if you have not done so:

     ssh-keygen #then follow on screen instructions
     cd ~/.ssh
     cp id_rsa.pub authorized_keys

Steps performed every time to connect your VS Code to the cluster

• From your local terminal, connect to the cluster using ssh htc  and once logged in, type sbatch tunnel.sbatch to start the remote server. Make sure that your job does run!

Sample output:

[user@login0b ~]$ sbatch tunnel.sbatch
Submitted batch job 1383495
[user@login0b ~]$ squeue -u user

  JOBID      PARTITION      NAME      USER      ST      TIME      NODES      NODELIST(REASON)
1383495            htc    tunnel      user       R      0:17          1           htc-1024-n0

• Open VS Code on your local machine and connect to your projects using "Remote Explorer" with "htcx" as the ssh target.

Changes you need to do to allocate resources with GPUs

• Add the following host to the ssh config file on your local machine (~/.ssh/config) and replace <name> with your username:

Host gpux
  ProxyCommand ssh htc "nc \$(squeue -M gpu --me --name=tunnel --states=R -h -O NodeList,Comment)"
  StrictHostKeyChecking no
  User <name>

• Change the contents of the sbatch file in your home directory on HTC (~/tunnel.sbatch) to:

#!/bin/bash
#SBATCH --output="tunnel.log"
#SBATCH --job-name="tunnel"
#SBATCH --cpus-per-task 2
#SBATCH --time 0-01:00:00
#SBATCH --clusters=gpu
#SBATCH --partition=a100
#SBATCH --gres=gpu:1

module load python/ondemand-jupyter-python3.8 # load the modules

# find open port
PORT=$(python -c 'import socket; s=socket.socket(); s.bind(("", 0));
print(s.getsockname()[1]); s.close()')
scontrol update JobId="$SLURM_JOB_ID" Comment="$PORT"
# start sshd server on the available port
echo "Starting sshd on port $PORT"
/usr/sbin/sshd -D -p ${PORT} -f /dev/null -h ${HOME}/.ssh/id_rsa