Scavenger Partition

To enable better utilization of available compute resources, each cluster has a "scavenger" partition defined. 

The scavenger partition allows groups to use up to 90% of the resources on the cluster, do not incurr a service unit cost, and do not enforce any time limits. With this, the jobs on this partition are preemptible and will only run if there are idle resources. They can also be canceled by Slurm as jobs enter the paid partitions.

 

Common Scavenger partition use cases

The scavenger partitions are great when you need to run many short jobs or when you have a long-running, restartable job

In either case, jobs should be able to save partial output or report any failures in their outputs in the event of cancellation. 

Many short jobs

When submitting many short jobs without requeue (the default), you control when and where to resubmit the preempted jobs.
The main downside to this is that you'll need to write a script to requeue preempted jobs or adjust the partition and resubmit.

If you choose to requeue, you can start these jobs and forget about them, but they may continue to get preempted over and over again.

Note: There is a cap 1000 jobs per user per cluster for the scavenger partition because the queue becomes unstable when users launch short jobs in quick succession.

Long restartable jobs

In that your job could be cancelled at any time, longer jobs should create intermediate outputs, and be set up to detect files to restart from when requeuing, making sure that input files in the correct places. Submit your job into the scavenger partition with the --requeue option.

 

Submitting to the scavenger partitions

You can request the scavenger partition with #SBATCH --partition=scavenger. This will queue your job with a QOS that sets it at a particularly low priority.

Note: For the gpu scavenger partition you must specify which type of node as well: 

#SBATCH --constraint=<partition>, where partition can be one of the following: gtx1080, titanx, k40, v100

Note: It is currently possible to submit a job to the paid partitions that, taking fairshare and other job priority factors into account, can have a lower priority than a scavenger job and not preempt them. This should be a rare occurence due to scavenger jobs being such a low priority as it is and may be prevented by submitting smaller batches of jobs at a time.

We are working on documenting this problem, so if you notice it happening, please submit a ticket with the output of 

sprio -M gpu -S "-y" --format %.15i,%.9r,%.10Y,%.10u

to show the sorted priority of jobs on queued on GPU, and then the output of

squeue -M gpu

to show your job pending, the reason, and the job running on scavenger that has not been preempted. 

 

 

Job Cancellation and Requeueing

If preempted by a paid partition job, the default behavior is for the job to be cancelled. 

There is an option to place the job back in the queue, however, with an SBATCH argument:

#SBATCH --requeue

 

Cluster specific notes for scavenger partition configuration

The PreemptMode for the scavenger QOS is "requeue".

HTC - Because users often submit many jobs to HTC, the jobs per user cap set to 100 for it's scavenger partition.