Firstly, use the gpu partition, instead of the default production partition.1 Second, like CPUs or memory, Slurm will not allow you to use a GPU just because it put your job on a sciml host. You must allocate them using --gres as documented in Slurm.
--gres=gpu:[type:]<number>An optional type can be supplied along with the number of GPUs requested.
Note that because we let regular CPU jobs use all but 4 CPUs per GPU on the GPU hosts, this can result in GPU jobs being blocked from accessing even idle GPUs if they request more than 4 CPUs, or ~large~ amounts of host memory (depending on the size of the CPU jobs sharing the host). Most GPU jobs need very little host resources, so it's not a problem to share, but if you have a job that can make good use of more than 4 CPUs per GPU or large amounts of host memory (not to be confused with GPU memory), please let us know.
Request any available GPU:
#SBATCH --partition=gpu
#SBATCH --gres:gpu:1Request 2 TitanRTX GPUs:
#SBATCH --partition=gpu
#SBATCH --gres=gpu:TitanRTX:2| TitanRTX | NVIDIA TitanRTX (max 4 per node) |
| T4 | NVIDIA T4 (max 16 per node) |
| A100 | NVIDIA A100 (max 4 per node) |
| A800 | NVIDIA A800 (max 8 per node) |
The most up-to-date information on what GPU types and counts are available can be queried directly from Slurm with sinfo -o %G | grep gpu.
Most people do not need to do this. Requesting node features is most relevant when we are in the middle of an OS migration, and you have jobs that are only compatible with one or the other. Another case is when your jobs have hardware requirements, or homogeneity requirements, because you are running an MPI job where CPU time would be wasted waiting for slower ranks to catch up to faster ranks. None of these situations are common with Farm jobs.
Feature constraints allow you to tell the batch system that a job must only run on a certain subset of nodes. See Slurm's documentation of the --constraint and --prefer options in the sbatch(1) and srun(1) man-pages for more detail.
farm19 node#!/bin/sh
#SBATCH --constraint=farm19
srun hostname
Generally speaking, jobs specified to require features not listed below will never run. They may queue indefinitely, or be outright rejected by the batch system upon submission. If you absolutely must continue to run such jobs, please submit your plan and timeline for no longer needing to as ServiceNow Incident. The most up-to-date information on what node features are available has can be queried directly from Slurm with sinfo -o %b.
| Feature | Nodes |
|---|---|
el9 | with AlmaLinux 9 (redundant since June 18th, 2024, now that every available node is EL9) |
farm18, farm19, farm23,farm25, sciml19, sciml21,sciml23, sciml24, sciml25 | by generation of hardware (see descriptions at scicomp.jlab.org.) farm are for CPU-only jobs andsciml for GPU jobs. |
cx4ib | having ConnectX-4 IB HCAs (a subset of farm19) |
It is intended that other requirements be able to expressed in terms of the above, e.g. Intel processors would be
farm18|sciml19|sciml25
and AMD would be
farm19|farm23|farm25|sciml21|sciml23|sciml24
Excluding the legacy mlx4 driver would be
cx4ib|farm23|farm25|sciml21|sciml23|sciml24|sciml25
Please submit a ServiceNow incident if you are unable to express your job's requirements to Slurm.
1. It used to be you couldn't access GPUs at all from production. That might no longer be the case, but you're less likely to have queuing issues if you submit to gpu for such jobs rather than production.