Slurm Feature Constraints

How to request Node features using Slurm constraints

When a job requires specific hardware or software[1] features, you can specify feature constraints. A user may specify these constraints in their sbatch file using -C or --constraint options. Functionality of AND, OR, and Matching OR operations are available. Additionally, the --prefer option can be used to specify desired but not required features. See SLURM documentation for more detail.

Example - request an EL9 node

#!/bin/bash
#SBATCH --constraint=el9
srun hostname

Example - request an EL7 Farm19 node 

#!/bin/bash
#SBATCH --constraint=el7&farm19
srun hostname

[1] Software features provide us a way to navigate operating system (OS) migrations, allowing you to test your job(s) against a particular OS configuration.

Feature List

Features not in a series listed below should be considered deprecated.  If a feature you use is deprecated, please submit a ServiceNow Incident.

Feature	Nodes
el7	with CentOS Linux 7 (default until May 21st, will not work after June 18th)
el9	with AlmaLinux 9
farm16, farm18, farm19, farm23, sciml19, sciml21, sciml23, sciml24	by generation of hardware (see descriptions at scicomp.jlab.org.) farm are for CPU-only jobs and sciml for GPU jobs.
cx4ib	having ConnectX-4 IB HCAs (a subset of farm19)

It is intended that other requirements be able to expressed in terms of the above, e.g. Intel processors for CPU-only jobs would be farm16|farm18 and AMD would be farm19|farm23.  Excluding the legacy mlx4 driver would be cx4ib|farm23.  The most up-to-date information on what node features are available has can be queried directly from Slurm with sinfo -o %b.  Please submit a ServiceNow incident if you are unable to express your job's requirements to Slurm.

Slurm GPU usage

We separate GPUs from the production partition. To use GPUs, please use the gpu partition.

How to request GPU(s)

If your job(s) require GPU access, you can specify that using --gres as documented by SLURM.

--gres=gpu:[type:]<number>

An optional type can be supplied along with the number of GPUs requested.

Examples requesting GPU type and count

Request any available GPU:

#SBATCH --partition=gpu
#SBATCH --gres:gpu:1

Request 2 TitanRTX GPUs:

#SBATCH --partition=gpu
#SBATCH --gres=gpu:TitanRTX:2

GPU Types

TitanRTX	NVIDIA TitanRTX (max 4 per node)
T4	NVIDIA T4 (max 16 per node)
A100	NVIDIA A100 (max 4 per node)
A800	NVIDIA A800 (max 4 per node)