GPU Computing

JUSUF GPU Nodes

JUSUF features a number of accelerated (GPU-equipped) compute nodes for applications that can take advantage of such devices via CUDA, OpenCL or OpenACC.

The GPU compute nodes feature one NVIDIA V100 PCIe GPU. The nodes are accessible in the gpus partition. In order to access this partition the argument -p gpus (or --partition gpus) must be provided to sbatch or salloc. As there is only one GPU per node no additional option for the number of GPUs has to be given (--gres=gpu:1 is the default).

GPU Visibility/Affinity

Through the job scheduler (Slurm), GPUs are associated to tasks. Per default, one task is associated to one GPU. This is achieved by setting the environment variable CUDA_VISIBLE_DEVICES which steers available GPUs to an application.

Consider the following example:

$ srun --tasks 4 bash -c 'echo "Rank: $PMI_RANK   CUDA_VISIBLE_DEVICES: $CUDA_VISIBLE_DEVICES"' | sort
Rank: 0   CUDA_VISIBLE_DEVICES: 0
Rank: 1   CUDA_VISIBLE_DEVICES: 1
Rank: 2   CUDA_VISIBLE_DEVICES: 2
Rank: 3   CUDA_VISIBLE_DEVICES: 3

In this case, task with ID 0 has access to GPU with ID 0, task with ID 1 has access to GPU with ID 1, and so on. An application launched like this with srun will see only one GPU with the app-internal ID always being 0. In other words: In the default, each GPU-using application will see one GPU per task (/rank). It is ensured that the process which accesses this GPU will be launched on a CPU core which has affinity to the GPU.

The default behavior can be changed by overriding CUDA_VISIBLE_DEVICES before the srun invocation. In that case, the value of the environment variable is not changed.

Note

Also for jobs with only one task, the behavior is the same currently. In that case, only one GPU is visible to task. Please manually export CUDA_VISIBLE_DEVICES=0,1,2,3. This behavior is currently being worked on and will soon be modified.

Job Script Examples

Example: MPI application starting 96 tasks on 4 nodes using 24 CPUs per node and 1 GPU per node:

#!/bin/bash -x
#SBATCH --account=<budget>
#SBATCH --nodes=4
#SBATCH --ntasks=96
#SBATCH --ntasks-per-node=24
#SBATCH --output=gpu-out.%j
#SBATCH --error=gpu-err.%j
#SBATCH --time=00:15:00
#SBATCH --partition=gpus

[#SBATCH --gres=gpu:1]

srun ./gpu-prog