Jacamar CI Runners

Jacamar CI is an open source project designed to be a bridge between traditional GitLab CI pipelines and HPC resources. More information about jacamar can be found in:

Getting Access

To utilize GitLab runners on JSC HPC systems, it’s necessary to have access to JSC GitLab with write permissions, which is only available for JSC users and partners. Within JSC GitLab, Jacamar CI runners are set up as shared runners, accessible from any project created on JSC GitLab. Additionally, employing the Jacamar runners necessitates access to the HPC systems where the CI jobs are intended to execute.

Jacamar employs an authentication mechanism that relies on the alignment of usernames between GitLab and HPC systems. Consequently, only JSC GitLab is suitable for initiating CI Jobs on JSC systems. Also, any endeavors to execute CI jobs on systems that users lack access to will be unsuccessful.

Runners Tags on JUWELS

On each of the systems, there are 2 types of runners available:

  1. Slurm runners

  2. Shell runners

Jacamar CI runners can be used to automate workflows that run over HPC systems. Usually on HPC systems, certain tasks entail execution on Login nodes. These tasks encompass actions like project cloning from Git repositories or preparing executables. For such tasks, Shell runners, which operate on the login nodes, are the appropriate choice.

On the other hand, when dealing with actual workloads that necessitate submission to Slurm for execution on compute nodes, using Slurm runners would be the ideal selection. Jacamar’s Slurm runners seamlessly integrate with the Slurm workload manager on the systems, streamlining the job submission process. Job options can be specified as scheduler variables within the CI definition file.

To indicate the preferred runner, distinct tags are available for each runner. These tags ensure that the CI jobs conclude on the intended system and with desired runner type. On JUWELS system, the following are the available tags and their corresponding runners:

System

Tags

Runner Type

JUWELS

juwels

Jacamar Slurm Runner

jacamar

compute

slurm

JUWELS

juwels

Jacamar Shell Runner

jacamar

login

shell

JUWELS_BOOSTER

juwels_booster

Jacamar Slurm Runner

jacamar

compute

slurm

JUWELS_BOOSTER

juwels_booster

Jacamar Shell Runner

jacamar

login

shell

Jacamar Example .gitlab-ci.yml File

To use Jacamar runners, the CI piplines should be defined in a .gitlab-ci.yml file in a Gitlab project. The following is a simple example for a .gitlab-ci.yml file:

variables:
  SCHEDULER_PARAMETERS: '-Aslurm_account -N2 --cpus-per-task 1'
  CUSTOM_CI_BUILDS_DIR: /path/to/preferred/dir

stages:
  - build
  - execute

job_juwels_login:
  id_tokens:
    SITE_ID_TOKEN:
      aud: "https://gitlab.jsc.fz-juelich.de"
  stage: build
  tags:
    - juwels
    - jacamar
    - login
    - shell
  script:
    - echo $SYSTEMNAME
  after_script:
    - hostname
    - id

job_juwels_compute:
  id_tokens:
    SITE_ID_TOKEN:
      aud: "https://gitlab.jsc.fz-juelich.de"
  stage: execute
  tags:
    - juwels
    - jacamar
    - compute
    - slurm
  artifacts:
    paths:
      - test.file
  script:
    - echo $SYSTEMNAME
    - touch test.file
  after_script:
    - hostname
    - id

Note

The example above showcases the use of both types of Jacamar runners in two separate stages. This example is meant to cover a range of scenarios, however, different use cases would require different setups.

Note

Updating GitLab to version 16.5+, any job run by Jacamar CI will no longer function without the user providing the id_tokens in their respective job. For more information, please check Migrating to new id_tokens from CI_JOB_JWT.

The following table contains description for some of the keys used in the previous example:

Key

description

SCHEDULER_PARAMETERS






SCHEDULER_PARAMETERS maps to sbatch / salloc command line
arguments, more details about slurm options can be found in
Summary of sbatch and srun Options. When using Jacamar Slurm runners,
SCHEDULER_PARAMETERS should be defined under the variables
section to set job parameters. --account or -A parameters must
be set so the slurm job submitted through the CI Job would use
the correct Slurm account.
CUSTOM_CI_BUILDS_DIR




By default, the output files from the jobs executed by Jacamar runners
would be placed in the home directory, which has limited space available
To select a different path to store the output files,
CUSTOM_CI_BUILDS_DIR key can be defined under the variables
section to define a preferred path.
script

script is the key where the commands to be executed in a Jacamar CI
job are defined. This applies to both Jacamar Slurm and Shell CI jobs.
after_script




after_script key can be used to define some actions to be done after
all the commands defined in the script section are executed. It is
noteworthy to mention that the steps defined in the after_script run
on the login nodes, even if slurm runners are being used. This can be
used to clean up directories, or move files after the job is done.

Further information on how to write a .gitlab-ci.yml script can be found in the GitLab CI/CD documentation.