Jacamar CI Runners

Jacamar CI is an open source project designed to be a bridge between traditional GitLab CI pipelines and HPC resources. More information about jacamar can be found in:

• Exascale Compute Project (ECP) CI Documentation

Getting Access

To utilize GitLab runners on JSC HPC systems, it’s necessary to have access to JSC GitLab with write permissions, which is only available for JSC users and partners. Within JSC GitLab, Jacamar CI runners are set up as shared runners, accessible from any project created on JSC GitLab. Additionally, employing the Jacamar runners necessitates access to the HPC systems where the CI jobs are intended to execute.

Jacamar employs an authentication mechanism that relies on the alignment of usernames between GitLab and HPC systems. Consequently, only JSC GitLab is suitable for initiating CI Jobs on JSC systems. Also, any endeavors to execute CI jobs on systems that users lack access to will be unsuccessful.

Runners Tags on JUWELS

On each of the systems, there are 2 types of runners available:

1. Slurm runners

2. Shell runners

Jacamar CI runners can be used to automate workflows that run over HPC systems. Usually on HPC systems, certain tasks entail execution on Login nodes. These tasks encompass actions like project cloning from Git repositories or preparing executables. For such tasks, Shell runners, which operate on the login nodes, are the appropriate choice.

On the other hand, when dealing with actual workloads that necessitate submission to Slurm for execution on compute nodes, using Slurm runners would be the ideal selection. Jacamar’s Slurm runners seamlessly integrate with the Slurm workload manager on the systems, streamlining the job submission process. Job options can be specified as scheduler variables within the CI definition file.

To indicate the preferred runner, distinct tags are available for each runner. These tags ensure that the CI jobs conclude on the intended system and with desired runner type. On JUWELS system, the following are the available tags and their corresponding runners:

System	Tags	Runner Type
JUWELS	juwels	Jacamar Slurm Runner
	jacamar
	compute
	slurm
JUWELS	juwels	Jacamar Shell Runner
	jacamar
	login
	shell
JUWELS_BOOSTER	juwels_booster	Jacamar Slurm Runner
	jacamar
	compute
	slurm
JUWELS_BOOSTER	juwels_booster	Jacamar Shell Runner
	jacamar
	login
	shell

Jacamar Example .gitlab-ci.yml File

To use Jacamar runners, the CI piplines should be defined in a .gitlab-ci.yml file in a Gitlab project. The following is a simple example for a .gitlab-ci.yml file:

variables:
  SCHEDULER_PARAMETERS: '-Aslurm_account -N2 --cpus-per-task 1'
  CUSTOM_CI_BUILDS_DIR: /path/to/preferred/dir

stages:
  - build
  - execute

job_juwels_login:
  id_tokens:
    SITE_ID_TOKEN:
      aud: "https://gitlab.jsc.fz-juelich.de"
  stage: build
  tags:
    - juwels
    - jacamar
    - login
    - shell
  script:
    - echo $SYSTEMNAME
  after_script:
    - hostname
    - id

job_juwels_compute:
  id_tokens:
    SITE_ID_TOKEN:
      aud: "https://gitlab.jsc.fz-juelich.de"
  stage: execute
  tags:
    - juwels
    - jacamar
    - compute
    - slurm
  artifacts:
    paths:
      - test.file
  script:
    - echo $SYSTEMNAME
    - touch test.file
  after_script:
    - hostname
    - id

Note

The example above showcases the use of both types of Jacamar runners in two separate stages. This example is meant to cover a range of scenarios, however, different use cases would require different setups.

Note

Updating GitLab to version 16.5+, any job run by Jacamar CI will no longer function without the user providing the id_tokens in their respective job. For more information, please check Migrating to new id_tokens from CI_JOB_JWT.

The following table contains description for some of the keys used in the previous example:

Key	description
SCHEDULER_PARAMETERS	SCHEDULER_PARAMETERS maps to sbatch / salloc command line arguments, more details about slurm options can be found in Summary of sbatch and srun Options. When using Jacamar Slurm runners, SCHEDULER_PARAMETERS should be defined under the variables section to set job parameters. --account or -A parameters must be set so the slurm job submitted through the CI Job would use the correct Slurm account.
CUSTOM_CI_BUILDS_DIR	By default, the output files from the jobs executed by Jacamar runners would be placed in the home directory, which has limited space available To select a different path to store the output files, CUSTOM_CI_BUILDS_DIR key can be defined under the variables section to define a preferred path.
script	script is the key where the commands to be executed in a Jacamar CI job are defined. This applies to both Jacamar Slurm and Shell CI jobs.
after_script	after_script key can be used to define some actions to be done after all the commands defined in the script section are executed. It is noteworthy to mention that the steps defined in the after_script run on the login nodes, even if slurm runners are being used. This can be used to clean up directories, or move files after the job is done.

Further information on how to write a .gitlab-ci.yml script can be found in the GitLab CI/CD documentation.