JURECA user documentation Logo

Contents:

  • Access
    • Prerequisites
    • SSH Login
      • Restrictions
      • OpenSSH
        • OpenSSH Installation
        • OpenSSH Key Generation
        • Key Upload, Key Restriction
        • Logging in to JURECA
        • Key Agent
        • OpenSSH Persistent Configuration
          • X Forwarding
        • Troubleshooting
        • Further Reading
      • PuTTY
        • PuTTY Installation
        • PuTTY Key Generation
        • PuTTY Persistent Configuration
    • MFA with TOTP
      • MFA Persistent Connection
    • Login Nodes
      • Example
    • Alternative Login Methods
  • Configuration
    • Hardware Configuration of the JURECA DC Module (Phase 2: as of May 2021)
    • Software Overview (as of Dec. 2018)
    • Old Configuration Information
      • Hardware Configuration of the JURECA DC Module (Phase 1: 2020 - Apr. 2021)
      • Hardware Configuration of the Cluster Module (2015 - 2020)
      • Hardware Configuration of the JURECA Booster Module (2017 - 09/2022)
      • Software Overview (until Dec. 2018)
      • Booster software stack (Dec. 2018 - Sep. 2022)
  • Environment
    • Shell
    • Active project
    • Available file systems
      • File systems for compute projects
        • Home directory ($HOME)
        • Project directory ($PROJECT)
        • Working directory ($SCRATCH)
      • File systems for data projects
        • Data directory ($DATA)
        • Archive directory ($ARCHIVE)
    • Machine identification file
    • Transferring files with scp, rsync, etc.
    • Using Git on JURECA
  • Software Modules
    • Basic module usage
    • Available compilers
    • MPI runtimes
    • GPUs and modules
    • Finding software packages
    • Stages
    • Stages Changelog
    • Scientific software at JSC
    • Requesting new software
    • Installing your own software with EasyBuild
    • European Environment for Scientific Software Installations (EESSI)
  • Building Software
    • Compiled Languages
      • Compilers
      • MPI Compiler Wrappers
      • CUDA
      • Build Systems
      • Exotic Languages
    • Interpreted Languages
      • Python
      • Julia
    • Pre-Compiled Binaries
      • Conda/Mamba
  • Batch system
    • Slurm Partitions
      • Hardware Overview
      • Available Partitions
    • Internet Access
    • Allocations, Jobs and Job Steps
    • Writing a Batch Script
      • Job Script Examples
    • Generic Resources, Features and Topology-aware Allocations
    • Job Steps
    • Dependency Chains
    • Interactive Sessions
    • Hold and Release Batch Jobs
    • Slurm commands
    • Summary of sbatch and srun Options
    • CPU Limiting Options
  • Processor Affinity
    • Slurm options
      • Terminology
      • --cpu-bind
        • Implicit types
        • Explicit types
      • --distribution
        • First part (node_level)
        • Second part (socket_level)
        • Third part (core_level)
        • Fourth part
      • --hint
    • Affinity visualization tool
    • Affinity examples
      • Default processor affinity
      • Further examples
  • MPMD: Multiple Program Multiple Data Execution Model
  • GPU Computing
    • JURECA GPU Nodes
    • GPU Visibility/Affinity
    • Setting GPU Clock Rates
    • NVIDIA Profiling Tools and Clock Speed
    • CUDA MPS: Multiple MPI Ranks per GPU
    • Job Script Examples
  • JURECA Evaluation Platform Overview
    • MI200 nodes
      • MI200 nodes Slurm considerations
      • Known Issues on MI200 nodes
        • Building for AMD GPUs
        • InfiniBand Cards
    • Graphcore IPU-POD4
      • Graphcore IPU-POD4 Slurm considerations
      • Graphcore Software
    • NVIDIA Arm HPC Dev Kit
      • Building for Arm
    • H100 node
      • H100 nodes Slurm considerations
      • H100 software modules
    • Grace Hopper nodes
      • Grace Hopper nodes Slurm considerations
      • Grace Hopper software modules
      • Known Issues on Grace Hopper nodes
        • Thermal throttling of H100 GPUs
        • Software Stage under construction
  • Visualization on JURECA
    • JURECA Visualization Nodes
    • How to Use the Visualization Nodes
    • Additional Documentation
  • GPFS File Systems in the Jülich Environment
    • Best practice notes
  • Data Transfer to and from JURECA
  • Known Issues on JURECA
    • Open Issues
    • Recently Resolved and Closed Issues
  • FAQ
    • FAQ about JURECA
      • How can Intel MPI be tuned for the JURECA InfiniBand network?
      • How can a compute node in an active allocation be accessed?
    • Error Messages on JURECA
      • What does the error message “vsetenv … failed” mean?
    • General FAQ
      • How to generate and upload ssh keys?
      • My job failed with “Transport retry count exceeded”
      • My job failed/was killed for no apparent reason
      • Estimating Power and Energy Usage of a Job
        • Power Measurements
        • Energy Measurements
    • FAQ about Data Management
      • How to access largedata on a limited number of computes within your jobs?
      • What file system to use for different data?
      • What data quotas do exist and how to list usage?
      • How to modify the users’s environment.
      • How to make the currently enabled budget visible:
      • How can I recall migrated data?
      • How can I see which data is migrated?
      • How to restore files?
        • Linux commands to manage ACLs
        • Which files have an access control list?
      • How to avoid multiple SSH connections on data transfer?
      • How to ensure correct group ID for Your project data?
      • Information about Maintenance JUNE 10-12, 2024
  • AI Workloads on JURECA
    • Working with JSC Filesystems and AI
    • Installing Python software for AI
    • Integrating with VSCode
    • Git on HPC
    • PyTorch Usage and Common Problems Guide
    • Quickstart guide for PyTorch Lightning and Hydra
    • Handling Datasets with Many Files
    • VLLM installation
    • Scraping Workflows
    • Blablador
    • Application-Specific AI documentation
  • Maintenance
  • Heterogeneous and Cross-Module Jobs
    • Heterogeneous Jobs
      • Specifying Individual Job Options
      • Running Job Components Side by Side
    • Loading Software in a Heterogeneous Environment
      • Uniform Architecture and Dependencies
      • Non Uniform Architectures and Mutual Exclusive Dependencies
  • Accounting
    • Accounting Mode
    • Command description
  • JSC Tools
    • JuDoor
    • Status Page
      • Notifications
        • Mail
        • Browser
      • Message of the Day (MOTD)
    • LLview Job Reporting
    • Pinning Tool
    • JUBE Benchmarking Environment
    • User tool jutil
      • Usage
        • Available subcommands
        • Available actions
        • Available options
      • Allowed user interfaces
    • UNICORE access
    • BLABLADOR documentation
  • Container Runtime on JURECA
    • What Containers Provide
    • Getting Access
    • Apptainer on JURECA
      • Backwards compatibility to Singularity
      • Apptainer Images
      • Launching Containers via Slurm
      • Interfacing with the System Environment
        • The two Models
        • Containerized Applications
        • User-Defined Software Stack
    • Container Build System
      • Building Container Images via CLI
      • Container Build System REST API
    • Apptainer image building
      • Converting Dockerfiles to Apptainer recipes
      • Rootless builds
      • Rootfull builds
  • Jacamar CI Runners
    • Getting Access
    • Runners Tags on JURECA
    • Jacamar Example .gitlab-ci.yml File
  • Parallel Debugging and Performance Analysis
    • Debugging with TotalView on JURECA
    • Using TotalView
    • Memory Debugging with TotalView
  • System Changelog
    • Current state
      • Installed software
    • Changelog entries
      • 2025-10-16 Software update
        • Update type: OS Packages
          • OS Packages
      • 2025-09-23 Software update
        • Update type: OS Packages
          • OS Packages
      • 2025-09-22 Update UCX
        • Update type: SW Modules
      • 2025-09-09 Software update
        • Update type: SW Modules
          • UCX-settings
      • 2025-09-02 Software update
        • Update type: OS Packages
          • OS Packages
      • 2025-07-22 Software update
        • Update type: OS Packages
          • OS Packages
      • 2025-07-01 Software update
        • Update type: OS Packages
          • OS Packages
      • 2025-06-17 Software update
        • Update type: OS Packages and Firmware
          • Firmware
          • OS Packages
      • 2025-04-29 Software update
        • Update type: OS Packages
          • OS Packages
      • 2025-03-13 Maintenance
        • Extension of HWAI partition
        • Slurm configuration changes
        • Update type: OS Packages
          • OS Packages
      • 2025-02-27 MemoryMax
        • Update type: Login nodes
      • 2025-02-05 Change MPI-settings for OpenMPI
        • Update type: SW Modules
      • 2025-01-29 Default UCX-settings module
        • Update type: SW Modules
      • 2024-12-11 Software update
        • Update type: OS Packages
          • OS Packages
      • 2024-11-26 Software update
        • Update type: OS Packages
          • OS Packages
      • 2024-10-18 Maintenance
        • Configuration changes
        • Update type: OS Packages
          • OS Packages
      • 2024-08-06 Software update
        • Update type: OS Packages
          • OS Packages
      • 2024-06-17 Software update
        • Update type: OS Packages
          • OS Packages
      • 2024-03-07 Software update
        • Update type: OS Packages
          • OS Packages:
      • 2024-02-29 Software update
        • Update type: OS Packages
          • OS Packages:
      • 2024-02-19 Software update
        • Update type: OS Packages
          • OS Packages:
      • 2024-01-22 Software update
        • Update type: OS Packages
          • OS Packages:
      • 2024-01-16 Software update
        • Update type: OS Packages, Batch system
          • OS Packages:
          • Batch System:
      • 2023-12-14 Software update
        • Update type: OS Packages, Batch system, SW Modules
          • OS Packages:
          • Software stack
      • 2023-11-02 PMIx update
        • Update type: OS Packages
          • Packages:
          • Configuration:
      • 2023-10-19 Software update
        • Update type: OS Packages, Firmware, Batch system, Configuration
          • Packages:
          • Configuration:
      • 2023-08-30 UCX-settings update
        • Update type: SW Modules
      • 2023-07-27 Software update
        • Update type: OS Packages, Batch system
      • 2023-07-03–2023-07-10 TS Update
        • Update type: Other
      • 2023-06-27 Rolling update
        • Update type: Rolling update, Slurm
          • Software Updates:
      • 2023-06-05 Rolling update
        • Update type: Rolling update, PSMgmt
          • Software Updates:
      • 2023-05-16 Rolling update
        • Update type: Rolling update, OS Packages, Storage
          • Software Updates:
      • 2023-03-09 Emergency maintenance/update
        • Update type: Maintenance, OS Packages, Storage, Skyways
          • Skyways:
          • Software Updates:
      • 2023-02-28 10:00 to 2023-02-28 13:30 offline Maintenance
        • Update type: Maintenance, SW Modules, Batch system, OS Packages, Firmware
          • Stage Update:
          • Slurm Update:
          • Software Updates:
          • Firmware Updates:
      • 2023-02-15 11:00 to 2023-02-16 13:15 online Maintenance
        • Update type: Maintenance, Firmware
          • Firmware Updates:
      • 2022-11-29 07:00 to 17:23 offline Maintenance
        • Update type: Maintenance, SW/FW/HW
          • SW Updates:
          • FW Updates:
          • Service Storage change:
          • GPFS GW OS Update Skyways:
      • 2022-11-02 09:00 to 18:26 offline Maintenance
        • Update type: Maintenance, SW/FW/HW
          • JURECA-DC:
          • JURECA Booster:
      • 2022-08-23 08:30 to 17:28 offline Maintenance
        • Update type: Maintenance, SW/FW
          • JURECA-DC:
          • JURECA Booster:
      • 2022-06-28 09:00 to 12:56 online Maintenance
        • Update type: Maintenance, SW
          • JURECA:
      • 2022-06-09 10:00 to 15:58 online Maintenance
        • Update type: Maintenance, SW/FW
          • JURECA-DC:
      • 2022-05-31 08:00 planned Offline Maintenance Booster Module
        • Update type: Maintenance, HW
          • JURECA Booster: (last service and support day -> all open HW-Tickets got addressed)
      • 2022-05-12 Offline maintenance
        • Update type: Maintenance, HW
          • JURECA:
      • 2022-05-03 Global maintenance with general updates
        • Update type: Maintenance, HW + SW
          • JURECA:
      • 2022-04-08 10:00 to 11:09 online Maintenance
        • Update type: Maintenance, SW
          • JURECA:
      • 2022-03-15&17 online Maintenance
        • Update type: Maintenance, SW
          • JURECA:
      • 2022-03-08 08:30 planned Offline Maintenance
        • Update type: Maintenance, SW, HW
          • JURECA:
          • JURECA-DC including logins and service island hosts+container:
      • 2022-03-08 Change in user installations
        • Update type: Announcement, SW Modules
          • Change in user installations
      • 2021-12-13 08:30 to 2021-12-15 10:00 Maintenance
        • Update type: Maintenance, SW, HW
          • JURECA:
          • JURECA-DC including logins and service island hosts+container:
          • JURECA-Booster
      • 2021-12-06 emergency Maintenance
      • 2021-11-04 Maintenance
        • Update type: Maintenance, SW, Acceptance tests
          • JURECA:
          • JURECA-DC including logins:
          • JURECA-Booster
      • 2021-11-03 emergency Maintenance
      • 2021-09-14 Maintenance
        • Update type: Maintenance, SW
          • JURECA:
          • JURECA DC
      • 2021-08-18 emergency Maintenance
      • 2021-07-22 Maintenance
        • Update type: Maintenance, SW, Acceptance tests
          • JURECA:
          • JURECA-DC:
      • 2021-07-15 Maintenance
        • Update type: Maintenance, HW, SW
          • JURECA:
          • JURECA-DC:
      • 2021-07-07 to 2021-07-08 and 2021-07-14 JURECA-DC Module reservation
      • 2021-06-29 Maintenance
        • Update type: Maintenance, HW, SW
          • JURECA:
          • JURECA-DC:
      • 2021-06-17 Maintenance
        • Update type: Maintenance, HW, SW
          • JURECA:
          • JURECA DC
          • JURECA Booster
      • 2021-06-07 08:30 to 2021-06-09 01:18 Maintenance
        • Update type: Maintenance, HW, SW
          • JURECA:
          • JURECA-DC:
      • 2021-05-31 to 2021-06-01 JURECA-DC Module maintenance reservation
      • 2021-05-27 to 2021-05-28 Maintenance
        • Update type: Maintenance, SW, Benchmarks
          • JURECA:
          • JURECA-DC:
      • 2021-05-20 Maintenance
        • Update type: Maintenance, SW, HW
          • JURECA:
          • JURECA-Booster:
          • JURECA-DC:
      • 2021-05-12 Maintenance without closing the logins
        • Update type: Maintenance, SW
          • JURECA:
            • JURECA-DC: ISSUES WITH CUDA_VISIBLE_DEVICES on PHASE 2 GPU NODES:
      • 2021-05-06 Maintenance
        • Update type: Maintenance, SW
          • JURECA:
          • JURECA-Booster:
          • JURECA-DC:
          • JURECA-TEST:
      • 2021-04-29 to 2021-04-30 Maintenance
        • Update type: Maintenance, Acceptance
          • JURECA-DC:
      • 2021-04-22 Maintenance
        • Update type: Maintenance, OS, SW
          • JURECA-DC:
          • JURECA DC/BOOSTER:
      • 2021-04-13 to 2021-04-15 Maintenance
        • Update type: Maintenance
          • JURECA-DC:
          • JURECA-TEST:
          • JURECA DC/BOOSTER:
          • JURECA-DC: HPST
          • Slurm:
      • 2021-04-08 Maintenance
        • Update type: Maintenance
          • JURECA-DC:
          • JURECA-DC: HPST
          • Slurm:
      • 2021-03-25 Maintenance
        • Update type: Maintenance
          • JURECA-DC:
          • JURECA-DC: HPST
          • Slurm:
      • 2021-03-18 Maintenance
        • Update type: Maintenance
          • JURECA-DC: Infiniband fabric cleanup (ports reseat, replace / switch fw updates/reboots…)
          • JURECA-DC: HPST
          • MPI-GW DC/Booster
          • Atos SBF software updates
          • GPFS at DC
      • 2021-03-11 Maintenance
        • Update type: Maintenance
          • psmgmt-5.1.39-2 update on JURECA
          • JURECA-DC: Infiniband conclusion of Phase1 and Phase2 installation
          • JURECA-DC: HPST connected to IB-Fabric again after replacing all cables
          • JURECA-Booster: OPA
      • 2021-03-02 Maintenance
        • Update type: Maintenance, SW Modules
          • psmgmt-5.1.38-2 update on JURECA
          • UCX has been changed to 1.9.0 from 1.8.1 for both ParaStationMPI and OpenMPI (easybuild)
      • 2021-02-25 Maintenance
        • Update type: Maintenance, SW Modules
          • pscom with gateway support added to JURECA-DC Easybuild stack
          • recompilation GCC at JURECA-DC (easybuild)
          • JURECA-DC: HPST maintenance/umounted until 2021-03-11 due to IB-Cable renew
      • Installation JURECA-DC module (including new Service Island for JURECA)
        • Phase1 end of 2020
          • JR-DC and JR-Booster update to CentOS 8.2
        • Phase2 beginning of 2021
      • 2020-07-16 Maintenance
        • Update type: Maintenance
          • Row0[2,3] - shutdown all located components
          • Slurm - delete all components at Row0[2,3] from config
          • Move Singularity installation from Easybuild to RPM
      • 2020-06-23 Maintenance
        • Update type: Maintenance
          • Reenable mmpmon/gpumon and add to HC
          • Bring IME into production
          • On booster jobs fail with OPA errors (libpsm2 bug)
      • 2020-02-04 New PGI compiler and Intel MPI version
        • Update type: SW Modules
      • 2020-01-21 Maintenance-HPST-IB-extension
        • Update type: Maintenance, Network
          • MVAPICH issues due to wrong infiniband-diags version
          • HPST IB extension - 3 Line-Cars and 72 Infiniband cables
          • Update IME Client Software
      • 2019-12-10 Maintenance
        • Update type: Maintenance, OS Packages, SW Modules
          • Update Slurm to 19.05.4-1.20191203git1b8453f491
          • jutil: update to version 19.12.0
          • Flexible module naming scheme
          • Update to CentOS 7.7
      • 2019-11-05 Maintenance 2019-11-05
        • Update type: Maintenance, Batch system
          • UFM REST API not showing all nodes connected to the fabric
          • JR-Booster WCDs/PDUs FW update - RPC2 communications module 14.0.0.3
          • Mellanox FW update
          • gdrcopy for JURECA
          • Update psmgmt-5.1.26-0
          • Booster Firmware Update
          • Update Slurm to 19.05
      • 2019-10-30 Beginning of the changelog
        • Update type: Announcement
  • Support
JURECA user documentation
  • Search

© 2025, JURECA administrators. Documentation distributed under CC BY-NC-SA 4.0.

  • Legal Notice
  • Data Protection
  • Accessibility

Built with Sphinx using a theme provided by Read the Docs.