Configuration

JUPITER Hardware Overview

JUPITER Booster Design

  • 5884 standard compute nodes (accelerated)
    • 4× NVIDIA GH200 Grace-Hopper Superchip (see also node image)

      • CPU: NVIDIA Grace (Arm Neoverse-V2), 72 cores at 3.1 GHz base frequency; 120 GB LPDDR5X memory at 512 GB/s (8532 MHz)

      • GPU: NVIDIA Hopper, 132 multiprocessors, 96 GB HBM3 memory at 4 TB/s

      • NVIDIA NVLink-C2C CPU-to-GPU link at 900 GB/s

      • TDP: 680 W (for full GH200 superchip)

    • NVLink 4 GPU-to-GPU links, 300 GB/s between pairs of GPUs (150 GB/s per direction), cNVLink between CPUs (pairs connected with 100 GB/s per direction)

    • Network: 4× InfiniBand NDR200 (Connect-X7)

  • 12 login nodes
    • 1× NVIDIA GH200 Grace-Hopper Superchip (differences highlighted)

      • CPU: NVIDIA Grace (Arm Neoverse-V2), 72 cores at 3.1 GHz base frequency; 480 GB LPDDR5X memory at 384 GB/s (6400 MHz)

      • GPU: NVIDIA Hopper, 132 multiprocessors, 96 GB HBM3 memory at 4 TB/s

      • NVIDIA NVLink-C2C CPU-to-GPU link at 900 GB/s

      • TDP: 900 W (for full GH200 superchip)

    • Network: InfiniBband NDR (Connect-X7)

    • 100 Gigabit Ethernet external connection

    • Local disk for operating system (1× 960 GB NVME)

  • Same fabric as Cluster, ExaSTORE, and ExaFLASH; direct InfiniBand connectivity

_images/jupiter-node-design--jedi.svg

Node diagram of the 4× NVIDIA GH200 node design of JUPITER Booster / JUPITER. Links and bandwidths are added.

JUPITER Cluster Design

The JUPITER Cluster is currently being built up.

  • 582 AMD nodes (will be installed soon)
    • 2× AMD Turin CPU

      • CPU: AMD Turin 9655 (x86-64), 96 cores at 2.6 GHz (SMT-2; AVX-512 with 512 bit data path)

    • Memory:

      • 546 nodes with 768 GB DDR5

      • 36 nodes with 1563 GB DDR5

    • Network: 1× InfiniBand NDR200 (Connect-X7)

  • 162 SiPearl nodes (will be installed later)
    • 2× SiPearl Rhea1 CPU

      • CPU: SiPearl Rhea 1 (Arm Neoverse-V1), 80 cores (2×256 bit SVE), 64 GB HBM2e memory

    • 512 GB memory (per node; in addition to the 2×64 GB HBM)

    • Network: 1× InfiniBand NDR200 (Connect-X7)

  • Same fabric as Booster, ExaSTORE, and ExaFLASH; direct InfiniBand connectivity

Network Design

JUPITER features a common network based on NVIDIA Mellanox Quantum InfiniBand NDR in a Dragonfly+ topology. 27 Dragonfly groups are available, of which 25 belong to JUPITER Booster, 1 to JUPITER Cluster, and 1 to the administrative sub-systems. The topology is shown in the overview image. JUPITER features adaptive routing.

_images/jupiter-dragonfly.svg

Connectivity of the JUPITER Dragonfly+ Network

A zoom into one of the JUPITER Booster Dragonfly groups can be seen in the following image.

_images/jupiter-dragonfly-cell.svg

In-group view of a JUPITER Booster DragonFly group.

240 nodes are combined in one DragonFly group, distributed over 5 physical racks. There are 15 L1 (lower) switches, and 16 L2 (higher) switches; so 3 switches per type per rack, with the 3rd rack having an additional L2 switch. Nodes are connected with split-cables to the L1 switches with 200 Gbit/s per HCA (800 Gbit/s per node). Between switches, connectivity with 400 Gbit/s per link is available.

Software Overview

  • Redhat Enterprise Linux (RHEL) 9 distribution

  • JUPITER Management Stack

    • Eviden Smart Management Center xScale

  • Scientific Software (EasyBuild)

    • Compilers: GCC, NVIDIA HPC Compiler (NVHPC)

    • MPI: OpenMPI, ParaStationMPI; both CUDA-aware

    • And more components, see module overview

  • IBM Storage Scale (GPFS) parallel file system