.. include:: system.rst Environment =========== This article describes the system environment, especially important file system locations and the concept of `active project`. Shell ----- The login shell on all servers in the |SYSTEM_NAME| Cluster is ``/bin/bash``. The persistent settings of the shell environment are governed by the content of ``$HOME/.bashrc``, ``$HOME/.profile`` or scripts sourced from within these files. Please use these files for storing your personal settings. It is not possible to change the login shell but users may switch to a personal shell within the login process. However, please note that only bash is fully supported on |SYSTEM_NAME| and usage of alternative shells may degenerate the user experience on the system. .. _environment_active_project: Active project -------------- Through their association with projects, users of |SYSTEM_NAME| are given access to both computational resources (a core-h budget) and storage resources (write access to directories on certain file systems with associated quotas). The locations of these directories are exposed via environment variables, e.g. ``$PROJECT``. However, since a user can be a member of several projects at the same time, the names of these environment variables have to contain the project name in order to avoid collisions: ``$PROJECT_``. Using the ``jutil`` command line utility, a project can be made the `active project`: .. code-block:: none $ jutil env activate -p All environment variables pointing to storage resources associated with ```` will be re-exported in their suffix-less form, e.g. ``$PROJECT_`` will be re-exported as just ``$PROJECT``. ```` will stay the active project until another project is made the active project or until the shell session ends. For more information look at the :ref:`jutil ` command usage. Available file systems ---------------------- The available parallel file systems on |SYSTEM_NAME| are mounted from JUST_. The following table gives an overview over the available file systems: .. note:: Except for ``$HOME`` the environment variables in the table below are only available once a project has been activated, see :ref:`environment_active_project`. The project specific forms are always available, e.g. ``$PROJECT_``. .. ifconfig:: system_name in ('jusuf') +-------------------+----------------------+-----------------+--------------------------------------------------------------------+ | Variable | Storage Location | Accessibilty | Description | +===================+======================+=================+====================================================================+ | ``$HOME`` | parallel file system | Login + Compute | Storage of user specific data (e.g. ssh-key) | +-------------------+----------------------+-----------------+--------------------------------------------------------------------+ | ``$PROJECT`` | parallel file system | Login + Compute | Storage of project related source code, binaries, etc. | +-------------------+----------------------+-----------------+--------------------------------------------------------------------+ | ``$SCRATCH`` | parallel file system | Login + Compute | Scratch file system for temporary data | +-------------------+----------------------+-----------------+--------------------------------------------------------------------+ | ``$CSCRATCH`` | flash cache system | Login + Compute | NVMe based cache layer for ``$SCRATCH`` file system | +-------------------+----------------------+-----------------+--------------------------------------------------------------------+ | ``$FASTDATA`` | parallel file system | Login + Compute | Storage location for large data (JUSTDSS) | +-------------------+----------------------+-----------------+--------------------------------------------------------------------+ | ``$DATA`` | parallel file system | Login | Storage location for large data (XCST) | +-------------------+----------------------+-----------------+--------------------------------------------------------------------+ | ``$ARCHIVE`` | parallel file system | Login | Storage location for archiving on tape | +-------------------+----------------------+-----------------+--------------------------------------------------------------------+ | ``$LOCALSCRATCH`` | Node-local disk | Compute | Scratch file system for temporary data, deleted after job finished | +-------------------+----------------------+-----------------+--------------------------------------------------------------------+ The same file systems are available on JUDAC_ data gateway which is the recommended system for initiating transfers of large files. .. ifconfig:: system_name in ('juwels', 'jureca') +-------------------+----------------------+-----------------+--------------------------------------------------------------------+ | Variable | Storage Location | Accessibilty | Description | +===================+======================+=================+====================================================================+ | ``$HOME`` | parallel file system | Login + Compute | Storage of user specific data (e.g. ssh-key) | +-------------------+----------------------+-----------------+--------------------------------------------------------------------+ | ``$PROJECT`` | parallel file system | Login + Compute | Storage of project related source code, binaries, etc. | +-------------------+----------------------+-----------------+--------------------------------------------------------------------+ | ``$SCRATCH`` | parallel file system | Login + Compute | Scratch file system for temporary data | +-------------------+----------------------+-----------------+--------------------------------------------------------------------+ | ``$CSCRATCH`` | flash cache system | Login + Compute | NVMe based cache layer for ``$SCRATCH`` file system | +-------------------+----------------------+-----------------+--------------------------------------------------------------------+ | ``$FASTDATA`` | parallel file system | Login | Storage location for large data (JUSTDSS) | +-------------------+----------------------+-----------------+--------------------------------------------------------------------+ | ``$DATA`` | parallel file system | Login | Storage location for large data (XCST) | +-------------------+----------------------+-----------------+--------------------------------------------------------------------+ | ``$ARCHIVE`` | parallel file system | Login | Storage location for archiving on tape | +-------------------+----------------------+-----------------+--------------------------------------------------------------------+ .. ifconfig:: system_name in ('judac') +-------------------+----------------------+-----------------+-------------------------------------------------------------------+ | Variable | Storage Location | Accessibilty | Description | +===================+======================+=================+===================================================================+ | ``$HOME`` | parallel file system | Access node | Storage of user specific data (e.g. ssh-key) | +-------------------+----------------------+-----------------+-------------------------------------------------------------------+ | ``$PROJECT`` | parallel file system | Access node | Storage of project related source code, binaries, etc. | +-------------------+----------------------+-----------------+-------------------------------------------------------------------+ | ``$SCRATCH`` | parallel file system | Access node | Scratch file system for temporary data | +-------------------+----------------------+-----------------+-------------------------------------------------------------------+ | ``$FASTDATA`` | parallel file system | Access node | Storage location for large data (JUSTDSS) | +-------------------+----------------------+-----------------+-------------------------------------------------------------------+ | ``$DATA`` | parallel file system | Access node | Storage location for large data (XCST) | +-------------------+----------------------+-----------------+-------------------------------------------------------------------+ | ``$ARCHIVE`` | parallel file system | Access node | Storage location for archiving on tape | +-------------------+----------------------+-----------------+-------------------------------------------------------------------+ It is highly recommended to access files always with help of these variables. File systems for compute projects ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Within the current usage model file systems are bound to compute or data projects. The following description is just an overview on how to use these file systems. For further information, please see: :ref:`What file system to use for different data?` Each compute project has access to the following file systems. Home directory ($HOME) """""""""""""""""""""" Home directories reside in the parallel file system. In order to hide the details of the home file system layout the full path to the home directory of each user is stored in the shell environment variable ``$HOME``. References to files in the home directory should always be made through the ``$HOME`` environment variable. The initialization of ``$HOME`` will be performed during the login process. The Home directory is limited in space and should only be used for storing smal user-specific data items (e.g. ssh-keys, configuration files). Project directory ($PROJECT) """""""""""""""""""""""""""" Project directories reside in the parallel file system, too. In order to hide the details of the project file system layout the full path to these directories is stored in shell environment variables. As an account can be bound to several projects the variables are marked accordingly ``$PROJECT_``. The data migrated at the transition to the new usage model was moved to the user-owned subdirectory ``$PROJECT_/account``. Please note that the project directory itself is writable for the project members, i.e. different organization schemes within the project (for example to enable easier sharing of data) are possible and entirely in the hand of the project PI and members. To activate a certain project for a current session or switch between projects one can use the tool :ref:`jutil `. During activation of a project environment variables will be exported and the environment variable ``$PROJECT`` is set to ``$PROJECT_``. This tool can also be used to perform tasks like querying project information/cpu and data quota. Working directory ($SCRATCH) """""""""""""""""""""""""""" Scratch directories are temporary storage locations residing in the parallel file system. They are used for applications with large size and I/O demands. Data in these directories are only for temporary use, they are automatically deleted (files after 90 days by modification and access date, empty directories after 3 days). The structure of the scratch directory and the corresponding environment variables are similar to the project directory. File systems for data projects ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File systems for data projects are used to store large data. The structure and environment variables are similar to ``$PROJECT`` and ``$SCRATCH``. Data projects have to be explicitly applied for and are independent from compute projects. Data directory ($FASTDATA) """""""""""""""""""""""""" Fastdata directories are used for applications with large data and I/O demands similar to the scratch file system. Contrary to ``$SCRATCH`` data in ``$FASTDATA`` is permanent and protected with snapshots. Data directory ($DATA) """""""""""""""""""""" Data directories are used to store a huge amount of data on disk based storage. The bandwidth is lower than in ``$FASTDATA``. Access to these directories is available from login nodes only. Archive directory ($ARCHIVE) """""""""""""""""""""""""""" Archive directories are used to store all files not in use for a longer time; data are migrated to tape storage by ISP-HSM (IBM Spectrum Protect for Space Management). Machine identification file --------------------------- To simplify users the handling of the shared ``$HOME`` file system on the different supercomputers JSC provides a machine identification file ``/etc/FZJ/systemname`` on all systems. ``/etc/FZJ/systemname`` stores the system name (such as juwels, jureca, jusuf, ...) and can be used perform system specific actions without the need to parse the hostname of the login or compute nodes. Below an example for the handling of different machines e.g. in ``.profile`` or ``.bashrc`` is provided: .. code-block:: none MACHINE=$(cat /etc/FZJ/systemname) if test "${MACHINE}" = "juwels"; then # perform JUWELS specific acctions elif test "${MACHINE}" = "jureca" ; then # perform JURECA specific actions fi The machine name can also be read within a Makefile using: .. code-block:: none $(shell cat /etc/FZJ/systemname) .. _file transfer: Transferring files with ``scp``, ``rsync``, etc. ------------------------------------------------ Since outgoing SSH connections are not allowed, file transfers to and from |SYSTEM_NAME| which use SSH as the underlying transport have to be initiated from the other system. So instead of .. ifconfig:: system_name == 'juwels' .. code-block:: none juwels$ scp my_file local: .. ifconfig:: system_name == 'jureca' .. code-block:: none jureca$ scp my_file local: .. ifconfig:: system_name == 'jusuf' .. code-block:: none jusuf$ scp my_file local: .. ifconfig:: system_name == 'judac' .. code-block:: none judac$ scp my_file local: you have to initiate the copy from the local system: .. ifconfig:: system_name == 'juwels' .. code-block:: none local$ scp juwels.fz-juelich.de:my_file . .. ifconfig:: system_name == 'jureca' .. code-block:: none local$ scp jureca.fz-juelich.de:my_file . .. ifconfig:: system_name == 'jusuf' .. code-block:: none local$ scp jusuf.fz-juelich.de:my_file . .. ifconfig:: system_name == 'judac' .. code-block:: none local$ scp judac.fz-juelich.de:my_file . In some cases, it might not be possible to directly transfer files between another system and |SYSTEM_NAME|. This might be, because the other system also disallows outgoing SSH connections, or the SSH client on the other system is too old and does not support the modern cryptographic algorithms required by JSC policy. As a work around, the files have to be transferred to a third system which can make connections to both |SYSTEM_NAME| and the other system. This can be automated with ``scp`` and its command line argument ``-3``: .. ifconfig:: system_name == 'juwels' .. code-block:: none local$ scp -3 other.hpc.example.com:my_file juwels.fz-juelich.de: .. ifconfig:: system_name == 'jureca' .. code-block:: none local$ scp -3 other.hpc.example.com:my_file jureca.fz-juelich.de: .. ifconfig:: system_name == 'jusuf' .. code-block:: none local$ scp -3 other.hpc.example.com:my_file jusuf.fz-juelich.de: .. ifconfig:: system_name == 'judac' .. code-block:: none local$ scp -3 other.hpc.example.com:my_file judac.fz-juelich.de: .. note:: An internet connection that provides fast speeds in both the upload and download direction is recommended for this approach. .. _Git HPC: Using Git on |SYSTEM_NAME| -------------------------- Since outgoing SSH connections are not allowed on |SYSTEM_NAME| and SSH authentication is not possible, it can be quite challenging to clone a Git repository. Here are some alternatives and possible workarounds. * Use ``https`` instead of ``ssh`` for Git: .. ifconfig:: system_name == 'juwels' .. code-block:: none juwels$ git clone https://github.com/libgit2/libgit2 ~/libgit2/ .. ifconfig:: system_name == 'jureca' .. code-block:: none jureca$ git clone https://github.com/libgit2/libgit2 ~/libgit2/ .. ifconfig:: system_name == 'jusuf' .. code-block:: none jusuf$ git clone https://github.com/libgit2/libgit2 ~/libgit2/ .. ifconfig:: system_name == 'judac' .. code-block:: none judac$ git clone https://github.com/libgit2/libgit2 ~/libgit2/ Existing repositories can be changed by modifying the URL of the remote: ``git remote set-url origin https://github.com/libgit2/libgit2``. Instead of authenticating using username and password of the Git hosting service for a http-based checkout, we strongly recommend using **Personal Access Tokens**. They can be configured with limited permissions (only push and pull, only pull/no push, …) and don't allow for accessing your full account on the Git hosting service. They can be configured through the website of the respective Git hosting service. With access tokens, `Git Credential Helpers `_ can be useful. .. note:: For users of the JSC Gitlab, Access Tokens are even more strongly recommended (as the JSC Gitlab uses the same LDAP credentials as JuDoor). * Use SSHFS: - On your local machine, install `SSHFS `_ (it might be there already) - Mount a directory of |SYSTEM_NAME| to your local machine; you can access files and directories as if they were local, but they actually still *live* on |SYSTEM_NAME| and are only copied on demand .. ifconfig:: system_name == "juwels" .. code-block:: none local$ sshfs juwels:~/ssh-fs/ ~/juwels-mount/ .. ifconfig:: system_name == "jureca" .. code-block:: none local$ sshfs jureca:~/ssh-fs/ ~/jureca-mount/ .. ifconfig:: system_name == "jusuf" .. code-block:: none local$ sshfs jusuf:~/ssh-fs/ ~/jsuf-mount/ .. ifconfig:: system_name == "judac" .. code-block:: none local$ sshfs judac:~/ssh-fs/ ~/jsuf-mount/ - Clone repository into your local directory which will automatically upload each file and directory to |SYSTEM_NAME| .. ifconfig:: system_name == "juwels" .. code-block:: none local$ git clone git@github.com:libgit2/libgit2.git ~/juwels-mount/ .. ifconfig:: system_name == "jureca" .. code-block:: none local$ git clone git@github.com:libgit2/libgit2.git ~/jureca-mount/ .. ifconfig:: system_name == "jusuf" .. code-block:: none local$ git clone git@github.com:libgit2/libgit2.git ~/jusuf-mount/ .. ifconfig:: system_name == "judac" .. code-block:: none local$ git clone git@github.com:libgit2/libgit2.git ~/judac-mount/ - Attention: This is slow! Have a look at the `mounting options `_ mentioned in the man page to potentially speed-up the process. * Use a bare Git repository on |SYSTEM_NAME| as a proxy: - Create bare repository on |SYSTEM_NAME|: .. ifconfig:: system_name == "juwels" .. code-block:: none juwels$ git init --bare ~/.my-bare-repo.git .. ifconfig:: system_name == "jureca" .. code-block:: none jureca$ git init --bare ~/.my-bare-repo.git .. ifconfig:: system_name == "jusuf" .. code-block:: none jusuf$ git init --bare ~/.my-bare-repo.git .. ifconfig:: system_name == "judac" .. code-block:: none judac$ git init --bare ~/.my-bare-repo.git - Locally, clone the original Git repository, add a new remote (the bare repository), and push everything: .. ifconfig:: system_name == "juwels" .. code-block:: none local$ git clone https://github.com/octocat/Hello-World && cd Hello-World local$ git remote add mirror-juwels juwels:~/.my-bare-repo.git/ local$ git push mirror-juwels master .. ifconfig:: system_name == "jureca" .. code-block:: none local$ git clone https://github.com/octocat/Hello-World && cd Hello-World local$ git remote add mirror-jureca jureca:~/.my-bare-repo.git/ local$ git push mirror-jureca master .. ifconfig:: system_name == "jusuf" .. code-block:: none local$ git clone https://github.com/octocat/Hello-World && cd Hello-World local$ git remote add mirror-jusuf jusuf:~/.my-bare-repo.git/ local$ git push mirror-jusuf master .. ifconfig:: system_name == "judac" .. code-block:: none local$ git clone https://github.com/octocat/Hello-World && cd Hello-World local$ git remote add mirror-judac judac:~/.my-bare-repo.git/ local$ git push mirror-judac master - On |SYSTEM_NAME|, now create a *proper* (non-bare) repository based on the bare repository, add some changes as an example, and push to the bare repository: .. ifconfig:: system_name == "juwels" .. code-block:: none juwels$ git clone ~/.my-bare-repo.git Hello-World && cd Hello-World juwels$ touch a-new-file && git add a-new-file && git commit -m "A new file" juwels$ git push origin master .. ifconfig:: system_name == "jureca" .. code-block:: none jureca$ git clone ~/.my-bare-repo.git Hello-World && cd Hello-World jureca$ touch a-new-file && git add a-new-file && git commit -m "A new file" jureca$ git push origin master .. ifconfig:: system_name == "jusuf" .. code-block:: none jusuf$ git clone ~/.my-bare-repo.git Hello-World && cd Hello-World jusuf$ touch a-new-file && git add a-new-file && git commit -m "A new file" jusuf$ git push origin master .. ifconfig:: system_name == "judac" .. code-block:: none judac$ git clone ~/.my-bare-repo.git Hello-World && cd Hello-World judac$ touch a-new-file && git add a-new-file && git commit -m "A new file" judac$ git push origin master - Finally, get the changes from |SYSTEM_NAME| back to your local copy and push them to the original origin: .. ifconfig:: system_name == "juwels" .. code-block:: none local$ git pull mirror-juwels master local$ git push origin master .. ifconfig:: system_name == "jureca" .. code-block:: none local$ git pull mirror-jureca master local$ git push origin master .. ifconfig:: system_name == "jusuf" .. code-block:: none local$ git pull mirror-jusuf master local$ git push origin master .. ifconfig:: system_name == "judac" .. code-block:: none local$ git pull mirror-judac master local$ git push origin master * Check out the Git repository locally, use ``rsync`` to two-way-sync to/from |SYSTEM_NAME| * *Note on Submodules*: Git repositories can have other Git repositories as external dependencies; so-called *submodules*. Since the address of the submodule is hard-coded into ``.gitmodules``, it might include incompatible ``ssh://`` URLs. In most cases, the ``ssh://`` part can be replaced with ``https://`` (as shown above) within the ``.gitmodules`` modules. Changes within this file need to be transported to the ``.git`` configuration directory by calling ``git submodule sync``. .. external links .. _JUST: https://www.fz-juelich.de/en/ias/jsc/systems/storage-systems/just .. _JUDAC: https://www.fz-juelich.de/en/ias/jsc/systems/storage-systems/judac