.. include:: system.rst

.. _uftp:

UFTP Service
============

UFTP (UNICORE FTP) is a file transfer tool similar to Unix' FTP. Its main features include
high-performance file transfers from client to server (and vice versa), list directories,
make/remove files or directories, sync files and data sharing.
In addition, users can easily share their data even with users who do not have
Unix-level access to the data.

The data access server (``uftpd``) is running on ``judacsrv.fz-juelich.de``. Users are authenticated
with a public/private SSH key pair to an authentication server. JSC operates the authentication server

.. code-block:: none

    https://uftp.fz-juelich.de/UFTP_Auth

for the purpose of UFTP data transfers from/to |SYSTEM_NAME|.

.. note::

    The UFTP SSH keys are independent from the keys used for SSH access to the systems.
    Users must maintain separate key pairs and follow the guidelines below concerning the
    UFTP key management. Failure to comply may result in account blockage as a preventive security measure.

Client setup
------------

**Installing the uftp client on your workstation**

The ``uftp`` client is a Java application (requires Java 11 or later) that you can install on your workstation machine
(Linux/Mac OS/Windows) to be able to transfer data between your workstation and JSC.

The latest version is available for download on `GitHub`_. The detailed documentation of the client is
available on the `UFTP manual page`_.

**Using the uftp client on JSC machines**

On JUWELS, JURECA and JUSUF the client is already installed and can be loaded with:

.. code-block:: none

    $ module load uftp

On |SYSTEM_NAME| the client is pre-installed.

You can use it to transfer data from JSC to other HPC sites offering a UFTP server (e.g. `HLRS`_, `LRZ`_, CSCS, ...).

**Alternative client: pyuftp**

As an alternative to the Java-based client, there is a Python client ``pyuftp`` available,
which can be installed with e.g. 

.. code-block:: none

    $ pip install pyuftp

Please have a look at the information availble on the `PyUFTP page`_.

Client-side firewall settings
-----------------------------
In some cases there is a client-side firewall that limits outgoing connections, which may preventive
UFTP from working. If this is the case, the firewall needs to be set to allow

    1. outgoing https traffic to uftp.fz-juelich.de, port 443

    2. outgoing TCP connections to judacsrv.fz-juelich.de (all ports!)

Key management
--------------

The UFTP installation at JSC uses public/private key pairs for authentication and authorization,
these keys can be generated by the standard SSH tools.

The *private key* resides on the workstation where the ``uftp`` client is running, the *public key*
must be on |SYSTEM_NAME| in the ``$HOME/.uftp/authorized_keys`` file.

.. note::

    For security reasons it is a critical requirement that the utilized SSH keys are not used for any other purpose than UFTP and are in particular
    not used to enable SSH-based access (including SCP and SFTP) to any system.

A UFTP public/private key pair can be generated using ``ssh-keygen`` with the ``-f id_uftp`` argument. Please see :doc:`here <access>` for information about key generation.

.. note::

    We suggest to protect the UFTP private key with a passphrase. If an automated data transfer is required, the key
    may be generated without passphrase as long as the above mentioned security measures are respected. In this case,
    we suggest replacing the key on regular basis. Please note that theft of the private key may allow a malicious
    attacker to access, modify and/or destroy your data. JSC reserves the right to change this policy at any time
    in case of a change to the threat assessment.

On your workstation where the ``uftp`` client will be running, generate a key pair:

.. code-block:: none

    $ cd $HOME
    $ mkdir -p .uftp
    $ cd .uftp
    $ ssh-keygen [see above] -f ./id_uftp

To authorize the key for use with UFTP from/to |SYSTEM_NAME|, copy the public key into the correct file on |SYSTEM_NAME|:

.. code-block:: none

    $ ssh user@judac.fz-juelich.de mkdir -p .uftp
    $ scp id_uftp.pub user@judac.fz-juelich.de:.uftp/authorized_keys

Note that similar to SSH, the ``.uftp/authorized_keys`` can contain multiple public keys.

Usage and examples
------------------

The following environment variables should be defined for convenience

.. code-block:: none

    $ export UFTP_USER=<your_remote_user_id>
    $ export UFTP_AUTH_URL=https://uftp.fz-juelich.de/UFTP_Auth/rest/auth/JUDAC:
    $ export UFTP_KEY=$HOME/.uftp/id_uftp

The ``UFTP_USER`` refers to your *remote user ID at JSC*.
You can specify it also on the command line with the ``-u username`` option

**Retrieve information about the remote server**

.. code-block:: none

    $ uftp info --user $UFTP_USER --identity $UFTP_KEY $UFTP_AUTH_URL

**List contents of a remote directory**

.. code-block:: none

    $ uftp ls --user $UFTP_USER --identity $UFTP_KEY $UFTP_AUTH_URL/p/home/jusers/$UFTP_USER/jureca

**Download a single file to the current directory**

.. code-block:: none

    $ uftp cp --user $UFTP_USER --identity $UFTP_KEY $UFTP_AUTH_URL/p/home/jusers/$UFTP_USER/jureca/test .

**Download multiple files using wildcards**

.. code-block:: none

    $ uftp cp --user $UFTP_USER --identity $UFTP_KEY $UFTP_AUTH_URL/p/home/jusers/$UFTP_USER/jureca/testdir/* .

**Uploading files using wildcards**

.. code-block:: none

    $ uftp cp --user $UFTP_USER --identity $UFTP_KEY "/tmp/test/*" $UFTP_AUTH_URL/p/home/jusers/$UFTP_USER/jureca

**Resuming file transfer(s)**

If a long-running transfer got interrupted, you can try to resume it by adding the "-R" option:

.. code-block:: none

    $ uftp cp -R --user $UFTP_USER --identity $UFTP_KEY $UFTP_AUTH_URL/p/home/jusers/$UFTP_USER/jureca/testdir/* .

(be careful with this, especially in case of multithreading / file splitting)

Data sharing
------------

By default, files will be shared for "anonymous" access. This will allow anyone who knows the sharing link to access the file using common HTTP tools. Shares can be limited to certain users.

First we need to set the following environment variable. On JURECA and JUWELS it is already set when loading the uftp module.

.. code-block:: none

    $ export UFTP_SHARE_URL=https://uftp.fz-juelich.de/UFTP_Auth/rest/share/JUDAC

List shares with

.. code-block:: none

    $ uftp share --user $UFTP_USER --identity $UFTP_KEY --list

To share a file with anybody

.. code-block:: none

    $ uftp share --user $UFTP_USER --identity $UFTP_KEY /p/home/jusers/$UFTP_USER/jureca/test

This will print the shared link on the screen. You can use ``curl`` or ``wget`` to download it. To restrict the access to a specific user use the ``--access`` option. For example,

.. code-block:: none

    $ uftp share --user $UFTP_USER --identity $UFTP_KEY --access "CN=schuller1, OU=ssh-local-users" /p/home/jusers/$UFTP_USER/jureca/test

Note that the ``CN=...`` part contains the remote user ID of the target user and ``OU=ssh-local-users`` is the same for all users.

.. external links

.. _GitHub: https://github.com/UNICORE-EU/uftp-javaclient/releases/
.. _UFTP manual page: https://uftp-docs.readthedocs.io/en/latest/user-docs/uftp-client/index.html
.. _PyUFTP page: https://pypi.org/project/pyuftp/
.. _HLRS: https://kb.hlrs.de/platforms/index.php/Data_Transfer_with_UFTP
.. _LRZ: https://doku.lrz.de/display/PUBLIC/Data+Transfer+Options+on+SuperMUC-NG