UFTP Service

UFTP is a file transfer tool and has a client/server architecture. Its main features include high-performance file transfers from client to server (and vice versa), list directories, make/remove files or directories, sync files and data sharing. Users can easily share their data even with users who do not have Unix-level access to the data.

The data access server (uftpd) is running on JUDAC. Users are authenticated with a public/private SSH key pair to an authentication server. JSC operates the authentication server

https://uftp.fz-juelich.de:9112/UFTP_Auth

for the purpose of UFTP data transfers from/to JUDAC.

Note

The UFTP SSH keys are independent from the keys used for SSH access to the systems. Users must maintain separate key pairs and follow the guidelines below concerning the UFTP key management. Failure to comply may result in account blockage as a preventive security measure.

Client setup

You can install the uftp client on your desktop machine (Linux/Mac OS) to be able to transfer data from your desktop to supercomputers and vice versa. The latest version is available for download on sourceforge. The detailed documentation of the client is available on the UNICORE manual page.

On JUWELS and JURECA the client is already installed and can be loaded with:

$ module purge && module use $OTHERSTAGES && module load Stages/Devel-2019a uftp

This webpage will be updated as soon as the uftp module is available in the production stage.

Key management

The UFTP installation at JSC uses public/private key pairs for authentication and authorization and leverages the SSH key infrastructure for this purpose.

Note

For security reasons it is a critical requirement that the utilized SSH keys are not used for any other purpose than UFTP and are in particular not used to enable SSH-based access (including SCP and SFTP) to any system.

A UFTP public/private key pair can be generated using ssh-keygen with the -f id_uftp argument. Please see here for information about key generation. Since older UFTP client versions do not support Ed25519, an RSA key may also be used:

$ ssh-keygen -b 4096 -t rs -f id_uftp

Note

We strongly suggest to protect the UFTP private key with a passphrase. If an automated data transfer is required, the key may be generated without passphrase as long as the above mentioned security measures are respected. In this case we suggest regular key exchange. Please note that theft of the private key may allow a malicious attacker to access, modify and/or destroy your data. JSC reserves the right to change this policy at any time in case of a change to the thread assessment.

The UFTP keys may only be stored in the directory ~/.uftp. Storage at any other location may lead to account blockage as a preventive security measure.

$ cd $HOME/../shared
$ mkdir -p .uftp
$ cd .uftp
$ ssh-keygen [see above] -f ./id_uftp

To authorize the key for JUDAC access, execute the following commands on JUDAC

$ mkdir -p $HOME/.uftp/
$ cat $HOME/../shared/.uftp/id_uftp.pub >> $HOME/.uftp/authorized_keys

Usage and examples

The following environment variables should be defined

$ export UFTP_USER=$USER
$ export UFTP_AUTH_URL=https://uftp.fz-juelich.de:9112/UFTP_Auth/rest/auth/|SYSTEM_NAME|
$ export UFTP_KEY=$HOME/../shared/.uftp/id_uftp

Defining UFTP_USER is not strictly necessary. If not specified on the command line with the -u username option, the client will use this environment variable as the username, if that is not defined, the current username ($USER) will be used which is identical to the JuDoor account name.

Retrieve information about the remote server

$ uftp info --identity $UFTP_KEY $UFTP_AUTH_URL

List contents of a remote directory

$ uftp ls --identity $UFTP_KEY $UFTP_AUTH_URL:/p/home/jusers/$USER/jureca

Download a single file to the current directory

$ uftp cp --identiy $UFTP_KEY $UFTP_AUTH_URL:/p/home/jusers/$USER/jureca/test .

Download multiple files using wildcards

$ uftp cp --identiy $UFTP_KEY $UFTP_AUTH_URL:/p/home/jusers/$USER/jureca/testdir/* .

Uploading files using wildcards

$ uftp cp --identiy $UFTP_KEY "/tmp/test/*" $UFTP_URL:/p/home/jusers/$USER/jureca

Data sharing

By default, files will be shared for “anonymous” access. This will allow anyone who knows the sharing link to access the file using common HTTP tools. Shares can be limited to certain users.

First we need to set the following environment variable. On JURECA and JUWELS it is already set when loading the uftp module.

$ export UFTP_SHARE_URL=https://uftp.fz-juelich.de:9112/UFTP_Auth/rest/share/|SYSTEM_NAME|

List shares with

$ uftp share --identity $UFTP_KEY --list

To share a file with anybody

$ uftp share --identity $UFTP_KEY /p/home/jusers/$USER/jureca/test

This will print the shared link on the screen. You can use curl or wget to download it. To restrict the access to a specific user use the --access option. For example,

$ uftp share --identity $UFTP_KEY --access "CN=schuller1, OU=ssh-local-users" /p/home/jusers/$USER/jureca/test

Note that the CN=... part contains the Unix user ID of the target user and OU=ssh-local-users is same for all users.