Introduction

I/O forwarding is a form of Remote Procedure Calling (RPC) in which calls to low-level I/O functions (open, write, etc.) are relayed over a communication channel to be performed by other processes. This is similar to collective I/O, but unlike with collective I/O, the functions do not have to be called collectively and the processes executing the low-level functions in general do not run the user application, but rather a dedicated RPC server. I/O forwarding can be useful

on heterogeneous architectures, if certain parts of the system have a better connection to the storage system, but are not suitable for running the user application (and thus MSA-aware collective I/O cannot be used), or
when the parts of the system running the user application do not have direct access to the storage system.

SIONlib offers I/O forwarding through the associated SIONfwd package. It implements RPC for the relevant low-level functions via MPI as a server executable and a client library that is used in SIONlib.

Usage

In order to use I/O forwarding in SIONlib, two changes have to be made, one in your application's use of SIONlib, the other to the way you launch your application.

In your application

Every time you use SIONlib to open a file (sion_open, sion_paropen_mpi, etc.) make sure to pass the sionfwd flag. For example

sion_paropen_mpi("my_file.sion", "bw", ...)

becomes

sion_paropen_mpi("my_file.sion", "bw,sionfwd", ...)

When launching your application

SIONfwd uses MPI's ports mechanism to dynamically establish connections between the RPC client (your application) and the server. This requires

that the server also be launched through the MPI process startup mechanism, but not into the same MPI_COMM_WORLD as your application
that all information necessary to establish connections be communicated from server to client.

Fortunately, SIONfwd comes with the necessary tools to facilitate this. Below is a bash script that uses the generic mpiexec startup mechanism to first launch the RPC server (on multiple processes) and then launch your application. mpiexec can be replaced by a different startup mechanism like srun if necessary.

#!/usr/bin/env bash
 
# provide helper function sionfwd-spawn
eval $(sionfwd-server bash-defs)
 
# spawn the RPC server and export environment variables for establishing connections
sionfwd-spawn mpiexec -n <number of server processes> sionfwd-server
 
# launch your application
mpiexec -n <number of application processes> <your application executable>
 
# shut down the RPC server
mpiexec sionfwd-server shutdown
 
wait