# I/O Forwarding {#io_forwarding}

## Introduction

I/O forwarding is a form of Remote Procedure Calling (RPC) in which calls to low-level I/O functions (`open`, `write`, etc.) are relayed over a communication channel to be performed by other processes.
This is similar to collective I/O, but unlike with collective I/O, the functions do not have to be called collectively and the processes executing the low-level functions in general do not run the user application, but rather a dedicated RPC server.
I/O forwarding can be useful
- on heterogeneous architectures, if certain parts of the system have a better connection to the storage system, or
- when the parts of the system running the user application do not have direct access to the storage system.

SIONlib offers I/O forwarding through the associated [SIONfwd package][SIONfwd].
It implements RPC for the relevant low-level functions via MPI as a server executable and a client library that is used in SIONlib.

[SIONfwd]: https://gitlab.version.fz-juelich.de/SIONlib/SIONfwd

## Usage

In order to use I/O forwarding in SIONlib, two changes have to be made, one in your application's use of SIONlib, the other to the way you launch your application.

### In your application

Every time you use SIONlib to open a file (`sion_open`, `sion_paropen_mpi`, etc.) select the appropriate lowlevel API using an options struct.
For example

```
sion_mpi_options_set_lowlevel_api(options, SION_LOWLEVEL_API_SIONFWD);
sion_paropen_mpi("my_file.sion", ..., options)
```

### When launching your application

SIONfwd uses MPI's ports mechanism to dynamically establish connections between the RPC client (your application) and the server.
This requires
- that the server also be launched through the MPI process startup mechanism, but not into the same `MPI_COMM_WORLD` as your application
- that all information necessary to establish connections be communicated from server to client.

Fortunately, SIONfwd comes with the necessary tools to facilitate this.
Below is a bash script that uses the generic `mpiexec` startup mechanism to first launch the RPC server (on multiple processes) and then launch your application.
`mpiexec` can be replaced by a different startup mechanism like `srun` if necessary.

```
#!/usr/bin/env bash

# provide helper function sionfwd-spawn
eval $(sionfwd-server bash-defs)

# spawn the RPC server and export environment variables for establishing connections
sionfwd-spawn mpiexec -n <number of server processes> sionfwd-server

# launch your application
mpiexec -n <number of application processes> <your application executable>

# shut down the RPC server
mpiexec sionfwd-server shutdown

wait
```
