SIONlib: Scalable I/O library for parallel access to task-local files

[go_db] Version (1.1p9) available (see also Release notes)

SIONlib is a small library for writing and reading binary data to/from from
several thousands of processors into one or a small number of physical files.
The library provides global open and close functions for accessing one file for
writing and reading in parallel. Only these function are collective, writing
and reading to/from this file can be done asynchronously. sionlib provides a
sion file handle and also an file pointer (FILE *) which allows to use the
standard C file I/O operations (POSIX). This file pointer points to a unique
position in the file.

SIONlib provides a simplified file handling for a parallel program that
formerly had to read or write binary data in parallel into separate files
(task-local files). After opening the file with sionlib, the I/O can be done
with the standard C-I/O functionality (fwrite, fread) without changing the
existing program. Instead of maintaining thousands of separate files for
storing the data, there is only one large file containing all the data.

SIONlib requires information on the BLOCKSIZE of the underlying filesystem,
i.e., the BLOCKSIZE is the smallest part of a file which can be locked. For
instance, IBM GPFS has blocksizes of one ore two Megabytes.

Given that each processor writes its data to same file, sionlib assigns
different regions (e.g., data blocks) of the file to a processor. Hence, for
each processor, sionlib needs the estimated data size which would be written to
the file (chunk size). sionlib can also handle extensions (additional chunks),
if the end of a data block is reached.

Both, the estimated chunk size and filesystem's blocksize are used to calculate
the individual start position in the file for each processor. While a certain
block of the file is locked only by one processor, efficient parallel writing
and reading is feasible.

Further informations:
Sion API
Sion File Format
Examples
Utilities
Further Documentation
Installation, Debugging, Error Messages
Release Notes & Copyright

Contact: sionlib.jsc@kfa-juelich.de
WWW: http://www.fz-juelich.de/jsc/sionlib
