SIONlib: Scalable I/O library for parallel access to task-local files

[go_db] Version (1.5.5) available (see also Release notes)

Welcome to SIONlib

SIONlib is a scalable I/O library for the parallel access to task-local files.
The library not only supports writing and reading binary data to or from from
several thousands of processors into a single or a small number of physical
files but also provides for global open and close functions to access SIONlib
file in parallel. SIONlib provides different interfaces: parallel access using
MPI, OpenMp, or their combination and sequential access for post-processing
utilities.

SIONlib is a joint project of:
Forschungszentrum Jülich, Jülich Supercomputing Centre
German Research School for Simulation Sciences
Laboratory for Parallel Programming

Besides collective open and close functions, a distinctive feature of SIONlib
is that all writing and reading is done asynchronously using, for instance,
standard C file I/O operations (POSIX). Due to this simplified file handling, I
/O can be done with the standard C-I/O functionality (fwrite, fread) without
changing the existing program. As an additional file management advantage, only
one large file containing all the data needs to be maintained - instead of
thousands of separate files.

In order function effectively, SIONlib requires information on the BLOCKSIZE of
the underlying filesystem, i.e., the BLOCKSIZE is the smallest part of a file
which can be locked. For instance, IBM GPFS has blocksizes of one ore two
Megabytes. Given that each processor writes its data to same file, SIONlib
assigns different regions (e.g., data blocks) of the file to a processor.
Hence, for each processor, SIONlib needs the estimated data size which would be
written to the file (chunk size). SIONlib can also handle extensions
(additional chunks), if the end of a data block is reached. Both, the estimated
chunk size and filesystem's blocksize are used to calculate the individual
start position in the file for each processor. While a certain block of the
file is locked only by one processor, efficient parallel writing and reading is
feasible.

To send us comments, questions, or bug reports, please send us an email.

Further informations:
[go_db] Sion API
[go_db] Sion File Format
[go_db] Examples
[go_db] Utilities
[go_db] Further Documentation and References
[go_db] Download
[go_db] Installation, Debugging, Error Messages
[go_db] Release Notes & Copyright

Contact: sionlib_jsc@fz-juelich.de
WWW: http://www.fz-juelich.de/jsc/sionlib
