SIONlib
1.7.7
Scalable I/O library for parallel access to task-local files
|
The use of hybrid applications (MPI+OpenMP) suggests a new structural layer to be available in SIONlib. Key-value pairs allow convenient saving and especially loading of multiple pieces of data within one single rank.
In general the keys are of data type uint64
, while the data is of data type void *
. All Implementations need to store the amount of data actually written for the specific key. During reads the current position inside a key's data needs to be stored.
In order to prevent additional global communication keys are local for each.
Enabling a SIONlib file for usage with the key-value is done by passing the keyword keyval to the open function with the value choosing the method, e.g. keyval=inline
for the inline implementation.
Opening a SIONlib file for read mode is possible using the keyval mode "keyval=unknown". Subsequent calls of sion_get_keyval_mode(int sid, int foo)
return the mode which can be compared to constants like SION_KEYVAL_INLINE for inline mode.
Using an open call without keyval argument fails on a file which is written using keyval.
The general interface is
In order to prevent a full search through all previous data for every key read from a SIONlib file and to keep track of the remaining bytes for a key, some caching and management is needed. Every matching or non-matching key which is read from file is added to the "key manager" which hold data of
Besides managing the pieces of information it also offers an interface for iterating over all available keys which is especially useful for dumping data of all keys.
The key manager saves information about keys in the "key value table". This hash table consists of two columns of types
uint64
void *
which hold the key and the according bucket, respectively. The buckets use linked lists in order to store the actual data.
The key manager resides inside the corresponding SIONlib file descriptor and is pointed by siondesc->keyvalptr.
From the output point of view the most straight forward implementation is writing the (key, length, data) inside a chunk.
symbol | description |
---|---|
k_i | key i |
d_i | data associated to k_i |
l_i | length of d_i |
c_i | chunk i |
For consecutive read and write operations separating the data and the key information is preferable. Caching key information makes it possible to write a compact block of information filling up the gaps that are left during output of the associated data.
The chunks are not shown in proportional size.
symbol | description |
---|---|
k_i | key i |
o_i | offset of key i |
d_i | data associated to k_i |
l_i | length of d_i |
c_i | chunk i |
The rules for writing the key information are