I/O of key-value pairs to SIONlib files. More...

Collaboration diagram for Key-Value API:

Data Structures
struct	_sion_key_stat_struct

Typedefs
typedef struct _sion_key_stat_struct	sion_key_stat_t

Functions
size_t	sion_fwrite_key (const void *data, uint64_t key, size_t size, size_t nitems, int sid)
	Writes data for key. More...

size_t	sion_fread_key (void *data, uint64_t key, size_t size, size_t nitems, int sid)
	Read data for key. More...

int	sion_fread_key_iterator_reset (int sid)
	Resets key iterator. More...

int	sion_fread_key_iterator_next (int sid, uint64_t key, size_t size)
	Forward to next key. More...

int	sion_seek_key (int sid, uint64_t key, int entrynum, int64_t posinentry)
	Seek to position in key. More...

int	sion_key_full_scan (int sid)
	Performs a full scan of all meta data in current file. More...

int	sion_key_list_iterator_reset (int sid)
	Resets key iterator. More...

int	sion_key_list_iterator_next (int sid, uint64_t *keyptr)
	Forward to next key. More...

int	sion_key_get_stat (int sid, uint64_t key, sion_key_stat_t *stat)
	get statistics about key More...

int	sion_get_keyval_mode (int sid)
	Return selected mode for key value. More...

char *	sion_keyval_type_to_str (int type)
	Returns key value mode as string. More...

Detailed Description

I/O of key-value pairs to SIONlib files.

Introduction

The use of hybrid applications (MPI+OpenMP) suggests a new structural layer to be available in SIONlib. Key-value pairs allow convenient saving and especially loading of multiple pieces of data within one single rank.

Implementations

In general the keys are of data type uint64_t, while the data is of data type void *. All Implementations need to store the amount of data actually written for the specific key. During reads the current position inside a key's data needs to be stored.

In order to prevent additional global communication keys are local for each.

Enabling Key Value Mode

Enabling a SIONlib file for usage with the key-value is done by passing the keyword keyval to the open function with the value choosing the method, e.g. keyval=inline for the inline implementation.

Opening a SIONlib file for read mode is possible using the keyval mode "keyval=unknown". Subsequent calls of sion_get_keyval_mode(int sid, int foo)

int sion_get_keyval_mode(int sid);

return the mode which can be compared to constants like SION_KEYVAL_INLINE for inline mode.

Using an open call without keyval argument fails on a file which is written using keyval.

General Interface

The general interface is

size_t sion_fwrite_key (void* data, uint64_t key, size_t size, size_t nitems, int sid);

size_t sion_fread_key(void* data, uint64_t key, size_t size, size_t nitems, int sid);

Key Information Management

In order to prevent a full search through all previous data for every key read from a SIONlib file and to keep track of the remaining bytes for a key, some caching and management is needed. Every matching or non-matching key which is read from file is added to the "key manager" which hold data of

the key itself
the offset of the key
the length of corresponding data
the number of bytes left unread from the key

Besides managing the pieces of information it also offers an interface for iterating over all available keys which is especially useful for dumping data of all keys.

The key manager saves information about keys in the "key value table". This hash table consists of two columns of types

uint64_t
void *

which hold the key and the according bucket, respectively. The buckets use linked lists in order to store the actual data.

The key manager resides inside the corresponding SIONlib file descriptor and is pointed by siondesc->keyvalptr.

Inline

Implementation Details

From the output point of view the most straight forward implementation is writing the (key, length, data) inside a chunk.

+---------------------+---------------------+---------------------+
| k_0 | l_0 | d_0 ... | k_1 | l_1 | d_1 ... | k_2 | l_2 | d_2 ... |
+-----------------------------------------------------------------+ ...
|                             c_0                                 |
+-----------------------------------------------------------------+

symbol	description
k_i	key i
d_i	data associated to k_i
l_i	length of d_i
c_i	chunk i

Advantages

Simple output implementation
Additional data structures are contained inside the existing ones

Disadvantages

Yields a performance penalty if keys are small and read / written consecutively
Since the key information is directly attached to the corresponding data, this format efficiently yields larger data sizes, which is not obvious for the user and might result in large unused disk space
Key manager is needed and might grow significantly large if many keys are used

Dense (not implemented yet)

Implementation Details

For consecutive read and write operations separating the data and the key information is preferable. Caching key information makes it possible to write a compact block of information filling up the gaps that are left during output of the associated data.

The chunks are not shown in proportional size.

+-----+-----------------------------------------------+
| o_0 | d_0   d_1   d_2 | (k_0 l_0 o_1) (k_1 l_1 o_2) |
+-----------------------------------------------------+...
      |                      c_0                      |
      +-----------------------------------------------+
   +-----------+-------------------------------------------+
   | d_3   d_4 | (k_2 l_2 o_3) (k_3 l_3 o_4) (k_4 l_4 o_5) |
...+-----------+-------------------------------------------+...
   |    c_1    |                    c_2                    |
   +-----------+-------------------------------------------+

symbol	description
k_i	key i
o_i	offset of key i
d_i	data associated to k_i
l_i	length of d_i
c_i	chunk i

The rules for writing the key information are

data never follows key information
key information is written at the first possible position

Advantages

Allows efficient read / write of consecutive keys.

Disadvantages

Needs key manager
Needs more bookkeeping than inline implementation

Function Documentation

◆ sion_fread_key()

size_t sion_fread_key	(	void *	data,
		uint64_t	key,
		size_t	size,
		size_t	nitems,
		int	sid
	)

Read data for key.

Reads data from sion file associated to key. This function uses the diffenent implementations of which one is chosen with the open call. SIONlib will maintain a file position per key so that subsequent reads are starting at last position of current record.

Parameters

[out]	data	buffer to be read into
[in]	key	key associated to the data
[in]	size	size of single item
[in]	nitems	number of items
[in]	sid	sion file handle

Returns: number of successfully read items

Definition at line 179 of file sion_keyvalue.c.

◆ sion_fread_key_iterator_next()

int sion_fread_key_iterator_next	(	int	sid,
		uint64_t *	key,
		size_t *	size
	)

Forward to next key.

Get next record for current task, data itself has to be read with sion_fread_key; this function will forward to next record although if data of last record is not or only read by part

Parameters

[in]	sid	sion file handle
[out]	key	filled with next key
[out]	size	filled with next size

Returns: SION_SUCCESS on success

Definition at line 277 of file sion_keyvalue.c.

◆ sion_fread_key_iterator_reset()

int sion_fread_key_iterator_reset ( int sid )

Resets key iterator.

Parameters

[in] sid sion file handle

Returns: SION_SUCCESS on success

Definition at line 250 of file sion_keyvalue.c.

◆ sion_fwrite_key()

size_t sion_fwrite_key	(	const void *	data,
		uint64_t	key,
		size_t	size,
		size_t	nitems,
		int	sid
	)

Writes data for key.

Writes data to sion file associating it to key. This function uses the diffenent implementations of which one is chosen with the open call.

Parameters

[in]	data	data to be written
[in]	key	key associated to the data
[in]	size	size of single item
[in]	nitems	number of items
[in]	sid	sion file handle

Returns: number of successfully written items

Definition at line 137 of file sion_keyvalue.c.

◆ sion_get_keyval_mode()

int sion_get_keyval_mode ( int sid )

Return selected mode for key value.

Parameters

[in] sid sion file handle

Returns: keyval mode, e.g. SION_KEYVAL_INLINE

Definition at line 339 of file sion_keyvalue.c.

References SION_KEYVAL_UNKNOWN.

◆ sion_key_full_scan()

int sion_key_full_scan ( int sid )

Performs a full scan of all meta data in current file.

Parameters

[in] sid sion file handle

Returns: SION_SUCCESS on success

Definition at line 222 of file sion_keyvalue.c.

◆ sion_key_get_stat()

int sion_key_get_stat	(	int	sid,
		uint64_t	key,
		sion_key_stat_t *	stat
	)

get statistics about key

Parameters

[in]	sid	sion file handle
[in]	key	key
[out]	stat	data structure which will be filled

Returns: SION_SUCCESS on success

Definition at line 432 of file sion_keyvalue.c.

◆ sion_key_list_iterator_next()

int sion_key_list_iterator_next	(	int	sid,
		uint64_t *	keyptr
	)

Forward to next key.

Parameters

[in]	sid	sion file handle
[out]	keyptr	filled with next key

Returns: SION_SUCCESS on success

Definition at line 405 of file sion_keyvalue.c.

◆ sion_key_list_iterator_reset()

int sion_key_list_iterator_reset ( int sid )

Resets key iterator.

Parameters

[in] sid sion file handle

Returns: SION_SUCCESS on success

Definition at line 378 of file sion_keyvalue.c.

◆ sion_keyval_type_to_str()

char* sion_keyval_type_to_str ( int type )

Returns key value mode as string.

Parameters

[in] type mode to be converted

Returns: string representation of mode

Definition at line 364 of file sion_keyvalue.c.

References SION_KEYVAL_HASH, SION_KEYVAL_INLINE, SION_KEYVAL_META, SION_KEYVAL_NONE, and SION_KEYVAL_UNKNOWN.

◆ sion_seek_key()

int sion_seek_key	(	int	sid,
		uint64_t	key,
		int	entrynum,
		int64_t	posinentry
	)

Seek to position in key.

This function searches for a key and on success the file position will be move to the searched key and position, so that the data can be read with sion_fread_key.

Parameters

[in]	sid	sion file handle
[in]	key	key to seek in
[in]	entrynum	nth occurrence of key or SION_ABSOLUTE_POS for absolute position
[in]	posinentry	byte-position in entry, or absolute byte-position in all entries in write-order (SION_ABSOLUTE_POS)

Returns: SION_SUCCESS on success

Definition at line 304 of file sion_keyvalue.c.

Data Structures

Typedefs

Functions

Detailed Description

Introduction

Implementations

Enabling Key Value Mode

General Interface

Key Information Management

Inline

Implementation Details

Advantages

Disadvantages

Dense (not implemented yet)

Implementation Details

Advantages

Disadvantages

Function Documentation

◆ sion_fread_key()

◆ sion_fread_key_iterator_next()

◆ sion_fread_key_iterator_reset()

◆ sion_fwrite_key()

◆ sion_get_keyval_mode()

◆ sion_key_full_scan()

◆ sion_key_get_stat()

◆ sion_key_list_iterator_next()

◆ sion_key_list_iterator_reset()

◆ sion_keyval_type_to_str()

◆ sion_seek_key()