 |
SIONlib
1.7.4
Scalable I/O library for parallel access to task-local files
|
|
SIONlib
1.7.4
Scalable I/O library for parallel access to task-local files
|
This is a general API description which gives a good overview over SIONlib's most commonly used functions. For a description of the basic description of the basic strategies see the file format description. For a more in-depth introduction please see the SIONlib tutorial slides.
This function opens a sion file for writing or reading data. The chunksizes are communicated globally which enables each task to calculate the correct offset to prevent different ranks from trying to access the same parts of the file. Hence this call is collective and needs to be called by all processes at the same time. With this information globally available following calls of sion_fwrite and sion_fread do not need any communication.
In write-mode SIONlib will create one or more physical files on disk containing data of all tasks (see also SIONlib file format). The number has to be defined by the parameters numfiles or lComm. If numfiles is greater than zero the tasks in gComm will be split to numfiles groups. The new communicator will be returned afterwards by parameter lcomm. If numfiles is less than zero, the communicator lcomm has to be a local communicator. Tasks which are in the same local communicator will write to the same physical file. All tasks in gComm have to be a member of one of the local communicators also.
On each task the function opens one of the physical files with the POSIX or ANSI C open command mode and sets the file pointer to a task-specific position.
Important: all parameters of the sion_open function are always pointers. The reason for this is that most of the parameter are input values if the file is opened for writing and are output parameters if the file is opened for reading.
For a list and description of the arguments please see sion_paropen_mpi().
This function closes the sion file in parallel on all tasks involved in opening the sion file. Therefore the local communicator, defined at sion_paropen_mpi, will be used internally to collect all the meta data from each task and to append it to the sion file on task zero of the local communicator. This function has to be called collectively on all tasks of the global communicator given at opening time.
There is currently no fault tolerance in the meta-data handling. The variable size meta-data (meta data block 2, see SIONlib file format) is held in memory on each task and is not written to disk before calling this function. This means, that the file cannot be read again if it was not closed correctly, e.g. because the disk space limit was reached before. This is a trade-off for the possibility so avoid any kind of communication between the open and the close calls.
For a list and description of the arguments please see sion_parclose_mpi().
This function ensures that there is enough space available for writing the requested number of bytes to the sion file. sion_ensure_free_space allocates a new block at the end of the sion file of size chunksize if there is not enough space to write the requested number of bytes in the current block.
This procedure is only be needed if the caller function from the user program do not count the number of bytes written. It is a task local function, which can be called independently from other MPI tasks. In some cases the function moves the file pointer to a new position and also flushes the local file pointer.
For a list and description of the arguments please see sion_ensure_free_space().
This function indicates whether the end of file is reached for this task or not. This means the file pointer points behind the last byte of last chunk of this task. If all bytes of the current chunk are already read and there are more chunks available for this task, sion_feof will advance the file pointer to the start position of the next chunk in the sion file. The function is a task local function, which can be called independently from other MPI tasks and does not invoke any communication.
For a list and description of the arguments please see sion_feof().
This function returns the number of bytes currently available in the current chunk (bytes not read). This function is a local function, which can be called independently from other MPI tasks.
For a list and description of the arguments please see sion_bytes_avail_in_block().
This function opens a sion file for a specific rank. It can be used to open the sion file independently from each task (e.g. if no MPI is available or only a subset of tasks chunks are needed).
Using this function the meta data at beginning of the sion file is read by each task instead of just once and distribute (sion_open). sion_open_rank only reads the task specific meta data of the meta data block at the end of the sion file. (For more information on the file format see SIONlib file format).
All metadata will be read and stored in internal data structures.
Warning: Only read operations are currently supported.
For a list and description of the arguments please see sion_open_rank().
This function opens all tasks of a sion file in serial mode. The meta-data of all tasks will be loaded into memory, which allows accessing different parts of the file by using sion_seek_fp(), for example to switch between ranks. Default position of the file pointer is in front of the first byte of the first task.
In the case of multi-file sion files sion_open() internally opens each physical file with sion_open recursively. For each physical file sion_open manages an individual sion file descriptor and data structure.
For a list and description of the arguments please see sion_open().
This function closes a sion file which was opened in serial mode with sion_open() or sion_open_rank(). In write mode this function will also save all meta data to the meta data block of the sion file at the beginning and the end of the file.
The function is a task local function, which can be called independently from other MPI tasks.
For a list and description of the arguments please see sion_close().
Returns pointers to internal fields, containing the number of chunks written by each task (sion_chunkcount) and their sizes (sion_chunksizes). This function is only needed if the sion file was opened for reading in serial mode.
For a list and description of the arguments please see sion_get_locations().
This function sets the file pointer to a new position according to the specified parameters. It can only be used when the sion file was opened for reading in serial mode. In Write mode currently only the rank can be modified! SION_CURRENT_BLK and SION_CURRENT_POS are required.
For a list and description of the arguments please see sion_seek().
Like sion_seek but exporting the file pointer. This is a legacy function for compatibility. For new code use sion_seek instead.
For a list and description of the arguments please see sion_seek_fp().
This is the basic function for writing data to the underlying storage.
Internally it calls sion_ensure_free_space to ensure that there is enough space left in the current chunk to write the requested number of bytes.
For a list and description of the arguments please see sion_fwrite().
This is the basic function for reading data from the underlying storage.
Internally it potentially executes multiple reads until either the requested number of elements or all available data is read from the file. This makes this function useful for reading more than a single chunk at a time without the need to loop over the chunks on the user side.
For a list and description of the arguments please see sion_fread().
file_mode argument The file mode is a string that contains a comma-separated list of key value pairs, where for some keys the values are optional. Beyond the mandatory information, whether the file should be opened in write "w" or read "r" mode, there are additional options that can be passed with the file mode string.
"w", "bw" or "wb": open file in write mode"r", "br" or "rb": open file in read mode"ansi" or "posix": use ANSI C or POSIX interface for the file access. ANSI C offers internal buffering and is the default."collective": use collective operations"collectivemerge" or "cmerge": use collective operations with collective merge mode"keyval[=MODE]": open file in key value mode. The different values for MODE are described on the key value mode page. The default is default."endianness=MODE": force the file to be opened with endianness MODE. Possible values are big and little. The current system's endianness is used by default.Example:
file_mode = "w,posix" opens the file for writing in POSIX modefile_mode = "w,keyval=inline" opens the file for writing in key mode using the inline method for storing the additional key value data. (inline currently is the default mode)
1.8.15