NAME¶

dirfile — a filesystem-based database format for time-ordered binary data

DESCRIPTION¶

The dirfile database format is designed to provide a fast, simple format for storing and reading binary time-ordered data. Dirfiles can be read using the GetData Library, which provides a reference implementaiton of these Standards.

The dirfile database is centred around one or more time-ordered data streams (a time stream). Each time stream is written to the filesystem in a separate file, as binary data. The name of these binary files correspond to the time stream's field name. Dirfiles support binary data fields for signed and unsigned integer types of 8 to 64 bits, as well as single and double precision floating-point real or complex data types.

Two time streams may have different constant sampling frequencies and mechanisms exist within the dirfile format to ensure these time streams remain properly sequenced in time.

To do this, the time streams in the dirfile are subdivided into frames. Each frame contains a fixed integer number of samples of each time stream. Two time streams in the same dirfile may have different numbers of samples per frame, but the number of samples per frame of any given time stream is fixed.

When synchronous retrieval of data from more than one time stream is required, position in the dirfile can be specified in frames, which will ensure synchronicity.

The binary files are all located in one ore more filesystem directories, rooted around a central directory, known as the dirfile directory. The dirfile as a whole may be referred to by its dirfile directory path.

Included in the dirfile along with the time streams is the dirfile format specification, which is one or more ASCII text files containing the dirfile database metadata. The primary file is the file called format located in the dirfile directory. This file and any additional files that it names, fully specify the dirfile's metadata. For the syntax of these files, see dirfile-format(5).

Version 3 of the Dirfile Standards introduced the large dirfile extension. This extension added the ability to distribute the dirfile metadata among multiple files (called fragments) in addition to the format file, as well as the ability to house portions of the database in subdirfiles. These subdirfiles may be fully fledged dirfiles in their own right, but may also be contained within a larger, parent dirfile. See dirfile-format(5) for information on specifying these subdirfiles.

In addition to the raw fields on disk, the dirfile format specification may also specify derived fields which are calculated by performing simple element-wise operations on one or more input fields. Derived fields behave identically to raw fields when read via GetData. See dirfile-format(5) for a complete list of derived field types. Dirfiles may also contain both numerical and character string constant scalar fields, also further outlined in dirfile-format(5).

Dirfiles are designed to be written to and read simultaneously. The dirfile specification dictates that one particular raw field (specified either explicitly or implicitly by the dirfile metadata) is to be used as the reference field: all other vector fields are assumed to have at least as many frames as the reference field has, and the size (in frames) of the reference field is used as the size of the dirfile as a whole.

Version 6 of the Dirfile Standards added the ability to encode the binary files on disk. Each fragment may have its own encoding scheme. Most commonly, encodings are used to compress the data files to same space. See dirfile-encoding(5) for information on encoding schemes.

AUTHORS¶

The Dirfile format was created by C. B. Netterfield <netterfield@astro.utoronto.ca>. It is now maintained by D. V. Wiebe <getdata@ketiltrout.net>.

SEE ALSO¶

dirfile-encoding(5), dirfile-format(5)

For an introduction to the GetData Library reference implementation, see gd_open(3),or gd_getdata(3), or visit the GetData Project website: <http://getdata.sourceforge.net/>.