Scroll to navigation

rozofs.conf(5) User Manuals rozofs.conf(5)

NAME

rozofs.conf - RozoFS common configuration file

DESCRIPTION

rozofs.conf is managed thanks to libconfig <http://www.hyperrealm.com/libconfig/> and thus complies to the libconfig grammar.

Boolean values must be set to either True or False.

String values must enclosed into double quotes.

numa_aware

That boolean enables to take into account the NUMA architecture of the mother board in order to collocate some RozoFS modules on a same node. This may improve the inter process communication performance.

file_distribution_rule

This parameter enables to choose the file distribution rule at file creation. The following rules are defined:

- 0 "size balancing" (default)

The cluster with the more available disk space is chosen.
The SIDs with the more available disk space are chosen within this cluster.
The device with the more available disk space is chosen within a storio.
- 1 "weighted round robin"

The clusters are selected in a round robin manner.
The SIDs with the more available disk space are chosen within this cluster.
The devices are allocated in a round robin manner within a storio.
The clusters and storages are re-ordered in decreasing cluster and storage available disk space order every 12s.
- 2 or 3 "strict round robin"

The clusters are selected in a round robin manner.
The SIDs are chosen in round robin manner.
The devices are allocated in a round robin manner within a storio.
- 4 "read round robin"

The clusters are distributed in a round robin manner.
The storages within the cluster are then chosen in round robin manner.
The devices of the storages within the inverse storages of the file distribution
are allocated in a round robin manner to favor reading on all the devices.
The devices of the storages within the forward and safe storages of the file distribution
are allocated in a size balancing manner to equalize the devices occupancy.

trashed_file_per_run

Maximum number of files that the remove bins thread of a slave export can remove in a run. A 2 second delay occurs between every run. This provides the maximum deletion rate.

deletion_delay

Minimum delay in seconds between the FID unlink in the export and the actual projections removal on the storage nodes.

nb_trash_thread

Number of trash threads that work in parallel (default: 2, maximum: 8)

statfs_period

caching delay in second of the data reported by a statfs(). The default value is 10 seconds. When the delay is 0, there is no caching.

core_file_directory

The directory under which the RozoFS modules store their core files in case a crash occurs.

nb_core_file

Maximum number of RozoFS core file to keep on this node whatever the RozoFS executable. The older core files are kept while the newest ones are removed.

nb_disk_thread

Specifies the number of threads within the STORIO process that can operate the disk read/write.

crc32c_check

Boolean (True or False) indicating if a check (thanks to CRC error detecting code) will be used for detecting accidental changes to raw data.

crc32c_generate

Boolean (True or False) indicating if a checksum must be generated (thanks to CRC error detecting code) for each write request.

crc32c_hw_forced

That boolean (True or False) can be asserted if the information reported by cpuid does not indicate that hardware CRC is supported whilst the current CPU supports it. This might be the case with VirtualBox.

disk_usage_threshold

when set to non zero value, the storios log a warning when a device percent usage is over or equal to this limit.

disk_read_threshold

when set to non zero value, the storios log a warning when the average read delay on a device is over or equal to this limit in micro seconds.

disk_write_threshold

when set to non zero value, the storios log a warning when the average write delay on a device is over or equal to this limit in micro seconds.

fid_recycle

Boolean (True or False). When set to TRUE, RozoFS starts recycling the FID upon file deletion. The threshold for which the FID recycling starts can be also configured within the rozofs.conf file .

trash_high_threshold

When fid_recycle is set to TRUE, that parameter provides the threshold for which the FID recycling starts .

wr_ack_on_inverse

Boolean (True or False). When set to TRUE, RozoFS acknowledges a write request once a count of inverse projections have been successfully written. Otherwise, by default it waits until a count of forward projections are written. By default, the write acknowlegdment anticipation is not set.

allow_disk_spin_down

This boolean has to be set to enable the spinning down of the disks of a storage node. The STORIO monitoring thread periodically checks the file systems mounted on its disks. This boolean prevents the check to be performed in case no modification takes place on the disk (i.e neither write, truncate nor delete). So when no access is done to the disks, the monitoring thread does not either access to the disk, and the disk spinning down can take place.

It is no use to set this boolean when you do not configure a spin down timeout (see hdparm -S).

export_dscp

Value that indicates the DSCP that is associated with TCP connections used to communicate with the Metadata server. By default the value corresponds to Expedited Forwarding (EF) class.

storio_dscp

Value that indicates the DSCP that is associated with TCP connections used to communicate with the Storage server. By default the value corresponds to Assured Forwarding (AF41) class.

export_attr_thread

Boolean (True or False). When set, this flag indicates that the export has its attribute writeback threads activated (default true).

rozofsmount_fuse_reply_thread

Boolean (True or False). When set, this flag indicates that the rozofsmount has its fuse reply threads activated (default true).

client_xattr_cache

Boolean (True or False). When set, this flag indicates that the rozofsmount caches the extended attributes. The timer used is the same as the one used for attributes (default false).

async_setattr

Boolean (True or False). When set, this flag indicates that the rozofsmount operates in asynchronous mode for setattr operations (default false).

export_versioning

Boolean (True or False). When set, this flag indicates that any deleted object (file/directory) are save under @rozofs-del@ pseudo directory of the parent directory (default false).

device_automount

Boolean (True or False). Default value is False. When set, the RozoFS devices are automatically mounted on the right directories provided that they have the destination mark file on them; the destination mark file is a file named storage_c<cluster>_s<storage>_<device> that tells what this RozoFS device is intended for. No mount instruction should be set in the fstab file for these devices.

when device_automount is True, the following configuration parameters become effective.

device_automount_path

This parameter gives the path where the devices have to be mounted. The default value is "/srv/rozofs/storages". That is device <device> of storage <storage> of cluster <cluster> will be mounted on directory <device_automount_path>/storage_c<cluster>_s<storage>/<device>. The root parameter in the storage.conf file is ignored and can even be omitted. All the paths are automatically created when needed.

device_automount_option

This parameter provides an optional mount option that should be supported by the lowlevel file system.

mkdir_ok_instead_of_eexist

When set to True this parameter makes the export respond OK instead of EEXIST on a directory creation while the directory already exist. This enables concurrent processes to all execute mkdir at the same time without getting any error. Be aware that this will prevent some application to make use of mkdir to simulate a lock between nodes.

mknod_ok_instead_of_eexist

When set to True this parameter makes the export respond OK instead of EEXIST on a file creation while the file already exist. This enables concurrent processes to all execute mkdir at the same time without getting any error. Be aware that this will prevent some application to make use of mknod to simulate a lock between nodes..

client_fast_reconnect

When set to True, the rozofsmount updates some of its timers in order to reconnect to the exportd as fast as possible.

disable_sync_attributes

When set to True it disables synchonous write of the attributes.

device_selfhealing_mode

This parameter is a string that can take the following values:

- ""
No automatic device rebuild can take place.
- spareOnly
Failed devices are automatically rebuilt on a spare disk when one is available. device_automount must be True for auto rebuild to take place on spare device. A spare device is an unmounted but formatted ext4/xfs disk with a "rorofs_spare" mark file at its root. A logical storage (cid/sid) can use a given spare device if the "rozofs_spare" mark file content fits with its spare-mark parameter defined in storage.conf.
- resecure
Identical to "spareOnly" while some spare device is available. When no more spare device is available, files of failed devices are automatically resecured on their spare sids.

For self-healing to be active, the export_hosts parameter must be configured too.

when self-healing is set, the following configuration parameters become effective.

device_selfhealing_delay

Gives the delay in minutes that a failure must last before starting an automatic rebuild of a failed device. The lowest possible value is 1.

device_self_healing_process

This parameter sets the number of rebuilding processes that will be launched in parallel in case of an automatic rebuild for self-healing purpose. This is also the default number of rebuilding processes for operator rebuild commands.

export_hosts

This parameter is needed for automatic rebuilds to take place. It gives the list of addresses (or dns names) where exportd daemon is running. When securing the exportd thanks to virtual IP address, only the virtual IP address is declared (i.e "192.168.10.1"). When no virtual IP address is used, the list of "physical" IP addresses of each host securing the exportd must be given (i.e "192.168.10.1/192.168.20.1").

device_selfhealing_read_throughput

This parameter provides the maximum bandwidth in MB/s that a single rebuild process can take.

spare_restore_enable

Set to False to disable spare file restoring. Default value is True, which enables the spare file restoring feature. This feature tries to relocate data saved in spare files to the nominal location. This feature also requires export_hosts parameter to be set. when spare file restoring is active, the following configuration parameters become effective.

spare_restore_loop_delay

This parameter gives in minutes the delay between two spare file restoring attempts.

spare_restore_read_throughput

This parameter is the maximum network/disk bandwidth that the spare file restoring process will take to run.

min_metadata_inodes

Minimum number of inodes that must be available on metadata device to allow a mknod/mkdir operation (default 256K)

min_metadata_MB

Minimum available size in MB that must be available on metadata device to allow a mknod/mkdir operation (default 2048 MB)

export nb_thin_thread

Number of thin provisioning threads that work in parallel. (default:2; maximum:4)

thin_scan_file_per_run

Number of files that the exportd can submit to storages to get the number of blocks (thin provisioning) in a run. A new run occurs every 2 seconds. This requires that the export has been configured with thin provisioning. (default:1000)

thin_lv1_hash_tb_size

Thin provisioning Level 1 hash table size in power of 2 (changing this parameter will take effect on the next export restart)

expthin_guard_delay_sec

Guard delay en seconds between two file scanning of a file that belongs to an export configured for thin provisioning. (default:10; maximum 600)

level2_cache_max_entries_kb

Maximum number of i-node entries that the export level 2 cache can contain (unit is K). (default 512K)

EXAMPLE

# sample RozoFS common configuration file
#
# Colocated when possible
#
numa_aware           = True;
#
# Round robin gices the better performance
#
file_distribution_rule = 2;
#
# Delete a maximum of 32 files every 2 seconds 
#
trashed_file_per_run = 32;
#
# Do not keep more than 1 core file module
#
nb_core_file         = 1;
#
# Need more disk threads
#
nb_disk_threads      = 8;
#
# CRC32 enabling
#
crc32c_check         = True;
crc32c_generate      = True;
crc32c_hw_forced     = True;
device_automount            = True;
#
# Very slow self healing : 2 processes to read 15MB/s each
#
device_selfhealing_mode            = "spareOnly";
device_selfhealing_delay           = 10;
device_self_healing_process        = 2;
device_selfhealing_read_throughput = 15;
export_hosts                       = "192.168.10.1/192.168.10.2";

FILES

/etc/rozofs/rozofs.conf (/usr/local/etc/rozofs/rozofs.conf)

The system wide configuration file.

AUTHOR

Fizians <http://www.fizians.com>

SEE ALSO

rozofs(7)

APRIL 2015 RozoFS