Scroll to navigation

GFPREP(1) Gfarm GFPREP(1)

NAME

gfprep - replicate Gfarm files in parallel

SYNOPSIS

gfprep [-nqvdpPBU] [-mx] [-X regexp] [-S source-domainname] [-h source-hostfile] [-D destination-domainname] [-H destination-hostfile] [-j num-of-processes] [-J num-of-processes] [-M total-byte] [-z minimum-byte] [-Z maximum-byte] [-w way-of-scheduling] [-W kilobytes-for-threshold] [-s KB/s-to-simulate] [-F num-for-readahead] [-I sec-to-update] [-N num-of-replicas] path

DESCRIPTION

gfprep creates replicas of Gfarm files in parallel.

When the path specifies a directory, files under the directory will be replicated recursively.

Path can be specified by a Gfarm URL or a path on a mount point of gfarm2fs.

A set of source/destination hosts can be specified by a domain name and/or a hostlist file. When both a domain name and a hostlist file are specified, a set of hosts is determined by both conditions. When a set of source hosts is specified, only files stored on the source hosts are replicated.

Less busy file system nodes will be selected for source or destination. The maximum number of parallel replications/copies per host is the number of CPUs. Please see the manual page of gfhost for information about the number of CPUs.

gfprep also retrieves the directory entries in parallel.

GFPREP OPTIONS

These are options only for gfprep.

-N num-of-replicas

Specifies the required number of file replicas. If there are a sufficient number of replicas in the destination hosts, neither any copying nor any removal will be done.

If this option is not specified, -N 1 is assumed.

-x

Removes extra file replicas when a set of destination nodes has more file replicas than specified by the -N option.

-m

Migrates file replicas from a set of source nodes to destination nodes. When specifying twice like -mm, it leaves the source file replica.

This option requires at least one of -S, -h, -D, and -H options.

This option and the -N option are mutually exclusive.

COMMON OPTIONS

There are several common options with gfprep. (see man gfpcopy)

EXAMPLES

To replicate files under the directory recursively.

$ gfprep -N 3 gfarm:///dir

$ cd /mnt/gfarm2fs
$ gfprep -N 3 dir

To replicate a file or remove surplus replicas of the file.

$ gfprep -N 3 -x gfarm:///dir/file

To migrate replicas from gfsd1.example.com to other nodes.

$ gfprep -m -S gfsd1.example.com gfarm:///

NOTES

When the -L or -m option is not specified, a source node is not always selected within a set of specified source file system nodes.

It is better to execute gfprep command near the metadata server. When you need to execute gfprep command far from the metadata server, increase the parallelism by the -j and -J options.

SEE ALSO

gfrep(1), gfpcopy(1), gfhost(1), gfarm2.conf(5)

14 Jan 2017 Gfarm