.\" Copyright (c) 2011-2022 Ganael LAPLANCHE <ganael.laplanche@martymac.org>
.\" All rights reserved.
.\"
.\" Redistribution and use in source and binary forms, with or without
.\" modification, are permitted provided that the following conditions
.\" are met:
.\" 1. Redistributions of source code must retain the above copyright
.\"    notice, this list of conditions and the following disclaimer.
.\" 2. Redistributions in binary form must reproduce the above copyright
.\"    notice, this list of conditions and the following disclaimer in the
.\"    documentation and/or other materials provided with the distribution.
.\"
.\" THIS SOFTWARE IS PROVIDED BY THE AUTHORS AND CONTRIBUTORS ``AS IS'' AND
.\" ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
.\" IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
.\" ARE DISCLAIMED.  IN NO EVENT SHALL THE AUTHORS OR CONTRIBUTORS BE LIABLE
.\" FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
.\" DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
.\" OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
.\" HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
.\" LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
.\" OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
.\" SUCH DAMAGE.
.\"
.Dd November 18, 2011
.Dt FPART 1
.Os
.Sh NAME
.Nm fpart
.Nd Sort and pack files into partitions
.Sh SYNOPSIS
.Nm
.Op Fl h
.Op Fl V
.Fl n Ar num | Fl f Ar files | Fl s Ar size
.Op Fl i Ar infile
.Op Fl a
.Op Fl o Ar outfile
.Op Fl 0
.Op Fl e
.Op Fl v
.Op Fl l
.Op Fl b
.Op Fl y Ar pattern
.Op Fl Y Ar pattern
.Op Fl x Ar pattern
.Op Fl X Ar pattern
.Op Fl z
.Op Fl zz
.Op Fl zzz
.Op Fl Z
.Op Fl d Ar depth
.Op Fl D
.Op Fl E
.Op Fl L
.Op Fl S
.Op Fl w Ar cmd
.Op Fl W Ar cmd
.Op Fl p Ar num
.Op Fl q Ar num
.Op Fl r Ar num
.Op Ar FILE or DIR...
.Sh DESCRIPTION
The
.Nm
utility helps you sort file trees and pack them into bags
(called "partitions").
.Sh GENERAL OPTIONS
.Bl -tag -width indent
.It Fl h
Print help
.It Fl V
Print version
.El
.Sh PARTITION CONTROL
.Bl -tag -width indent
.It Ic -n Ar num
Create exactly
.Ar num
partitions and try to generate partitions with the same size and number of
files.
This option cannot be used in conjunction with
.Fl f ,
.Fl s
or
.Fl L .
.It Ic -f Ar files
Create partitions containing at most
.Ar files
files or directories.
This option can be used in conjunction with
.Fl s
and
.Fl L .
.It Ic -s Ar size
Create partitions with a maximum size of
.Ar size
bytes.
With this option, a special partition 0 may be used to handle files that do not
fit in a regular partition, given the provided
.Ar size
limit.
This option can be used in conjunction with
.Fl f
and
.Fl L .
You can use a human-friendly unit suffix here (k, m, g, t, p).
.El
.Sh INPUT CONTROL
.Bl -tag -width indent
.It Ic -i Ar infile
Read file list from
.Ar infile .
If
.Ar infile
is
.Dq Li "-" ,
then list is read from stdin.
.It Fl a
Input contains arbitrary values; just sort them (do not crawl filesystem).
Input must follow the
.Dq Li "size(blank)path"
scheme.
This option is incompatible with crawling-related options.
.El
.Sh OUTPUT CONTROL
.Bl -tag -width indent
.It Ic -o Ar outfile
Output partitions' contents to
.Ar outfile
template.
Multiple files will be generated given that template.
Each
.Ar outfile
will get partition number as a suffix.
If
.Ar outfile
is
.Dq Li "-" ,
then partitions will be printed to stdout, with partition number used as a
prefix (so you can grep partitions you are interested in, or do whatever you
want).
.It Fl 0
End filenames with a null (\(cq\&\e0\(cq\&) character when using option
.Fl o .
.It Fl e
When adding directories (see
.Sx DIRECTORY HANDLING
), add an ending
.Dq Li "/"
to each directory entry.
.It Fl v
Verbose mode (may be specified more than once).
.El
.Sh FILESYSTEM CRAWLING CONTROL
.Bl -tag -width indent
.It Fl l
Follow symbolic links (default: do not follow).
.It Fl b
Do not cross filesystem boundaries (default: cross).
.It Ic -y Ar pattern
Include files or directories matching
.Ar pattern
only (and discard all other files).
This option may be specified several times.
.Ar Pattern
may be a leaf (file or directory) name or a specific path.
Shell pattern matching characters
.Dq ( Li \&[ ,
.Dq Li \&] ,
.Dq Li * ,
.Dq Li \&? )
may be used.
Include patterns are ignored when computing size of directories.
.It Ic -Y Ar pattern
Same as
.Fl y
but case insensitive.
This option may not be available on your platform (at least
.Fx
and
GNU/Linux support it, Solaris does not).
.It Ic -x Ar pattern
Exclude files or directories matching
.Ar pattern .
This option can be used in conjunction with
.Fl y
and
.Fl Y .
In this case, exclusion is performed after.
This option may be specified several times.
.Ar Pattern
may be a leaf (file or directory) name or a specific path.
Shell pattern matching characters
.Dq ( Li \&[ ,
.Dq Li \&] ,
.Dq Li * ,
.Dq Li \&? )
may be used.
Exclude patterns also apply when computing size of directories.
.It Ic -X Ar pattern
Same as
.Fl x
but case insensitive.
This option may not be available on your platform (at least FreeBSD and
GNU/Linux support it, Solaris does not).
.El
.Sh DIRECTORY HANDLING
.Bl -tag -width indent
.It Fl z
Pack empty directories.
By default, fpart will pack files only (except when using the
.Fl d
or
.Fl D
options).
This option can be useful for tools such as
.Xr rsync 1
to be able to recreate a full file tree when used with fpart (e.g. using
rsync's --files-from option).
See the
.Fl zz
option to also pack un-readable directories.
.It Fl zz
Treat un-readable or erroneous (partly-read) directories as empty, causing them to
be packed anyway.
Partly-read (non-empty) directories can end up being packed while some of their
children have already been packed.
.It Fl zzz
Pack all directories (as empty).
Useful when 3rd party tools need directory entries to update them (e.g. cpio or
tar).
.It Fl Z
Pack un-readable/erroneous directories in dedicated partitions.
This option helps isolating erroneous parts of a filesystem.
Used in conjuction with FPART_PARTERRNO variable, hooks can try to handle or
work around the error.
Requires live mode (option
.Fl L )
and option
.Fl zz (
or
.Fl zzz )
.It Ic -d Ar depth
After a certain
.Ar depth ,
pack directories instead of files (directories themselves will be added to
partitions, instead of their content).
You can force a specific file to be packed anyway by listing it on the command
line explicitly.
.It Fl D
Implies
.Fl z .
Pack leaf directories: if a directory contains files only, it will be packed as
a single entry.
You can force a specific file to be packed anyway by listing it on the command
line explicitly.
.It Fl E
Implies
.Fl D .
Pack directories only (work on a per-directory basis): in that mode, no file
will be packed.
Instead, each directory will be packed as a single entry with a size being the
sum of all top-level files' sizes.
You can force a specific file to be packed anyway by listing it on the command
line explicitly.
.El
.Sh LIVE MODE
.Bl -tag -width indent
.It Fl L
Live mode (default: disabled).
When using this mode, partitions will be generated while crawling filesystem.
This option saves time and memory but will never produce special partition 0
(see options
.Fl s
and
.Fl S
).
As a consequence, it will generate partitions slightly larger than the size
specified with option
.Fl s .
This option can be used in conjunction with options
.Fl f
and
.Fl s ,
but not with option
.Fl n .
.It Fl S
Skip big files (default: disabled).
In live mode, no special partition 0 can be produced and big files are added to
the current partition as they are found while crawling the filesystem.
That can lead to huge partitions.
That option makes fpart skip files bigger than the specified maximum partition
size (option
.Fl s
) and print them to stdout (even when using option
.Fl o
) as belonging to a pseudo-partition S (as
in 'S'kipped).
It allows a consumer to handle them immediately through a separate process (no
fpart hook will be executed for skipped files).
That option can only be used in Live mode (option
.Fl L
), when a maximum partition size has been given (option
.Fl s
).
.It Ic -w Ar cmd
When using live mode, execute
.Ar cmd
when starting a new partition (before having opened next output file, if any).
.Ar cmd
is run in a specific environment that provides several variables describing the
state of the program:
.Ev FPART_HOOKTYPE
("pre-part" or "post-part"),
.Ev FPART_PARTFILENAME
(current partition's output file name),
.Ev FPART_PARTNUMBER
(current partition number),
.Ev FPART_PARTSIZE
(current partition size),
.Ev FPART_PARTNUMFILES
(number of files in current partition),
.Ev FPART_PARTERRNO
(0 if every single partition's entry has been read without error, else last
erroneous entry's errno.
For error detection to work properly, you may need to rebuild fpart using embedded
fts library, depending on the version shipped with your OS),
.Ev FPART_PID
(PID of fpart).
Note that variables may or may not be defined, depending on requested options
and current partition's state when the hook is triggered.
Also, note that hooks are executed in a synchronous way while crawling
filesystem, so 1) avoid executing commands that take a long time to return as it
slows down filesystem crawling and 2) do not presume cwd (PWD) is the one fpart
has been started in, as it is regularly changed to speed up crawling (use
absolute paths within hooks).
.It Ic -W Ar cmd
Same as
.Fl w ,
but executes
.Ar cmd
when finishing a partition (after having closed last output file, if any).
.El
.Sh SIZE HANDLING
.Bl -tag -width indent
.It Ic -p Ar num
Preload each partition with
.Ar num
bytes.
You can use a human-friendly unit suffix here (k, m, g, t, p).
.It Ic -q Ar num
Overload each file size with
.Ar num
bytes.
You can use a human-friendly unit suffix here (k, m, g, t, p).
.It Ic -r Ar num
Round each file size up to next
.Ar num
bytes multiple.
This option can be used in conjunction with overloading, which is done *before*
rounding.
You can use a human-friendly unit suffix here (k, m, g, t, p).
.El
.Sh EXAMPLES
Here are some examples:
.Bl -tag -width indent
.It Li "fpart -n 3 -o var-parts /var"
Produce 3 partitions, with (hopefully) the same size and number of files.
Three files: var-parts.1, var-parts.2 and var-parts.3 are generated as output.
.It Li "fpart -s 4724464025 -o music-parts /path/to/music ./*.mp3"
Produce partitions of 4.4 GB, containing music files from /path/to/music as well
as MP3 files from current directory; with such a partition size, each partition
content will be ready to be burnt to a DVD.
Files music-parts.0 to music-parts.n, are generated as output.
.It Li "find /usr ! -type d | fpart -f 10000 -i - /home | grep '^1 '"
Produce partitions containing 10000 files each by examining /usr first and then
/home and display only partition 1 on stdout.
.It Li "du * | fpart -n 2 -a"
Produce two partitions by using
.Xr du 1
output.
Fpart will not examine the file system but instead use arbitrary values printed
by
.Xr du 1
and sort them.
.El
.Sh SEE ALSO
.Xr du 1 ,
.Xr find 1 ,
.Xr fpsync 1 ,
.Xr grep 1 ,
.Xr rsync 1
.Sh AUTHOR, AVAILABILITY
Fpart has been written by
.An Gana\(:el LAPLANCHE
and is available under the BSD
license on
.Lk http://contribs.martymac.org
.Sh BUGS
No bug known (yet).