.\" This manpage has been automatically generated by docbook2man
.\" from a DocBook document. This tool can be found at:
.\"
.\" Please send any bug reports, improvements, comments, patches,
.\" etc. to Steve Cheng .
.TH "DATAPACKER" "1" "19 May 2023" "John Goerzen" "datapacker Manual"
.SH NAME
datapacker \- Tool to pack files into the minimum number of bins
.SH SYNOPSIS
\fBdatapacker\fR [ \fB-0\fR ] [ \fB-a \fIACTION\fB\fR ] [ \fB-b \fIFORMAT\fB\fR ] [ \fB-d\fR ] [ \fB-p\fR ] [ \fB-S \fISIZE\fB\fR ] \fB-s \fISIZE\fB\fR \fB\fIFILE\fB\fR\fI ...\fR
\fBdatapacker\fR \fB-h\fR | \fB--help\fR
.SH "DESCRIPTION"
.PP
\fBdatapacker\fR is a tool to group files by size. It is
designed to group files such that they fill fixed-size
containers (called "bins") using the minimum number of
containers. This is useful, for instance, if you want to
archive a number of files to CD or DVD, and want to organize
them such that you use the minimum possible number of CDs or
DVDs.
.PP
In many cases, \fBdatapacker\fR executes almost instantaneously.
Of particular note, the hardlink action
(see OPTIONS below) can be used to effectively copy data into
bins without having to actually copy the data at all.
.PP
\fBdatapacker\fR is a tool in the traditional Unix style; it can
be used in pipes and call other tools.
.SH "OPTIONS"
.PP
Here are the command-line options you may set for
\fBdatapacker\fR\&. Please note that \fB-s\fR and at
least one file (see FILE SPECIFICATION below) is mandatory.
.TP
\fB-0\fR
.TP
\fB--null\fR
When reading a list of files from standard input (see
FILE SPECIFICATION below), expect the input to be
separated by NULL (ASCII 0) characters instead of one
per line. Especially useful with \fBfind
-print0\fR\&.
.TP
\fB-a \fIACTION\fB\fR
.TP
\fB--action=\fIACTION\fB\fR
Defines what action to take with the matches. Please
note that, with any action, the output will be sorted by
bin, with bin 1 first. Possible
actions include:
.RS
.TP
\fBprint\fR
Print one human-readable line per
file. Each line contains the bin number (in the
format given by \fB-b\fR), an ASCII tab
character, then the filename.
.TP
\fBprintfull\fR
Print one semi-human-readable line per
bin. Each line contains the bin number, then a list
of filenames to place in that bin,
with an ASCII tab character after the
bin number and between each filename.
.TP
\fBprint0\fR
For each file, output the bin number (according to the
format given by \fB-b\fR), an ASCII
NULL character, the filename, and another ASCII
NULL character. Ideal for use with xargs
-0 -L 2\&.
.TP
\fBexec:\fICOMMAND\fB\fR
For each file, execute the specified COMMAND via
the shell. The program COMMAND will be passed
information on its command line as indicated below.
It is an error if the generated command line for a
given bin is too large for the system.
A nonzero exit code from any COMMAND will cause
\fBdatapacker\fR to terminate. If COMMAND contains
quotes, don't forget to quote the entire command,
as in:
.nf
datapacker '--action=exec:echo "Bin: $1"; shift; ls "$@"'
.fi
The arguments to the given command will be:
.RS
.TP 0.2i
\(bu
argv[0] ($0 in shell) will
be the name of the shell used to invoke the
command -- $SHELL or
/bin/sh\&.
.TP 0.2i
\(bu
argv[1] ($1 in shell) will
be the bin number, formatted according to
\fB-b\fR\&.
.TP 0.2i
\(bu
argv[2] and on ($2 and on
in shell) will be the files to place in that
bin
.RE
.TP
\fBhardlink\fR
For each file, create a hardlink at
\fIbin\fR/\fIfilename\fR
pointing to the original input filename. Creates
the directory \fIbin\fR as
necessary. Alternative locations and formats for
\fIbin\fR can be specified
with \fB-b\fR\&. All bin directories and
all input must reside on the same filesystem.
After you are done processing the results of the
bin, you may safely delete the bins without
deleting original data. Alternatively, you could
leave the bins and delete the original data.
Either approach will be workable.
It is an error to attempt to make a hard link
across filesystems, or to have two input files
with the same filename in different paths.
\fBdatapacker\fR will exit on either of these situations.
See also \fB--deep-links\fR\&.
.TP
\fBsymlink\fR
Like \fBhardlink\fR, but create
symlinks instead. Symlinks can span filesystems,
but you will lose information if you remove the
original (pre-bin) data. Like
\fBhardlink\fR, it is an error to have a
single filename occur in multiple input directories
with this option.
See also \fB--deep-links\fR\&.
.RE
.TP
\fB-b \fIFORMAT\fB\fR
.TP
\fB--binfmt=\fIFORMAT\fB\fR
Defines the output format for the bin name. This format
is given as a %d input to a function
that interprets it as
\fBprintf\fR(3) would.
This can be useful both to define the name and the
location of your bins. When running \fBdatapacker\fR with
certain arguments, the bin format can be taken to be a
directory in which files in that bin are linked. The
default is %03d, which outputs
integers with leading zeros to make all bin names at
least three characters wide.
Other useful variants could include
destdir/%d to put the string
"destdir/" in front of the bin number,
which is rendered without leading zeros.
.TP
\fB-d\fR
.TP
\fB--debug\fR
Enable debug mode. This is here for future expansion
and does not currently have any effect.
.TP
\fB-D\fR
.TP
\fB--deep-links\fR
When used with the symlink or hardlink action, instead
of making all links in a single flat directory under the
bin, mimic the source directory structure under the
bin. Makes most sense when used with
\fB-p\fR, but could also be useful without it
if there are files with the same name in different
source directories.
.TP
\fB--help\fR
Display brief usage information and exit.
.TP
\fB-p\fR
.TP
\fB--preserve-order\fR
Normally, \fBdatapacker\fR uses an efficient algorithm that
tries to rearrange files such that the number of bins
required is minimized. Sometimes you may instead wish
to preserve the ordering of files at the expense of
potentially using more bins. In these cases, you would
want to use this option.
As an example of such a situation: perhaps you have
taken one photo a day for several years. You would like
to archive these photos to CD, but you want them to be
stored in chronological order. You have named the files
such that the names indicate order, so you can pass the
file list to \fBdatapacker\fR using \fB-p\fR to
preserve the ordering in your bins. Thus, bin 1 will
contain the oldest files, bin 2 the second-oldest, and
so on. If \fB-p\fR wasn't used, you might
use fewer CDs, but the photos would be spread out across
all CDs without preserving your chronological order.
.TP
\fB-s \fISIZE\fB\fR
.TP
\fB--size=\fISIZE\fB\fR
Gives the size of each bin in bytes. Suffixes such as
"k", "m", "g", etc. may be used to indicate kilobytes,
megabytes, gigabytes, and so forth. Numbers such as
1.5g are valid, and if needed, will
be rounded to the nearest possible integer value.
The size of the first bin may be overridden with
\fB-S\fR\&.
Here are the sizes of some commonly-used bins. For each
item, I have provided you with both the underlying
recording capacity of the disc and a suggested value for
\fB-s\fR\&. The suggested value for
\fB-s\fR is lower than the underlying
capacity because there is overhead imposed by the
filesystem stored on the disc. You will perhaps find
that the suggested value for \fB-s\fR is
lower than optimal for discs that contain few large
files, and higher than desired for discs that contain
vast amounts of small files.
.RS
.TP 0.2i
\(bu
CD-ROM, 74-minute (standard): 650m / 600m
.TP 0.2i
\(bu
CD-ROM, 80-minute: 703m / 650m
.TP 0.2i
\(bu
CD-ROM, 90-minute: 790m / 740m
.TP 0.2i
\(bu
CD-ROM, 99-minute: 870m / 820m
.TP 0.2i
\(bu
DVD+-R: 4.377g / 4g
.TP 0.2i
\(bu
DVD+R, dual layer: 8.5g / 8g
.RE
.TP
\fB-S\fR
.TP
\fB--size-first\fR
The size of the first bin. If not given, defaults to
the value given with \fB-s\fR\&. This may be
useful if you will be using a mechanism outside
\fBdatapacker\fR to add additional information to the first
bin: perhaps an index of which bin has which file, the
information necessary to make a CD bootable, etc. You
may use the same suffixes as with \fB-s\fR
with this option.
.TP
\fB--sort\fR
Sorts the list of files to process before acting upon
them. When combined with \fB-p\fR, causes
the output to be sorted. This option has no effect save
increasing CPU usage when not combined with
\fB-p\fR\&.
.SS "FILE SPECIFICATION"
.PP
After the options, you must supply one or more files to
consider for packing into bins. Alternatively, instead of
listing files on the command line, you may list a single
hyphen (-), which tells \fBdatapacker\fR to
read the list of files from standard input (stdin).
.PP
\fBdatapacker\fR never recurses into subdirectories. If you
want a recursive search -- finding all files in a given
directory and all its subdirectories -- see the second
example in the EXAMPLES section below. \fBdatapacker\fR is
designed to integrate with
\fBfind\fR(1) in this situation to
let you take advantage of find's built-in powerful recursion
and filtering features.
.PP
When reading files from standard input, it is assumed that
the list contains one distinct filename per line. Seasoned
POSIX veterans will recognize the inherent limitations in
this format. For that reason, when given
\fB-0\fR in conjunction with the single file
-, \fBdatapacker\fR will instead expect, on
standard input, a list of files, each one terminated by an
ASCII NULL character. Such a list can be easily generated
with \fBfind\fR(1) using its
\fB-print0\fR option.
.SH "EXAMPLES"
.TP 0.2i
\(bu
Put all JPEG images in ~/Pictures
into bins (using hardlinks) under the pre-existing directory
~/bins, no more than 600MB per bin:
.nf
datapacker -b ~/bins/%03d -s 600m -a hardlink ~/Pictures/*.jpg
.fi
.TP 0.2i
\(bu
Put all files in ~/Pictures or any
subdirectory thereof into 600MB bins under ~/bins, using
hardlinking. This is a simple example to follow if you
simply want a recursive search of all files.
.nf
find ~/Pictures -type f -print0 | \\
datapacker -0 -b ~/bins/%03d -s 600m -a hardlink -
.fi
.TP 0.2i
\(bu
Find all JPEG images in ~/Pictures or
any subdirectory thereof, put them into bins (using
hardlinks) under the pre-existing directory
~/bins, no more than 600MB per bin:
.nf
find ~/Pictures -name "*.jpg" -print0 | \\
datapacker -0 -b ~/bins/%03d -s 600m -a hardlink -
.fi
.TP 0.2i
\(bu
Find all JPEG images as above, put them in 4GB bins,
but instead of putting them
anywhere, calculate the size of each bin and display it.
.nf
find ~/Pictures -name "*.jpg" -print0 | \\
datapacker -0 -b ~/bins/%03d -s 4g \\
\&'--action=exec:echo -n "$1: "; shift; du -ch "$@" | grep total' \\
-
.fi
This will display output like so:
.nf
/home/jgoerzen/bins/001: 4.0G total
/home/jgoerzen/bins/002: 4.0G total
/home/jgoerzen/bins/003: 4.0G total
/home/jgoerzen/bins/004: 992M total
.fi
Note: the grep pattern in this example
is simple, but will cause unexpected results if any
matching file contains the word "total".
.TP 0.2i
\(bu
Find all JPEG images as above, and generate 600MB ISO
images of them in ~/bins. This will generate the ISO
images directly without ever hardlinking files into
~/bins.
.nf
find ~/Pictures -name "*.jpg" -print0 | \\
datapacker -0 -b ~/bins/%03d.iso -s 4g \\
\&'--action=exec:BIN="$1"; shift; mkisofs -r -J -o "$BIN" "$@"' \\
-
.fi
You could, if you so desired, pipe this result directly
into a DVD-burning application. Or, you could use
growisofs to burn a DVD+R in a single
step.
.SH "ERRORS"
.PP
It is an error if any specified file exceeds the value given
with -s or -S\&.
.PP
It is also an error if any specified files disappear while
\fBdatapacker\fR is running.
.SH "BUGS"
.PP
Reports of bugs should be reported online at the
\fBdatapacker\fR homepage.
Debian users are encouraged to instead use the
Debian
bug-tracking system.
.SH "COPYRIGHT"
.PP
\fBdatapacker\fR, and this manual, are Copyright (C) 2008 John Goerzen.
.PP
All code, documentation, and build scripts are under the following
license unless otherwise noted:
.PP
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
.PP
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
.PP
You should have received a copy of the GNU General Public License
along with this program. If not, see
\&.
.PP
The GNU General Public License is available in the file COPYING in the source
distribution. Debian GNU/Linux users may find this in
/usr/share/common-licenses/GPL-3.
.PP
If the GPL is unacceptable for your uses, please e-mail me; alternative
terms can be negotiated for your project.
.SH "AUTHOR"
.PP
\fBdatapacker\fR, its libraries, documentation, and all included files, except where
noted, was written by John Goerzen and
copyright is held as stated in the COPYRIGHT section.
.PP
\fBdatapacker\fR may be downloaded, and information found, from its
homepage \&.
.SH "SEE ALSO"
.PP
\fBmkisofs\fR(1),
\fBgenisoimage\fR(1)