MAKE-TEMPLATE, MT¶

Reads image and FILES, creates `.jigdo' and `.template'. This is the main functionality of jigdo-file.

It is possible to specify both --image=- and --files-from=-. In this case, first the list of files is read from standard input until an empty line is encountered. Everything following it is assumed to be the image data. This can be useful if you use mkisofs(1) or similar programs that can output the complete image on their standard output, because there is no need to store the image on disc temporarily.

If a FILES argument contains the characters `//' (Unix) or `\.\' (Windows), this has special meaning. In the final jigdo file that users will download, each of the parts is referenced in the `[Parts]' section with a URI of the form `Label:some/filename'. (See `FORMAT OF .JIGDO FILES' below for a detailed description.) The `[Servers]' section gives a mapping of labels to servers on the internet, with lines like `Label=http://myserver.org/jigdofiles/'. Using this information, jigdo will create the final download URI for the part, `http://myserver.org/jigdofiles/some/filename'. Specifying `//' (or `\.\') in a file or directory name serves to `cut off' the names at the right directory level. For example, if the Unix path of one of your FILES is `/path/some/filename', you can tell jigdo-file to cut off after the `/path' by passing it the argument `/path//some/filename', or `/path//' if you want the whole directory scanned. The path names need not be absolute; `somedirectory//' is also possible.

--label Label=/path

Specify a name to use as the label name for a path on disc. (Influences the output jigdo file.) If you used `//' in the FILES arguments as described above, jigdo-file will by default pick label names automatically (`A', `B' etc.). With this option, you can give labels more meaningful names. Note that the label name will only be used if one or more FILES begin with `/path//'.

Try to use label names that start with uppercase characters, to disambiguate them clearly from protocol names like `http', `ftp'.

--uri Label=http://some.server.org/

By default, using --label as described above will cause lines of the form `Label=file:/path/' to be written to the `[Servers]' section of the output jigdo file. If you want to override the `file:' URI so that the line reads `Label=http://some.server.org/', you can do so by specifying --uri along with --label. Giving just --uri Label=... without the corresponding --label Label=... has no effect, and even if you specify both, an entry is only added to the `[Servers]' section if the label is referenced by at least one `[Parts]' entry.

The supplied value is not quoted by the program; if it contains characters such as space or any of the characters #"'\ then you must quote it. (Under Unix, you may need to quote the value twice to also protect it from the shell, e.g. \\\\ or '\\' to get a single backslash in the URI.)

The mapping specified with an --uri option is ignored if it is already present in the output jigdo file.

Users of the Windows version may notice that the `\' directory separators are converted into `/' in the `file:' URIs that are generated by default. This is done to increase cross-platform compatibility of `file:' - the print-missing command of the Windows version will automatically re-convert the characters when it prints the URIs. In case you supply your own `file:' URIs under Windows using --uri, you must also exchange `/' and `\'.

-0 to -9

Set amount of compression in the output template file, from -0 (no compression) to -9 (maximum compression). The default is -9, which can make the template generation quite slow. By default, the compression algorithm used is the same as for gzip(1).

--gzip and --bzip2

Choose between the gzip and bzip2 compression algorithms. The default is gzip. Bzip2 usually gives a better compression ratio, but compression is significantly slower than with gzip.

--min-length=BYTES

Set minimum length of a part for jigdo-file to look for it in the image. The default is 1k. Parts smaller than this will never be found in the image, so their data will be included in the template file. The search algorithm used requires such a minimum length, otherwise template generation could become extremely slow. If you know for sure that all your FILES are larger than a certain amount, you can increase jigdo-file's speed slightly by specifying the amount with this option. There is a hard-wired absolute minimum of 256 bytes - anything lower will silently be set to 256.

--merge=FILE

Include the contents of FILE in the output `.jigdo' file. The file can contain data which you want added to the output (for example, a `[Servers]' section with a list of your servers as entries), or it can be the jigdo file output by an earlier run of jigdo-file.

It is possible to specify the same file for input with --merge and for output with --jigdo. However, you will also need to use --force to make the program overwrite the old version of the jigdo file with the new one. FILE can be `-' for standard input.

When adding new information to the supplied file, jigdo-file will not insert new lines into the `[Parts]' section if an entry for the same checksum (but not necessarily with the same URI!) already exists, and it will not insert new lines into the `[Servers]' section if a completely identical entry already exists.

When reading in the existing FILE, the behaviour is slightly different: The program preserves entries in the `[Parts]' section with identical checksum, but different URIs. For completely identical entries (same checksum and URI), only one entry is preserved and the duplicates are removed. The `[Servers]' section is left untouched.

--image-section

This is the default. Causes jigdo-file to add an `[Image]' section to the `.jigdo' file.

As an exception, a new `[Image]' section is not added if you use --merge and the file to merge contains an `[Image]' section with a line which reads `Template-MD5Sum=' (end of line after the `='). In this case, the generated template data's MD5 checksum value is just added after the `=' of the first line of this form in the file - no whole new `[Image]' section is appended. This behaviour is useful because it allows you to pass via --merge an `[Image]' section with arbitrary content and then have the MD5 checksum automatically added by jigdo-file. The section `FORMAT OF .JIGDO FILES' below explains the `[Image]' section contents in greater detail.

--no-image-section

Do not include an `[Image]' section in the `.jigdo' file. You need to add one yourself if you use this option. However, doing that is not easy (you also need to add a `Template-MD5Sum' line with the correct checksum, or jigdo will complain), so use of this option is discouraged.

--servers-section

This is the default. Causes jigdo-file to add a `[Servers]' section to the `.jigdo' file. This default section uses `file:' URIs, which allows for immediate reassembly of the image from the local filesystem, and is also useful if you want to edit the file manually and replace the `file:' URIs with other URIs.

--no-servers-section

Do not add a `[Servers]' section at the end of the `.jigdo' file. Useful e.g. if you are going to append the section with a script.

--match-exec=SHELLCOMMAND

Whenever a file is found in the image, execute the supplied command string by passing it to a shell. jigdo-file sets up a number of environment variables with information about the file match. For example, if the file `/path//a/b/file' was found in the image and `Label:a/b/file' is going to be written to the `.jigdo' file:

Please be careful to correctly quote the string passed to this option, otherwise your supplied command will not work with filenames that contain spaces. As an example, to create a backup of hard links to the matched files, use the following option: --match-exec='mkdir -p "${LABEL:-.}/$MATCHPATH" && ln -f "$FILE" "${LABEL:-.}/$MATCHPATH$LEAF"'

By default, no command is executed. Use --match-exec="" to remove a command string which was set with an earlier use of this option.

--greedy-matching

This is the default. Imagine that your image contains a .tar file which in turn contains another file x, and that you provide both the .tar and the files inside it on the command line. When jigdo-file scans the image, it encounters the beginning of the .tar file, and then the file x.

At this point, a decision must be made: Should the smaller file x be recorded as matched, or should it be ignored in favour of the larger (and thus better) match of the .tar file? Unfortunately, at this point it is not clear whether there will actually be a full match of the .tar, so by default, the program prefers the small match.

--no-greedy-matching

In the case where a large partial match is present and a shorter match has been confirmed, ignore the small match. (See the option above.)

MAKE-IMAGE, MI¶

Reads `.template' and FILES, creates image (or `imagename.tmp'). Provides a rudimentary way of reassembling images - jigdo is usually better suited for this task. However, in contrast to jigdo, no `.jigdo' file is required.

If the image is to be written to a file (and not to standard output), it is possible to create the image in several steps, with several invocations of `jigdo-file make-image', as follows: You first invoke jigdo-file, specifying as many files as are available at this time. The program scans the files, and those that are contained in the image are copied to a temporary file, whose name is formed by appending `.tmp' to the image filename.

For all further files which could be parts of the image, you repeat this process. As soon as all parts are present, the temporary file will be truncated slightly (to delete some administrative data that jigdo-file appends at the end) and renamed to the final image name. The possibility of reassembling the image in several steps is especially useful for gathering files from removable media, e.g. several older CDs.

Scripts using make-image can detect whether image creation is complete by checking the exit status: 0 signals successful creation, whereas 1 means that more files need to be supplied. Other errors result in an exit status of 2 (`recoverable', e.g. file not found) or 3 (non-recoverable, e.g. write error).

--check-files

This is the default. Whenever any part is copied to the image, re-check its checksum against the checksum stored in the template. It is recommended that you leave this switched on, even if it slows down image creation a bit.

--no-check-files

Do not check files' checksums when copying them to the image. This can be safely used when no cache file is used (which means that files will be written to the image immediately after being scanned) or the whole image is checked later with the verify command.

PRINT-MISSING, PM¶

Reads `.jigdo', `.template' and (if present) `imagename.tmp', outputs a list of URIs still needed to completely reassemble the image.

Together with the make-image command, this provides most of the functionality of jigdo on the command line.

For each part that is not yet present in the temporary image file, the file checksum is looked up in the `[Parts]' section of the jigdo file. Any label in the corresponding entry is then expanded according to the label definitions in the `[Servers]' section and printed on standard output. jigdo allows you to specify several alternative locations for each label in this section, but print-missing will only output the first one for each missing part.

If the checksum cannot be found in the `[Parts]' section (this Should Not Happen unless you deleted that section), a lookup is instead made for `MD5Sum:<checksum>' and `SHA256Sum:<checksum>', just like with jigdo. (Thus, if you want to get rid of the `[Parts]' section, you can do so if you rename each part to its own checksum.)

--uri Label=http://some.server.org/

Override the entries in the `.jigdo' file for any label with a URI of your choice. With the example above, a `[Parts]' entry of `Label:some/filename' will cause the line `http://some.server.org/some/filename' to be printed.

The supplied value is not quoted by the program; if it contains characters such as space or any of the characters #"'\ then you must quote it. (Under Unix, you may need to quote the value twice to also protect it from the shell, e.g. \\\\ or '\\' to get a single backslash in the URI.)

PRINT-MISSING-ALL, PMA¶

Just like print-missing, this command outputs a list of URIs still needed to completely reassemble the image. However, all alternative download locations are printed instead of just one. In the output, the URIs for a file are separated from other files' URIs with blank lines. The --uri option has the same effect as for print-missing.

VERIFY, VER¶

Reads image (presumably generated with make-image) and `.template', checks for correct checksum of image.

The template data does not only contain checksums of the individual parts, but also of the image as a whole. make-image already performs a number of internal checks, but if you like, you can additionally check the image with this command.

SCAN, SC¶

Reads all the FILES and enters them into the cache, unless they are already cached. The --cache option must be present for this command.

--no-scan-whole-file

This is the default. This only causes the first --checksum-block-size bytes of each file to be read. If the cache is used later by jigdo-file make-image, the rest of the file will be read once these first bytes are recognized in the input image.

--scan-whole-file

Immediately read the entire file contents and store them in the cache.

MD5SUM, MD5¶

Reads all the FILES and prints out MD5 checksums of their contents. This command is quite similar to md5sum(1), except that the checksum is output in the Base64-like encoding which is also used elsewhere by jigdo-file.

The FILES arguments are processed in the same way as with the other commands, which means that recursion automatically takes place for any arguments that are directories, and that symbolic links are not listed except when the file(s) they point to are not reachable directly.

In the checksum list printed on standard output, only the part of the filename following any `//' (or `\.\' on Windows) is printed. Any --cache will be used for querying files' MD5 checksums and/or writing the checksums of scanned files.

SHA256SUM, SHA256¶

Reads all the FILES and prints out SHA256 checksums of their contents. This command is quite similar to sha256sum(1), except that the checksum is output in the Base64-like encoding which is also used elsewhere by jigdo-file.

The FILES arguments are processed in the same way as with the other commands, which means that recursion automatically takes place for any arguments that are directories, and that symbolic links are not listed except when the file(s) they point to are not reachable directly.

In the checksum list printed on standard output, only the part of the filename following any `//' (or `\.\' on Windows) is printed. Any --cache will be used for querying files' SHA256 checksums and/or writing the checksums of scanned files.

LIST-TEMPLATE, LS¶

Reads a `.template' file and outputs low-level information about the image and all parts contained in it, including offset, length and checksum.

You can also use this command with temporary image files (by specifying something like --template=imagename.tmp) - in that case, the output also distinguishes between parts that have been written to the image and parts that haven't.

The exact output format may change incompatibly between different jigdo releases. The following different types of lines can be output. `have-file' only occurs for `.tmp' files, indicating a file that has already been successfully written to the temporary file:

in-template offset-in-image length
need-file-md5 offset-in-image length file-md5sum filestart-rsyncsum
have-file-sha256 offset-in-image length file-sha256sum filestart-rsyncsum
image-info-sha256 image-length rsyncsum-size image-sha1sum 

JIGDO SECTION¶

[Jigdo]
Version=1.1
Generator=jigdo-file/1.0.0

Information about the version of the jigdo file format used, and the program that generated it. There should be one such section per `.jigdo' file.

IMAGE SECTION¶

[Image]
Filename="filename for saving on user's disc"
Template="URI where to fetch template file"
Template-MD5Sum=OQ8riqT1BuyzsrT9964A7g
Template-SHA256Sum=MVJIxGifflRF9K8ERdbqoyns4Ucw9Xy1ubdnE6CtMbo
ShortInfo=single-line description of the image (200 characters max.)
Info=long description (5000 characters max.)

The value for the `Template' entry can be either an URL (absolute or relative to the URL of the jigdo file) or a string of the form `Label:pathname' (UNIMPLEMENTED), as described below.

The `Template-MD5Sum' and/or `Template-MD5Sum' entries are added by jigdo-file and specify the checksum of the generated `.template' file. It is used by jigdo to detect cases where the downloaded template data is corrupted or belongs to a different image.

Unlike other entry values, the values of the `ShortInfo' and `Info' entries are not split up into words, instead all quoting is preserved.

The value of the `Info' entry is special in that jigdo(1) can optionally parse XML markup it contains. If the markup has errors such as unbalanced/unsupported tags, the string is displayed literally, without XML parsing. Supported tags are <b></b> (bold), <i></i> (italic), <tt></tt> (typewriter font), <u></u> (underline), <big></big> (larger font), <small></small> (smaller font) and <br/> (linebreak). Supported entities include &lt; (`<'), &gt; (`>') and &amp; (`&'). Note that the whole `Info' entry must be on one line in the jigdo file.

This section may occur multiple times, but all except the first one will be ignored. This is useful e.g. when creating a `.jigdo' file for a DVD image when you already have `.jigdo' files for CDs with the same content: You can simply `[Include]' (see below) the CDs' jigdo files at the end of the DVD jigdo file, after its `[Image]' section.

PARTS SECTION¶

[Parts]
xJNkjrq8NYMraeGavUpllw=LabelA:part0
GoTResP2EC6Lb_2wTsqOoQ=LabelA:part1
kyfebwu6clbYqqWUdFIyaw=LabelB:some/path/part2
-J9UAimo0Bqg9c0oOXI1mQ=http://some.where.com/part3

All lines in the section, which provides the mapping from checksums to URIs, have the same format: On the left side of the `=' the checksum (encoded with a Base64-like encoding) is given, and on the right a string corresponding to the part with this checksum; either a complete URI or a string of the form `Label:pathname', which is expanded into one or more URIs by looking up the definition(s) for the Label in the `[Servers]' section.

In case a particular checksum cannot be found in any `[Parts]' section by jigdo, the program will perform a lookup for `MD5Sum:<checksum>' and `SHA256Sum:<checksum>', e.g. for `MD5Sum:xJNkjrq8NYMraeGavUpllw' if you deleted the line for `part0' above.

A checksum appearing multiple times in this section indicates alternative download locations for the part.

There may be any number of `[Parts]' sections in the file; they are all considered when looking up checksums.

jigdo-file always puts the `[Parts]' section at the end of the file, and it even rearranges any file specified with --merge to have only one such section at the end. This is done to allow jigdo to display the information from the `[Image]' section while the rest of that file is still being downloaded.

SERVERS SECTION¶

[Servers]
LabelA=http://myserver.org/
LabelA=ftp://mirror.myserver.org/
LabelB=LabelC:subdirectory/
LabelC=http://some.where.com/jigdo/

All lines in the section, which provides the mapping from server labels to server locations, have the same format: On the left side of the `=' the label name is given, and on the right the value to expand the label name to.

A label name appearing multiple times in this section indicates alternative download locations for the parts that use the label in the `[Parts]' section. This notation makes it very easy to add mirrors to the jigdo file.

As shown by the example above, the label values may themselves reference other labels. In this case, the entry `LabelB:some/path/part2' in the `[Parts]' section will expand to `http://some.where.com/jigdo/subdirectory/some/path/part2'. Loops in the label definitions result in undefined behaviour and must be avoided.

There may be any number of `[Servers]' sections in the file; they are all considered when looking up labels. Either of `[Parts]' or `[Servers]', but not both, can be omitted from the jigdo file.

COMMENT SECTION¶

[Comment]
Any text, except that lines must not begin with `['.

All text following a `[Comment]' or `[comment]' line is ignored, up to the next line with a section label.

INCLUDE DIRECTIVE¶

[Include http://some.url/file.jigdo]

Lines of this form cause the content of the specified jigdo file to be downloaded and parsed just like the main jigdo file. The effect will be the same as copying the included file's contents into the file which contains the include directive. (Exception: Any relative URLs are always resolved using the URL of the `.jigdo' file that contains that relative URL.)

The URL argument can be an absolute or relative URL. Relative URLs are assumed to be relative to the URL of the jigdo file which contains the include directive. Includes can be nested, but it is an error to create a loop of include directives. It is not possible to use URLs of the form `Label:pathname'.

The URL cannot be quoted with "". Any `]' characters in the argument must be escaped as `%5D', and any spaces as `%20'.

Include directives are only supported by jigdo, they are ignored by jigdo-lite.

An include directive terminates any previous section, but it does not start a new one. In other words, a new section must always be started after the include line, jigdo does not allow normal entries to appear below the `[Include]'.

PREPARING YOUR CD IMAGE FOR DISTRIBUTION¶

You have created a CD image `image.iso' from some of the files stored in the directory `/home/ftp' on your harddisc, which is also available online as `ftp://mysite.org'. As you don't want to waste space by effectively hosting the same data twice (once as files on the FTP server, once inside the image), and you are fed up with users' downloads aborting after 200MB and their restarting the download dozens of times, you decide to use jigdo. How do you prepare the image for download?

In fact, only one command is necessary:

People can now point jigdo at `ftp://mysite.org/image.jigdo' to download your image. The template file needs to be accessible as `ftp://mysite.org/image.template'.

Note that nothing prevents you from doing the same for an FTP server that isn't administrated by you - in that case, you only need to host the `.jigdo' and `.template' files on your own server/homepage.

PREPARING AN ARBITRARY LARGE FILE FOR DISTRIBUTION¶

We assume that you have a large file that is not a filesystem, e.g. `movie.mpeg'. Because of space problems, you want to distribute the data on two servers.

In this case, the parts of the image need to be generated artificially with the split command. For example, to create chunks of 4MB each, use `split -b 4m movie.mpeg part'. Copy the resulting files `partXX' into two directories `1' and `2' that you create, according to how you want the files distributed between the servers. Next, create the jigdo and template files with `jigdo-file make-template --image=movie.mpeg 1// 2//'. You will need to edit the `.jigdo' file and provide the right URIs for the two servers that you are going to upload the `partXX' files to.

CUSTOMIZED VERSIONS OF IMAGES¶

Because it is possible to assign a different URI for each part of an image if necessary, jigdo is very flexible. Only one example is the possibility of customized versions of images: Suppose that someone is distributing a CD image, and that you want to make a few small changes to it and redistribute your own version. You download the `official.iso' CD image with jigdo (passing it the URL of `official.jigdo'), write it to CD-R, make your changes (say, adding files from the `myfiles' directory on your harddisc) and produce your own version, `myversion.iso'. Next, you instruct jigdo-file to create the jigdo and template files for your modified image, using the command

Now you can upload the `.jigdo' file, the `.template' file and also the files in `myfiles' to `http://my.homepage.net/'. Thus, for people to download your modified image, you do not need to upload the complete image contents to your web space, but only the changes you made!

(In case you only made very few changes, you could also omit the `myfiles' parameter in the command above, then all your changes end up in the new template file.)

COMBINING MANY JIGDO-MANAGED IMAGES INTO ONE¶

It is also no problem to combine data from several sources that use jigdo. For example, if of five different and unrelated servers each one distributes a different CD image via jigdo, you can create a customized DVD image that contains the data from all these CDs. When people use jigdo to download your image, the individual files on the DVD are fetched from the same sources as the original CDs.

Consequently, even though you will be distributing a 3.2GB file via your web space, the actual amount of data that is stored on your server will only be in the order of several MBs.