NAME¶

chirp_stream_files - move data to/from chirp servers in parallel

SYNOPSIS¶

chirp_stream_files [options] <copy|split|join> <localfile> { <hostname[:port]> <remotefile>

DESCRIPTION¶

chirp_stream_files is a tool for moving data from one machine to and from many machines, with the option to split or join the file along the way. It is useful for constructing scatter-gather types of applications on top of Chirp.

chirp_stream_files copy duplicates a single file to multiple hosts. The <localfile> argument names a file on the local filesystem. The command will then open a connection to the following list of hosts, and stream the file to all simultaneously.

chirp_stream_files split divides an ASCII file up among multiple hosts. The first line of <localfile> is sent to the first host, the second line to the second, and so on, round-robin.

chirp_stream_files join collects multiple remote files into one. The argument <localfile> is opened for writing, and the remote files for reading. The remote files are read line-by-line and assembled round-robin into the local file.

In all cases, files are accessed in a streaming manner, making this particularly efficient for processing large files. A local file name of - indicates standard input or standard output, so that the command can be used in a pipeline.

OPTIONS¶

 -a,--auth=<flag>

Enable authentication mode: unix, hostname, address, ticket, kerberos, or globus.

 -b,--block-size=<size>

Set transfer buffer size. (default is 1048576 bytes)

 -d,--debug=<flag>

Enable debugging for this subsystem.

 -i,--tickes=<files>

Comma-delimited list of tickets to use for authentication.

 -t,--timeout=<time>

Timeout for failure. (default is 3600s)

 -v,--version

Show program version.

 -h,--help

Show help text.

ENVIRONMENT VARIABLES¶

List any environment variables used or set in this section.

EXIT STATUS¶

On success, returns zero. On failure, returns non-zero.

EXAMPLES¶

To copy the file mydata to three locations:

% chirp_stream_files copy mydata server1.somewhere.edu /mydata


                                 server2.somewhere.edu /mydata


                                 server2.somewhere.edu /mydata

To split the file mydata into subsets at three locations:

% chirp_stream_files split mydata server1.somewhere.edu /part1


                                  server2.somewhere.edu /part2


                                  server2.somewhere.edu /part3

To join three remote files back into one called newdata:

% chirp_stream_files join newdata server1.somewhere.edu /part1


                                  server2.somewhere.edu /part2


                                  server2.somewhere.edu /part3

COPYRIGHT¶

The Cooperative Computing Tools are Copyright (C) 2022 The University of Notre Dame. This software is distributed under the GNU General Public License. See the file COPYING for details.

SEE ALSO¶

• Cooperative Computing Tools Documentation
• Chirp User Manual
• chirp(1) chirp_status(1) chirp_fuse(1) chirp_get(1) chirp_put(1) chirp_stream_files(1) chirp_distribute(1) chirp_benchmark(1) chirp_server(1)