NAME¶
htcp, htmv, htrm, htls, htll, htmkdir, htfind, htping - file transfers
and queries via HTTP/HTTPS/SiteCast
SYNOPSIS¶
htcp, htmv [options] Source-URL[s] Destination-URL
htrm, htls, htll, htmkir, htfind [options] Target-URL[s]
htping [options]
DESCRIPTION¶
htcp is a client to fetch files or directory listings from remote servers
using HTTP or HTTPS, or to put or delete files or directories onto remote
servers using HTTPS. htcp is similar to
scp(1), but uses HTTP/HTTPS rather
than ssh as its transfer protocol. htcp can also use the HTCP protocol to
query HTTP(S) fileservers via SiteCast.
When talking to a fileserver with HTTPS, htcp can run "anonymously",
with a standard X.509 user certificate and key, or with a GSI Proxy. This
makes htcp very useful in Grid environments where many users have certificates
and where jobs and users have access to GSI proxies.
URLs¶
htcp supports the file:, http: and https: URL schemes as sources and
destinations. If no scheme is given, the URL scheme is assumed to be file: and
relative to the current directory if not an absolute path.
If multiple sources are given during a copy, they will be used in turn and the
destination must be a directory (directories are indicated by a trailing /)
However, source and destination cannot both refer to remote servers.
OPTIONS¶
- -v/--verbose
- Turn on debugging information. Used once, this option will
enable htcp's messages to stderr. Used twice, will also enable the
underlying libcurl messages.
- --delete
- Instead of copying files, delete all the URLs given on the
command line. Calling the program as htrm has the same effect.
- --list
- Instead of copying files, output lists of files located in
the URL-directories given on the command line. Calling the program as htls
has the same effect.
- --long-list
- Instead of copying files, output long listings of files
located in the URL-directories given on the command line. If available,
the size in bytes and modification time of each file is given. Calling the
program as htll has the same effect.
- --mkdir
- Instead of copying files, attempt to create a directory on
a remote server with HTTP PUT. The server must support the convention that
PUT to a URL with a trailing slash means create a directory. No file body
is sent. Calling the program as htmkdir has the same effect.
- --move
- Move/rename files on a single remote server, given the two,
absolute URLs of the remote file names. Server must support HTTP/WebDAV
MOVE. Calling the program as htmv has the same effect.
- --ping
- Query specified multicast groups with the HTCP NOP
("No Operation") code. SiteCast enabled servers will respond
immediately with a NOP reply, and all of the responses will be listed,
with the round trip time in milliseconds. Any waiting times specified in
the --groups option will be ignored. Calling the program as htping has the
same effect. (--groups must be used for this option to work.)
- --find
- Query specified multicast groups with the HTCP TST code.
SiteCast enabled servers will respond with TST replies if they have the
files corresponding to the given SiteCast target URL(s). All of the
transfer URLs returned will be listed. Waiting times specified in the
--groups option will be used to space out the multicast queries, but the
program listens for responses continuously. Calling the program as htfind
has the same effect. (--groups must be used for this option to work.)
- --groups <IP Groups>
- IP multicast groups to use for SiteCast queries. IP Groups
is a comma separated list of groups, in the format:
nnn.nnn.nnn.nnn:port[:ttl[:seconds]] The IP number and port must be
specified. The IP time-to-live, ttl, controls how many networks the
multicast packets may pass through - the default, 1, limits packets to the
local network. Multiple groups may be specified, separated by commas. If
multiple groups are specified, then seconds is the time to wait before
making the next multicast - 1 second is the default.
- --timeout <seconds>
- A request timeout used for multicast ping.
- --anon
- Do not attempt to use X.509 user certificates or GSI
proxies to authenticate to the remote HTTPS server. This means you are
"anonymous", but the server's identity may still be verified and
the connection is still encrypted.
- --cert <X.509 cert path> and --key <X.509 key
path>
- Path to the PEM-encoded X.509 or GSI Proxy user certificate
and key to use for HTTPS connections, intead of "anonymous
mode." If only one of --key or --cert is given, then that will be
tried for both. If neither is given, then the following order of
precedence is used: the file name held by the variable X509_USER_PROXY;
the file /tmp/x509up_uID (with Unix UID equal to ID); the file names held
by X509_USER_CERT / X509_USER_KEY; the files ~/.globus/usercert.pem and
~/.globus/userkey.pem (where ~/ is the home directory of the user.)
- --capath <X.509 CA root certs directory or file>
- Path to the PEM-encoded CA root certificates to use when
verifying remote servers' host certificates in HTTPS connections. Ideally
this should be a directory of hash.0 files as described in the OpenSSL
verify(1) man page, but a file may be used instead. If --capath is not
given, the value of the environment variable X509_CERT_DIR will be tried.
If this is not valid, then /etc/grid-security/certificates will be used.
- --no-verify
- Do not use CA root certificates to verify remote servers'
host certificates. This is useful for testing sites before their
certificate is set up properly, but leaves you vulnerable to "man in
the middle" attacks by hostile servers masquerading as your target.
- --grid-http
- Try to use GridHTTP redirection for HTTPS URLs. Compatible
servers will perform authentication and authorization on the HTTPS
connection and then redirect to HTTP for the GET or PUT file transfer.
htcp makes the HTTP request using the GRID_AUTH_PASSCODE single-use
passcode obtained via HTTPS. The --grid-http option will be ignored for
directory operations or HTTP URLs. If a redirected transfer isn't
possible, a normal HTTPS data transfer will be attempted.
- --sitecast
- Try to use SiteCast to locate remote files which are to be
copied (currently only for the fetching of remote files.) If no
location is found via SiteCast, then a direct request for the given URL is
tried. (--groups must be used for this option to work.)
- --domain <SiteCast domain>
- Try to use SiteCast to locate remote files which are to be
copied (currently only for the fetching of remote files) if the
domain component of the URL matches the SiteCast domain given. If no
location is found via SiteCast, then a direct request for the given URL is
tried. (--groups must be used for this option to work.)
FILES¶
- /tmp/x509up_uID
- Default GSI Proxy file for Unix UID equal to ID.
- /etc/grid-security/certificates
- Default location for trusted Certification Authority root
certificates to use when checking server certificates.
- /tmp/.ca-roots-XXXXXX
- Prior to 7.9.8, the underlying curl library did not support
the CA root certificates directory. If built with an old version of
libcurl, htcp will concatenate the certificates in the CA roots directory
into a unique temporary file and use that.
ENVIRONMENT¶
- X509_CERT_DIR
- Holds directory to search for Certification Authority root
certificates when verifying server certificates. (Tried if --capath is not
given on the command line.)
- X509_USER_PROXY
- Holds file name of a GSI Proxy to use as user certificate.
(Tried if --cert or --key are not given on the command line.)
- X509_USER_CERT and X509_USER_KEY
- Holds file name of X.509 user certificate and key. (Tried
if X509_USER_PROXY is not valid.)
EXIT CODES¶
0 is returned on complete success. Curl error codes are returned when reported
by the underlying curl library, and CURLE_HTTP_RETURNED_ERROR (22) is returned
when the HTTP(S) server returns a code outside the range 200-299. The manpage
libcurl-errors(3) lists all the curl error codes.
TO DO¶
Recursive copying. Server-side wildcards. Parallel streams. Better error
recovery.
AUTHOR¶
Andrew McNab <Andrew.McNab@manchester.ac.uk>
htcp is part of GridSite:
http://www.gridsite.org/
SEE ALSO¶
scp(1), curl(1), wget(1), verify(1),
libcurl-errors(3)
