-reset boolean

By default, false. If true, any previous changes and any changes on the same call before the reset option will be returned to default values.

-retries integer

Default value is 3. If Amazon returns a 500 error, a retry after an exponential backoff delay will be tried this many times before finally throwing the 500 error. This applies to each call to S3::REST from the higher-level commands, but not to S3::REST itself. That is, S3::REST will always return httpstatus 500 if that's what it receives. Functions like S3::Put will retry the PUT call, and will also retry the GET and HEAD calls used to do content comparison. Changing this to 0 will prevent retries and their associated delays. In addition, socket errors (i.e., errors whose errorCode starts with "S3 socket") will be similarly retried after backoffs.

-accesskeyid idstring

-secretaccesskey idstring

Each defaults to an empty string. These must be set before any calls are made. This is your S3 ID. Once you sign up for an account, go to http://www.amazonaws.com/, sign in, go to the "Your Web Services Account" button, pick "AWS Access Identifiers", and your access key ID and secret access keys will be available. All S3::REST calls are authenticated. Blame Amazon for the poor choice of names.

-service-access-point FQDN

Defaults to "s3.amazonaws.com". This is the fully-qualified domain name of the server to contact for S3::REST calls. You should probably never need to touch this, unless someone else implements a compatible service, or you wish to test something by pointing the library at your own service.

-slop-seconds integer

When comparing dates between Amazon and the local machine, two dates within this many seconds of each other are considered the same. Useful for clock drift correction, processing overhead time, and so on.

-use-tls boolean

Defaults to false. This is not yet implemented. If true, S3::REST will negotiate a TLS connection to Amazon. If false, unencrypted connections are used.

-bucket-prefix string

Defaults to "TclS3". This string is used by S3::SuggestBucketName if that command is passed an empty string as an argument. It is used to distinguish different applications using the Amazon service. Your application should always set this to keep from interfering with the buckets of other users of Amazon S3 or with other buckets of the same user.

-default-compare always|never|exists|missing|newer|date|checksum|different

Defaults to "always." If no -compare is specified on S3::Put, S3::Get, or S3::Delete, this comparison is used. See those commands for a description of the meaning.

-default-separator string

Defaults to "/". This is currently unused. It might make sense to use this for S3::Push and S3::Pull, but allowing resources to have slashes in their names that aren't marking directories would be problematic. Hence, this currently does nothing.

-default-acl private|public-read|public-read-write|authenticated-read|keep|calc

Defaults to an empty string. If no -acl argument is provided to S3::Put or S3::Push, this string is used (given as the x-amz-acl header if not keep or calc). If this is also empty, no x-amz-acl header is generated. This is not used by S3::REST.

-default-bucket bucketname

If no bucket is given to S3::GetBucket, S3::PutBucket, S3::Get, S3::Put, S3::Head, S3::Acl, S3::Delete, S3::Push, S3::Pull, or S3::Toss, and if this configuration variable is not an empty string (and not simply "/"), then this value will be used for the bucket. This is useful if one program does a large amount of resource manipulation within a single bucket.

verb GET|PUT|DELETE|HEAD

This required item indicates the verb to be used.

resource string

This required item indicates the resource to be accessed. A leading / is added if not there already. It will be URL-encoded for you if necessary. Do not supply a resource name that is already URL-encoded.

?rtype torrent|acl?

This indicates a torrent or acl resource is being manipulated. Do not include this in the resource key, or the "?" separator will get URL-encoded.

?parameters dict?

This optional dictionary provides parameters added to the URL for the transaction. The keys must be in the correct case (which is confusing in the Amazon documentation) and the values must be valid. This can be an empty dictionary or omitted entirely if no parameters are desired. No other error checking on parameters is performed.

?headers dict?

This optional dictionary provides headers to be added to the HTTP request. The keys must be in lower case for the authentication to work. The values must not contain embedded newlines or carriage returns. This is primarily useful for adding x-amz-* headers. Since authentication is calculated by S3::REST, do not add that header here. Since content-type gets its own key, also do not add that header here.

?inbody contentstring?

This optional item, if provided, gives the content that will be sent. It is sent with a tranfer encoding of binary, and only the low bytes are used, so use [encoding convertto utf-8] if the string is a utf-8 string. This is written all in one blast, so if you are using non-blocking mode and the inbody is especially large, you may wind up blocking on the write socket.

?infile filename?

This optional item, if provided, and if inbody is not provided, names the file from which the body of the HTTP message will be constructed. The file is opened for reading and sent progressively by [fcopy], so it should not block in non-blocking mode even if the file is very large. The file is transfered in binary mode, so the bytes on your disk will match the bytes in your resource. Due to HTTP restrictions, it must be possible to use [file size] on this file to determine the size at the start of the transaction.

?S3chan channel?

This optional item, if provided, indicates the already-open socket over which the transaction should be conducted. If not provided, a connection is made to the service access point specified via S3::Configure, which is normally s3.amazonaws.com. If this is provided, the channel is not closed at the end of the transaction.

?outchan channel?

This optional item, if provided, indicates the already-open channel to which the body returned from S3 should be written. That is, to retrieve a large resource, open a file, set the translation mode, and pass the channel as the value of the key outchan. Output will be written to the channel in pieces so memory does not fill up unnecessarily. The channel is not closed at the end of the transaction.

?resultvar varname?

This optional item, if provided, indicates that S3::REST should run in non-blocking mode. The varname should be fully qualified with respect to namespaces and cannot be local to a proc. If provided, the result of the S3::REST call is assigned to this variable once everything has completed; use trace or vwait to know when this has happened. If this key is not provided, the result is simply returned from the call to S3::REST and no calls to the eventloop are invoked from within this call.

?throwsocket throw|return?

This optional item, if provided, indicates that S3::REST should throw an error if throwmode is throw and a socket error is encountered. It indicates that S3::REST should return the error code in the returned dictionary if a socket error is encountered and this is set to return. If throwsocket is set to return or if the call is not blocking, then a socket error (i.e., an error whose error code starts with "S3 socket" will be returned in the dictionary as error, errorInfo, and errorCode. If a foreground call is made (i.e., resultvar is not provided), and this option is not provided or is set to throw, then error will be invoked instead.

error errorstring

errorInfo errorstring

errorCode errorstring

If an error is caught, these three keys will be set in the result. Note that S3::REST does not consider a non-2XX HTTP return code as an error. The errorCode value will be formatted according to the ERROR REPORTING description. If these are present, other keys described here might not be.

httpstatus threedigits

The three-digit code from the HTTP transaction. 2XX for good, 5XX for server error, etc.

httpmessage text

The textual result after the status code. "OK" or "Forbidden" or etc.

outbody contentstring

If outchan was not specified, this key will hold a reference to the (unencoded) contents of the body returned. If Amazon returned an error (a la the httpstatus not a 2XX value), the error message will be in outbody or written to outchan as appropriate.

outheaders dict

This contains a dictionary of headers returned by Amazon. The keys are always lower case. It's mainly useful for finding the x-amz-meta-* headers, if any, although things like last-modified and content-type are also useful. The keys of this dictionary are always lower case. Both keys and values are trimmed of extraneous whitespace.

always

Always copy the data. This is the default.

never

Never copy the data. This is essentially a no-op, except in S3::Push and S3::Pull where the -delete flag might make a difference.

exists

Copy the data only if the destination already exists.

missing

Copy the data only if the destination does not already exist.

newer

Copy the data if the destination is missing, or if the date on the source is newer than the date on the destination by at least S3::Configure -slop-seconds seconds. If the source is Amazon, the date is taken from the Last-Modified header. If the source is local, it is taken as the mtime of the file. If the source data is specified in a string rather than a file, it is taken as right now, via [clock seconds].

date

Like newer, except copy if the date is newer or older.

checksum

Calculate the MD5 checksum on the local file or string, ask Amazon for the eTag of the resource, and copy the data if they're different. Copy the data also if the destination is missing. Note that this can be slow with large local files unless the C version of the MD5 support is available.

different

Copy the data if the destination does not exist. If the destination exists and an actual file name was specified (rather than a content string), and the date on the file differs from the date on the resource, copy the data. If the data is provided as a content string, the "date" is treated as "right now", so it will likely always differ unless slop-seconds is large. If the dates are the same, the MD5 checksums are compared, and the data is copied if the checksums differ.

-blocking boolean

See above for standard definition.

-parse-xml xmlstring

See above for standard definition.

-result-type REST

The dictionary returned by S3::REST is the return value of S3::ListAllMyBuckets. In this case, a non-2XX httpstatus will not throw an error. You may not combine this with -parse-xml.

-result-type xml

The raw XML of the body is returned as the result (with no encoding applied).

-result-type pxml

The XML of the body as parsed by xsxp::parse is returned.

-result-type dict

A dictionary of interesting portions of the XML is returned. The dictionary contains the following keys:

-result-type names

A list of bucket names is returned with all other information stripped out. This is the default result type for this command.

-result-type owner

A list containing two elements is returned. The first element is the owner's ID, and the second is the owner's display name.

-bucket bucketname

The standard bucket argument.

-blocking boolean

The standard blocking argument.

-parse-xml xmlstring

The standard parse-xml argument.

-max-count integer

If supplied, this is the most number of records to be returned. If not supplied, the code will iterate until all records have been found. Not compatible with -parse-xml. Note that if this is supplied, only one call to S3::REST will be made. Otherwise, enough calls will be made to exhaust the listing, buffering results in memory, so take care if you may have huge buckets.

-prefix prefixstring

If present, restricts listing to resources with a particular prefix. One leading / is stripped if present.

-delimiter delimiterstring

If present, specifies a delimiter for the listing. The presence of this will summarize multiple resources into one entry, as if S3 supported directories. See the Amazon documentation for details.

-result-type REST|xml|pxml|names|dict

This indicates the format of the return result of the command.

-bucket

This specifies the bucket into which the resource will be written. Leading and/or trailing slashes are removed for you, as are spaces.

-resource

This is the full name of the resource within the bucket. A single leading slash is removed, but not a trailing slash. Spaces are not trimmed.

-blocking

The standard blocking flag.

-file

If this is specified, the filename must exist, must be readable, and must not be a special or directory file. [file size] must apply to it and must not change for the lifetime of the call. The default content-type is calculated based on the name and/or contents of the file. Specifying this is an error if -content is also specified, but at least one of -file or -content must be specified. (The file is allowed to not exist or not be readable if -compare never is specified.)

-content

If this is specified, the contentstring is sent as the body of the resource. The content-type defaults to "application/octet-string". Only the low bytes are sent, so non-ASCII should use the appropriate encoding (such as [encoding convertto utf-8]) before passing it to this routine, if necessary. Specifying this is an error if -file is also specified, but at least one of -file or -content must be specified.

-acl

This defaults to S3::Configure -default-acl if not specified. It sets the x-amz-acl header on the PUT operation. If the value provided is calc, the x-amz-acl header is calculated based on the I/O permissions of the file to be uploaded; it is an error to specify calc and -content. If the value provided is keep, the acl of the resource is read before the PUT (or the default is used if the resource does not exist), then set back to what it was after the PUT (if it existed). An error will occur if the resource is successfully written but the kept ACL cannot be then applied. This should never happen. Note: calc is not currently fully implemented.

-x-amz-meta-*

If any header starts with "-x-amz-meta-", its contents are added to the PUT command to be stored as metadata with the resource. Again, no encoding is performed, and the metadata should not contain characters like newlines, carriage returns, and so on. It is best to stick with simple ASCII strings, or to fix the library in several places.

-content-type

This overrides the content-type calculated by -file or sets the content-type for -content.

-compare

This is the standard compare mode argument. S3::Put returns 1 if the data was copied or 0 if the data was skipped due to the comparison mode so indicating it should be skipped.

-bucket

This specifies the bucket from which the resource will be read. Leading and/or trailing slashes are removed for you, as are spaces.

-resource

This is the full name of the resource within the bucket. A single leading slash is removed, but not a trailing slash. Spaces are not trimmed.

-blocking

The standard blocking flag.

-file

If this is specified, the body of the resource will be read into this file, incrementally without pulling it entirely into memory first. The parent directory must already exist. If the file already exists, it must be writable. If an error is thrown part-way through the process and the file already existed, it may be clobbered. If an error is thrown part-way through the process and the file did not already exist, any partial bits will be deleted. Specifying this is an error if -content is also specified, but at least one of -file or -content must be specified.

-timestamp

This is only valid in conjunction with -file. It may be specified as now or aws. The default is now. If now, the file's modification date is left up to the system. If aws, the file's mtime is set to match the Last-Modified header on the resource, synchronizing the two appropriately for -compare date or -compare newer.

-content

If this is specified, the contentvarname is a variable in the caller's scope (not necessarily global) that receives the value of the body of the resource. No encoding is done, so if the resource (for example) represents a UTF-8 byte sequence, use [encoding convertfrom utf-8] to get a valid UTF-8 string. If this is specified, the -compare is ignored unless it is never, in which case no assignment to contentvarname is performed. Specifying this is an error if -file is also specified, but at least one of -file or -content must be specified.

-compare

This is the standard compare mode argument. S3::Get returns 1 if the data was copied or 0 if the data was skipped due to the comparison mode so indicating it should be skipped.

-headers

If this is specified, the headers resulting from the fetch are stored in the provided variable, as a dictionary. This will include content-type and x-amz-meta-* headers, as well as the usual HTTP headers, the x-amz-id debugging headers, and so on. If no file is fetched (due to -compare or other errors), no assignment to this variable is performed.

-bucket

This specifies the bucket from which the resource will be read. Leading and/or trailing slashes are removed for you, as are spaces.

-resource

This is the full name of the resource within the bucket. A single leading slash is removed, but not a trailing slash. Spaces are not trimmed.

-blocking

The standard blocking flag.

-dict

If specified, the resulting dictionary from the S3::REST call is assigned to the indicated (not necessarily global) variable in the caller's scope.

-headers

If specified, the dictionary of headers from the result are assigned to the indicated (not necessarily global) variable in the caller's scope.

-status

If specified, the indicated (not necessarily global) variable in the caller's scope is assigned a 2-element list. The first element is the 3-digit HTTP status code, while the second element is the HTTP message (such as "OK" or "Forbidden").

-blocking boolean

See above for standard definition.

-bucket

This specifies the bucket from which the resource will be read. Leading and/or trailing slashes are removed for you, as are spaces.

-resource

This is the full name of the resource within the bucket. A single leading slash is removed, but not a trailing slash. Spaces are not trimmed.

-parse-xml xml

The XML from a previous GetACL can be passed in to be parsed into dictionary form. In this case, -result-type must be pxml or dict.

-result-type REST

The dictionary returned by S3::REST is the return value of S3::GetAcl. In this case, a non-2XX httpstatus will not throw an error.

-result-type xml

The raw XML of the body is returned as the result (with no encoding applied).

-result-type pxml

The XML of the body as parsed by xsxp::parse is returned.

-result-type dict

This fetches the ACL, parses it, and returns a dictionary of two elements.

 

The first element has the key "owner" whose value is the canonical ID of the owner of the resource.

 

The second element has the key "acl" whose value is a dictionary. Each key in the dictionary is one of Amazon's permissions, namely "READ", "WRITE", "READ_ACP", "WRITE_ACP", or "FULL_CONTROL". Each value of each key is a list of canonical IDs or group URLs that have that permission. Elements are not in the list in any particular order, and not all keys are necessarily present. Display names are not returned, as they are not especially useful; use pxml to obtain them if necessary.

-blocking boolean

See above for standard definition.

-bucket

This specifies the bucket from which the resource will be read. Leading and/or trailing slashes are removed for you, as are spaces.

-resource

This is the full name of the resource within the bucket. A single leading slash is removed, but not a trailing slash. Spaces are not trimmed.

-owner

If this is provided, it is assumed to match the owner of the resource. Otherwise, a GET may need to be issued against the resource to find the owner. If you already have the owner (such as from a call to S3::GetAcl, you can pass the value of the "owner" key as the value of this option, and it will be used in the construction of the XML.

-acl

If this option is specified, it provides the ACL the caller wishes to write to the resource. If this is not supplied or is empty, the value is taken from S3::Configure -default-acl. The ACL is written with a PUT to the ?acl resource.

 

If the value passed to this option starts with "<", it is taken to be a body to be PUT to the ACL resource.

 

If the value matches one of the standard Amazon x-amz-acl headers (i.e., a canned access policy), that header is translated to XML and then applied. The canned access policies are private, public-read, public-read-write, and authenticated-read (in lower case).

 

Otherwise, the value is assumed to be a dictionary formatted as the "acl" sub-entry within the dict returns by S3::GetAcl -result-type dict. The proper XML is generated and applied to the resource. Note that a value containing "//" is assumed to be a group, a value containing "@" is assumed to be an AmazonCustomerByEmail, and otherwise the value is assumed to be a canonical Amazon ID.

 

Note that you cannot change the owner, so calling GetAcl on a resource owned by one user and applying it via PutAcl on a resource owned by another user may not do exactly what you expect.

-bucket

This specifies the bucket from which the resource will be deleted. Leading and/or trailing slashes are removed for you, as are spaces.

-resource

This is the full name of the resource within the bucket. A single leading slash is removed, but not a trailing slash. Spaces are not trimmed.

-blocking

The standard blocking flag.

-status

If specified, the indicated (not necessarily global) variable in the caller's scope is set to a two-element list. The first element is the 3-digit HTTP status code. The second element is the HTTP message (such as "OK" or "Forbidden"). Note that Amazon's DELETE result is 204 on success, that being the code indicating no content in the returned body.

-bucket

This names the bucket into which data will be pushed.

-directory

This names the local directory from which files will be taken. It must exist, be readable via [glob] and so on. If only some of the files therein are readable, S3::Push will PUT those files that are readable and return in its results the list of files that could not be opened.

-prefix

This names the prefix that will be added to all resources. That is, it is the remote equivalent of -directory. If it is not specified, the root of the bucket will be treated as the remote directory. An example may clarify.

S3::Push -bucket test -directory /tmp/xyz -prefix hello/world

In this example, /tmp/xyz/pdq.html will be stored as http://s3.amazonaws.com/test/hello/world/pdq.html in Amazon's servers. Also, /tmp/xyz/abc/def/Hello will be stored as http://s3.amazonaws.com/test/hello/world/abc/def/Hello in Amazon's servers. Without the -prefix option, /tmp/xyz/pdq.html would be stored as http://s3.amazonaws.com/test/pdq.html.

-blocking

This is the standard blocking option.

-compare

If present, this is passed to each invocation of S3::Put. Naturally, S3::Configure -default-compare is used if this is not specified.

-x-amz-meta-*

If present, this is passed to each invocation of S3::Put. All copied files will have the same metadata.

-acl

If present, this is passed to each invocation of S3::Put.

-delete

This defaults to false. If true, resources in the destination that are not in the source directory are deleted with S3::Delete. Since only regular files are considered, the existance of a symlink, pipe, device, or directory in the local source will not prevent the deletion of a remote resource with a corresponding name.

-error

This controls the behavior of S3::Push in the event that S3::Put throws an error. Note that errors encountered on the local file system or in reading the list of resources in the remote bucket always throw errors. This option allows control over "partial" errors, when some files were copied and some were not. S3::Delete is always finished up, with errors simply recorded in the return result.

-progress

If this is specified and the indicated script prefix is not empty, the indicated script prefix will be invoked several times in the caller's context with additional arguments at various points in the processing. This allows progress reporting without backgrounding. The provided prefix will be invoked with additional arguments, with the first additional argument indicating what part of the process is being reported on. The prefix is initially invoked with args as the first additional argument and a dictionary representing the normalized arguments to the S3::Push call as the second additional argument. Then the prefix is invoked with local as the first additional argument and a list of suffixes of the files to be considered as the second argument. Then the prefix is invoked with remote as the first additional argument and a list of suffixes existing in the remote bucket as the second additional argument. Then, for each file in the local list, the prefix will be invoked with start as the first additional argument and the common suffix as the second additional argument. When S3::Put returns for that file, the prefix will be invoked with copy as the first additional argument, the common suffix as the second additional argument, and a third argument that will be "copied" (if S3::Put sent the resource), "skipped" (if S3::Put decided not to based on -compare), or the errorCode that S3::Put threw due to unexpected errors (in which case the third argument is a list that starts with "S3"). When all files have been transfered, the prefix may be invoked zero or more times with delete as the first additional argument and the suffix of the resource being deleted as the second additional argument, with a third argument being either an empty string (if the delete worked) or the errorCode from S3::Delete if it failed. Finally, the prefix will be invoked with finished as the first additional argument and the return value as the second additional argument.

-bucket

This names the bucket from which data will be pulled.

-directory

This names the local directory into which files will be written It must exist, be readable via [glob], writable for file creation, and so on. If only some of the files therein are writable, S3::Pull will GET those files that are writable and return in its results the list of files that could not be opened.

-prefix

The prefix of resources that will be considered for retrieval. See S3::Push for more details, examples, etc. (Of course, S3::Pull reads rather than writes, but the prefix is treated similarly.)

-blocking

This is the standard blocking option.

-compare

This is passed to each invocation of S3::Get if provided. Naturally, S3::Configure -default-compare is used if this is not provided.

-timestamp

This is passed to each invocation of S3::Get if provided.

-delete

If this is specified and true, files that exist in the -directory that are not in the -prefix will be deleted after all resources have been copied. In addition, empty directories (other than the top-level -directory) will be deleted, as Amazon S3 has no concept of an empty directory.

-error

See S3::Push for a description of this option.

-progress

See S3::Push for a description of this option. It differs slightly in that local directories may be included with a trailing slash to indicate they are directories.

-bucket

The bucket from which resources will be deleted.

-blocking

The standard blocking option.

-prefix

The prefix for resources to be deleted. Any resource that starts with this string will be deleted. This is required. To delete everything in the bucket, pass an empty string for the prefix.

-error

If this is "throw", S3::Toss rethrows any errors it encounters. If this is "break", S3::Toss returns with a normal return after the first error, recording that error in the return result. If this is "continue", which is the default, S3::Toss continues on and lists all errors in the return result.

-progress

If this is specified and not an empty string, the script prefix will be invoked several times in the context of the caller with additional arguments appended. Initially, it will be invoked with the first additional argument being args and the second being the processed list of arguments to S3::Toss. Then it is invoked with remote as the first additional argument and the list of suffixes in the bucket to be deleted as the second additional argument. Then it is invoked with the first additional argument being delete and the second additional argument being the suffix deleted and the third additional argument being "deleted" or "notdeleted" depending on whether S3::Delete threw an error. Finally, the script prefix is invoked with a first additional argument of "finished" and a second additional argument of the return value.

file mkdir ./tempfiles
S3::Pull -bucket sample -prefix of/interest -directory ./tempfiles \
  -timestamp aws
do_my_process ./tempfiles other arguments
S3::Push -bucket sample -prefix of/interest -directory ./tempfiles \
  -compare newer -delete true

S3::Pull -bucket sample -prefix of/interest -directory ./myfiles \
  -compare never -delete true