NAME¶
urlgrabber - a high-level cross-protocol url-grabber.
SYNOPSIS¶
urlgrabber [OPTIONS] URL [FILE]
DESCRIPTION¶
urlgrabber is a binary program and python module for fetching
files. It is designed to be used in programs that need common (but not
necessarily simple) url-fetching features.
OPTIONS¶
--help, -h
help page specifying available options to the binary
program.
--copy-local
ignored except for
file:// urls, in which case it
specifies whether urlgrab should still make a copy of the file, or simply
point to the existing copy.
--throttle=NUMBER
if it's an int, it's the bytes/second throttle limit. If
it's a float, it is first multiplied by bandwidth. If throttle == 0,
throttling is disabled. If None, the module-level default (which can be set
with set_throttle) is used.
--bandwidth=NUMBER
the nominal max bandwidth in bytes/second. If throttle is
a float and bandwidth == 0, throttling is disabled. If None, the module-level
default (which can be set with set_bandwidth) is used.
--range=RANGE
a tuple of the form first_byte,last_byte describing a
byte range to retrieve. Either or both of the values may be specified. If
first_byte is None, byte offset 0 is assumed. If last_byte is None, the last
byte available is assumed. Note that both first and last_byte values are
inclusive so a range of (10,11) would return the 10th and 11th bytes of the
resource.
--user-agent=STR
the user-agent string provide if the url is HTTP.
--retry=NUMBER
the number of times to retry the grab before bailing. If
this is zero, it will retry forever. This was intentional... really, it was
:). If this value is not supplied or is supplied but is None retrying does not
occur.
--retrycodes
a sequence of errorcodes (values of e.errno) for which it
should retry. See the doc on URLGrabError for more details on this. retrycodes
defaults to -1,2,4,5,6,7 if not specified explicitly.
MODULE USE EXAMPLES¶
In its simplest form, urlgrabber can be a replacement for
urllib2's open, or even python's file if you're just reading:
from urlgrabber import urlopen
fo = urlopen(url)
data = fo.read()
fo.close()
Here, the url can be http, https, ftp, or file. It's also pretty smart so if you
just give it something like /tmp/foo, it will figure it out. For even more
fun, you can also do:
from urlgrabber import urlopen
local_filename = urlgrab(url) # grab a local copy of the file
data = urlread(url) # just read the data into a string
Now, like urllib2, what's really happening here is that you're using a
module-level object (called a grabber) that kind of serves as a default.
That's just fine, but you might want to get your own private version for a
couple of reasons:
* it's a little ugly to modify the default grabber because you have to
reach into the module to do it
* you could run into conflicts if different parts of the code
modify the default grabber and therefore expect different
behavior
Therefore, you're probably better off making your own. This also gives you lots
of flexibility for later, as you'll see:
from urlgrabber.grabber import URLGrabber
g = URLGrabber()
data = g.urlread(url)
This is nice because you can specify options when you create the grabber. For
example, let's turn on simple reget mode so that if we have part of a file, we
only need to fetch the rest:
from urlgrabber.grabber import URLGrabber
g = URLGrabber(reget='simple')
local_filename = g.urlgrab(url)
The available options are listed in the module documentation, and can usually be
specified as a default at the grabber-level or as options to the method:
from urlgrabber.grabber import URLGrabber
g = URLGrabber(reget='simple')
local_filename = g.urlgrab(url, filename=None, reget=None)
AUTHORS¶
Written by: Michael D. Stenner <mstenner@linux.duke.edu>
Ryan Tomayko <rtomayko@naeblis.cx>
This manual page was written by Kevin Coyner
<kevin@rustybear.com> for the Debian system (but may be used by
others). It borrows heavily on the documentation included in the urlgrabber
module. Permission is granted to copy, distribute and/or modify this
document under the terms of the GNU General Public License, Version 2 any
later version published by the Free Software Foundation.
