NAME¶

linkchecker - command line client to check HTML documents and websites for broken links

SYNOPSIS¶

linkchecker [options] [file-or-url]...

DESCRIPTION¶

LinkChecker features

EXAMPLES¶

The most common use checks the given domain recursively:

$ linkchecker http://www.example.com/

Beware that this checks the whole site which can have thousands of URLs. Use the -r option to restrict the recursion depth.

Don't check URLs with /secret in its name. All other links are checked as usual:

$ linkchecker --ignore-url=/secret mysite.example.com

Checking a local HTML file on Unix:

$ linkchecker ../bla.html

Checking a local HTML file on Windows:

C:\> linkchecker c:empest.html

You can skip the http:// url part if the domain starts with www.:

$ linkchecker www.example.com

You can skip the ftp:// url part if the domain starts with ftp.:

$ linkchecker -r0 ftp.example.com

Generate a sitemap graph and convert it with the graphviz dot utility:

$ linkchecker -odot -v www.example.com | dot -Tps > sitemap.ps

OPTIONS¶

CONFIGURATION FILES¶

Configuration files can specify all options above. They can also specify some options that cannot be set on the command line. See linkcheckerrc(5) for more info.

OUTPUT TYPES¶

Note that by default only errors and warnings are logged. You should use the option --verbose to get the complete URL list, especially when outputting a sitemap graph format.

REGULAR EXPRESSIONS¶

LinkChecker accepts Python regular expressions. See https://docs.python.org/howto/regex.html for an introduction. An addition is that a leading exclamation mark negates the regular expression.

COOKIE FILES¶

A cookie file contains standard HTTP header (RFC 2616) data with the following possible names:

Multiple entries are separated by a blank line. The example below will send two cookies to all URLs starting with http://example.com/hello/ and one to all URLs starting with https://example.org/:

Host: example.com
Path: /hello
Set-cookie: ID="smee"
Set-cookie: spam="egg"

Host: example.org
Set-cookie: baggage="elitist"; comment="hologram"

PROXY SUPPORT¶

To use a proxy on Unix or Windows set the http_proxy or https_proxy environment variables to the proxy URL. The URL should be of the form http://[user:pass@]host[:port]. LinkChecker also detects manual proxy settings of Internet Explorer under Windows systems. On a Mac use the Internet Config to select a proxy. You can also set a comma-separated domain list in the no_proxy environment variable to ignore any proxy settings for these domains. The curl_ca_bundle environment variable can be used to identify an alternative certificate bundle to be used with an HTTPS proxy.

Setting a HTTP proxy on Unix for example looks like this:

$ export http_proxy="http://proxy.example.com:8080"

Proxy authentication is also supported:

$ export http_proxy="http://user1:mypass@proxy.example.org:8081"

Setting a proxy on the Windows command prompt:

C:\> set http_proxy=http://proxy.example.com:8080

PERFORMED CHECKS¶

All URLs have to pass a preliminary syntax test. Minor quoting mistakes will issue a warning, all other invalid syntax issues are errors. After the syntax check passes, the URL is queued for connection checking. All connection check types are described below.

SITEMAPS¶

Sitemaps are parsed for links to check and can be detected either from a sitemap entry in a robots.txt, or when passed as a FILE-OR-URL argument in which case detection requires the urlset/sitemapindex tag to be within the first 70 characters of the sitemap. Compressed sitemap files are not supported.

PLUGINS¶

There are two plugin types: connection and content plugins. Connection plugins are run after a successful connection to the URL host. Content plugins are run if the URL type has content (mailto: URLs have no content for example) and if the check is not forbidden (ie. by HTTP robots.txt). Use the option --list-plugins for a list of plugins and their documentation. All plugins are enabled via the linkcheckerrc(5) configuration file.

RECURSION¶

Before descending recursively into a URL, it has to fulfill several conditions. They are checked in this order:

Note that the directory recursion reads all files in that directory, not just a subset like index.htm.

NOTES¶

URLs on the commandline starting with ftp. are treated like ftp://ftp., URLs starting with www. are treated like http://www.. You can also give local files as arguments. If you have your system configured to automatically establish a connection to the internet (e.g. with diald), it will connect when checking links not pointing to your local host. Use the --ignore-url option to prevent this.

Javascript links are not supported.

If your platform does not support threading, LinkChecker disables it automatically.

You can supply multiple user/password pairs in a configuration file.

ENVIRONMENT¶

RETURN VALUE¶

The return value is 2 when

The return value is 1 when

Else the return value is zero.

LIMITATIONS¶

LinkChecker consumes memory for each queued URL to check. With thousands of queued URLs the amount of consumed memory can become quite large. This might slow down the program or even the whole system.

FILES¶

$XDG_CONFIG_HOME/linkchecker/linkcheckerrc - default configuration file

$XDG_DATA_HOME/linkchecker/failures - default failures logger output filename

linkchecker-out.TYPE - default logger file output name

SEE ALSO¶

linkcheckerrc(5)

https://docs.python.org/library/codecs.html#standard-encodings - valid output encodings

https://docs.python.org/howto/regex.html - regular expression documentation

AUTHOR¶

Bastian Kleineidam <bastian.kleineidam@web.de>

COPYRIGHT¶

2000-2016 Bastian Kleineidam, 2010-2024 LinkChecker Authors