'\" -*- coding: UTF-8 -*- .if \n(.g .ds T< \\FC .if \n(.g .ds T> \\F[\n[.fam]] .de URL \\$2 \(la\\$1\(ra\\$3 .. .if \n(.g .mso www.tmac .TH awffull.conf 5 2008-Dec-13 "" "" .SH NAME AWFFull \- A Webalizer Fork, Full o' features .SH DESCRIPTION awffull.conf is the configuration file for awffull(1). awffull.conf is a standard ASCII(7) text files that may be created or edited using any standard editor. .PP Blank lines and lines that begin with a pound sign ('#') are ignored. .PP Any other lines are considered to be configuration lines, and have the form \(oqKeyword Value\(cq, where the \(oqKeyword\(cq is one of the currently available configuration keywords, and \(oqValue\(cq is the value to assign to that particular option. .PP Any text found after the keyword up to the end of the line is considered the keyword's value, so you should not include anything after the actual value on the line that is not actually part of the value being assigned. The file \*(T<\fIsample.conf\fR\*(T> provided with the distribution contains lots of useful documentation and examples as well. .PP Some \(oqKeywords\(cq will accept a 2^nd value. In those situations, the first value may be enclosed in double quotes (") to allow for whitespace. .PP Keywords are Case Insensitive. Values are Case Sensitive, with some gotchas: See Ignore* for details. .SH WILDCARDS Wildcards within AWFFull are a little non standard and may cause some confusion. .PP Wildcards are only valid within the Value of certain keywords .PP A Value can have either a leading or trailing '*' to signify a wildcard character. If no wildcard is found, a match can occur anywhere in the string. Given a string \(oqwww.yourmama.com\(cq, the values \(oqyour\(cq, \(oq*mama.com\(cq and \(oqwww.your*\(cq will all match. .PP Thus the use of the wildcard signifies that the other end of the Value is anchored at the Beginning or End of a field to be searched against. .PP eg. A Value of \(oqBot*\(cq implies that the field (probably UserAgent in this case) MUST start with the letters Bot. Or in the case of a Hostname \(oq*.gov.au\(cq implies a match ONLY against Australian Government hostnames. .SH "RUN OPTIONS" The Run Options are the generic ones that tell AWFFull where stuff is and how to generally operate. Some of these can modify the results that AWFFull will produce. .TP OutputDir OutputDir is where you want to put the output files. This should should be a full path name, however relative ones might work as well. If no output directory is specified, the current directory will be used. .TP LogFile LogFile defines the web server log file to use. If not specified here or on on the command line, input will default to STDIN. If the log filename ends in '.gz' (ie: a gzip compressed file), it will be decompressed on the fly as it is being read. .TP LogType LogType defines the log type being processed. Normally, AWFFull expects a CLF or Combined web server log as input. Using this option, you can process ftp logs as well (\*(T<\fIxferlog\fR\*(T> as produced by wu-ftpd and others), or Squid native logs. Values can be 'auto' \&'clf', 'combined', 'ftp', 'domino' or 'squid', with 'auto' the default. The 'auto' value means that AWFFull will try and work out what log format you are sending to it. If no joy, AWFFull will immediately exit. .TP GeoIP GeoIP enables or disables the use of the GeoIP capability for more accurate detection of countries. Default is \(oqno\(cq. NOTE! Do not enable GeoIP if you analyse files that have had the IP Address translated to a Fully Qualified Host Name. Use either raw IP Addresses and GeoIP, or Names and disable GeoIP. ie. Don't use GeoIP AND DNShistory. .TP GeoIPDatabase GeoIPDatabase is the location of the GeoIP database file. Default is \*(T<\fI/usr/share/GeoIP/GeoIP.dat\fR\*(T>, which is where a default GeoIP install will put it. Note that the database is updated monthly. For the details see: \(lahttp://www.maxmind.com/app/geoip_country\(ra .TP Incremental Incremental processing allows multiple partial log files to be used instead of one huge one. Useful for large sites that have to rotate their log files more than once a month. AWFFull will save its internal state before exiting, and restore it the next time run, in order to continue processing where it left off. This mode also causes AWFFull to scan for and ignore duplicate records (records already processed by a previous run). See the \*(T<\fIREADME\fR\*(T> file for additional information. The value may be 'yes' or 'no', with a default of 'no'. The file \*(T<\fIawffull.current\fR\*(T> is used to store the current state data, and is located in the output directory of the program (unless changed with the IncrementalName option below). Please read at least the section on Incremental processing in the \*(T<\fIREADME\fR\*(T> file before you enable this option. .TP TimeMe TimeMe allows you to force the display of timing information at the end of processing. A value of 'yes' will force the timing information to be displayed. A value of 'no' has no effect. .TP IgnoreHist IgnoreHist should not be used in a standard configuration, but it is here because it is useful in certain analysis situations. If the history file is ignored, the main \(oq\*(T<\fIindex.html\fR\*(T>\(cq file will only report on the current log files contents. Incremental data (if present) is still processed. Useful when you want to reproduce the reports from scratch, for example. USE WITH CAUTION! Valid values are \(oqyes\(cq or \(oqno\(cq. Default is \(oqno\(cq. .TP IncrementalName IncrementalName allows you to specify the filename for saving the incremental data in. It is similar to the HistoryName option where the name is relative to the specified output directory, unless an absolute filename is specified. The default is a file named \(oq\*(T<\fIawffull.current\fR\*(T>\(cq kept in the normal output directory. If you don't specify Incremental as 'yes' then this option has no meaning. .TP HistoryName HistoryName allows you to specify the name of the history file produced by AWFFull. The history file keeps the data for up to 12 months worth of logs, used for generating the main HTML page (\*(T<\fIindex.html\fR\*(T>). The default is a file named \*(T<\fIawffull.hist\fR\*(T>, stored in the specified output directory. If you specify just the filename (without a path), it will be kept in the specified output directory. Otherwise, the path is relative to the output directory, unless absolute (leading /). .SH "ANALYSIS OPTIONS" These are the basic analysis options that one can and should modify to start fine tuning AWFFull against a given website. .TP PageType PageType lets you tell AWFFull what types of URL's you consider a 'page'. Most people consider html and cgi documents as pages, while not images and audio files. If no types are specified, defaults will be used ('htm', 'html', 'cgi' and HTMLExtension if different for web logs, 'txt' for ftp logs). Putting the more likely page types first in the list should increase the speed of a run. Do Not Use Wildcards Here. It will not work. .TP NotPageType NotPageType is the direct and incompatible opposite of PageType. You can use one set or the other, but not both. PageType specifies what *is* a Page, NotPageType specifies what *isn't*, and hence by implication, everything else is a page. Neither method is more or lessor correct than the other. It's more what is more accurate for *your* site. Do not add the "." or use any wildcards. As a general rule. There are some assumed internal optimisations that may otherwise break. Those who understand pcre's would do well to examine the source of parser.c if they wish to extract greater flexibility from the below. .TP FoldSeqErr FoldSeqErr forces AWFFull to ignore sequence errors. This is useful for Netscape and other web servers that cache the writing of log records and do not guarantee that they will be in chronological order. The use of the FoldSeqErr option will cause out of sequence log records to be treated as if they had the same time stamp as the last valid record. The default action is to ignore out of sequence log records. .TP SearchEngine The SearchEngine keywords allow specification of search engines and their query strings on the URL. These are used to locate and report what search strings are used to find your site. The first word is a substring to match in the referrer field that identifies the search engine, and the second is the URL variable used by that search engine to define it's search terms. .TP VisitTimeout VisitTimeout allows you to set the default timeout for a visit (sometimes called a 'session'). The default is 30 minutes, which should be fine for most sites. Visits are determined by looking at the time of the current request, and the time of the last request from the site. If the time difference is greater than the VisitTimeout value, it is considered a new visit, and visit totals are incremented. Value is the number of seconds to timeout (default=1800=30min) .TP TrackPartialRequests TrackPartialRequests is used to track 206 codes. This gives two additional columns in the Top URLs tables. The first to "Hits" counts the number of partial requests The second to "Volume" counts the volume in partial requests This option is more of use to those with lots of PDF's. .TP MangleAgents The MangleAgents allows you to specify how much, if any, AWFFull should mangle user agent names. This allows several levels of detail to be produced when reporting user agent statistics. There are six levels that can be specified, which define different levels of detail suppression. Level 5 shows only the browser name (MSIE or Mozilla) and the major version number. Level 4 adds the minor version number (single decimal place). Level 3 displays the minor version to two decimal places. Level 2 will add any sub-level designation (such as Mozilla/3.01Gold or MSIE 3.0b). Level 1 will attempt to also add the system type if it is specified. The default Level 0 displays the full user agent field without modification and produces the greatest amount of detail. User agent names that can't be mangled will be left unmodified. .TP AssignToCountry AssignToCountry allows a form of override to force given domains to a specified country. Use the standard 2 letter country codes. Can also use org, com, net and so on, if more appropriate. With judicious use of AllSites, GroupSite and 'whois', this can cover the majority of your users without too much effort. .TP IndexAlias AWFFull normally strips the string 'index.' off the end of URL's in order to consolidate URL totals. For example, the URL /somedir/index.html is turned into /somedir/ which is really the same URL. This option allows you to specify additional strings to treat in the same way. You don't need to specify 'index.' as it is always scanned for by AWFFull, this option is just to specify _additional_ strings if needed. If you don't need any, don't specify any as each string will be scanned for in EVERY log record... A bunch of them will degrade performance. Also, the string is scanned for anywhere in the URL, so a string of 'home' would turn the URL /somedir/homepages/brad/home.html into just /somedir/ which is probably not what was intended. .TP IgnoreIndexAlias The opposite (in a way) of IndexAlias is IgnoreIndexAlias. This will STOP any URL variable stripping, as well as ignoring the default "index." setting, or any that you set above. .SH "IGNORE* OPTIONS" The Ignore* keywords allow you to completely ignore, or filter away, log records based on hostname, URL, user agent, referrer or user name. Use the same syntax as the Hide* keywords, where the value can have a leading or trailing wildcard '*'. .TP IgnoreURL Filters out traffic accessing certain URLs. eg You may wish to avoid seeing traffic that accesses administration functions, thus "IgnoreURL /admin*". URLs are case sensitive. .TP IgnoreSite Ignore sites that visit this website. Ignore by what is presented to awffull - name or IP Address. Sites are lowercased prior to filtering, so if Ignore'ing by name, do use a lowercased Value. .TP IgnoreReferrer Ignore specified referrers. Very useful for filtering away SPAM Referrers. Referrers are partially case sensitive. \eo/ The host portion is lowercased; the URI is case sensitive. .TP IgnoreUser Ignore specified users. User names are lowercased prior to filtering. .TP IgnoreAgent Agents are case sensitive. .SH "INCLUDE* OPTIONS" The Include* keywords allow you to force the inclusion of log records based on hostname, URL, user agent, referrer or user name. The Include* keywords take precedence over the Ignore* keywords. .PP Note: Using Ignore/Include combinations to selectively process parts of a web site is _extremely inefficient_!!! Avoid doing so if possible ie: grep or gawk the records to a separate file if you really want that kind of report. .TP IncludeURL .TP IncludeSite .TP IncludeReferrer .TP IncludeUser .TP IncludeAgent .SH "SEGMENTING OPTIONS" Segmenting is a bit like the Ignore* and Include* keywords. Where it differs is in "remembering". Such that, as a \(oqsession\(cq (or \(oqvisit\(cq) moves away from the original entry condition, that session is still tracked. So if you segment on a referal from Google, only sessions that were referred to the analysed website, from Google, will be tracked. Even as that same session accesses other pages within the website. .PP eg. Google -> Site Page 1 -> Site Page 2 -> Site Page 3 .PP Whereas Ignore/Include would only filter the first interaction. eg. Google -> Site Page 1 .PP By "session" (or \(oqvisit\(cq) it is meant that the time limitation of a session (typically 30 minutes timeout) will impact. So in the above example from Google, if the last step (from Page 2 to Page 3) occurred 31+ minutes after the Page 1 to Page 2 transition, then this final step would NOT be included. The trail would be: .PP Google -> Site Page 1 -> Site Page 2 .PP Please do be aware that currently AWFFull uses IP Addresses to determine the continuation of a given session. This will be most flawed if you have a user population that sits behind corporate firewalls, or ISP Proxies. To mention two major problem areas. Why do Segmenting? \(lahttp://judah.webanalyticsdemystified.com/2007/11/a-few-tips-on-web-analytics-segmentation.html\(ra .PP \(oqSegment analysis will tell you different things about your audience than you will realize from studying overall population metrics.\(cq .PP \(oqThe goal of segmentation is to maximize future value of that segment by optimizing your marketing mix.\(cq .PP With apologies to Judah for mixing his phrase order around. :-) .TP SegCountry Segment by Country: Only track sessions that come from the following countries. This will be determined by: .RS .TP 0.4i 1. Use of AssignToCountry overrides .TP 0.4i 2. GeoIP lookups if so configured and enabled .TP 0.4i 3. Hostname TLD. eg .au .RE The third option is generally going to be the worst for accuracy. eg. We have plenty of Australian IP addresses that otherwise resolve to .com or .net etc. It is strongly advised to enable GeoIP if you wish to use this option. .TP SegReferer Segment by Referer: Only track sessions that originated from the following referers. NOTE!!!! SegReferer only works against the HOST name. Not the full URL. .SH "DISPLAY OPTIONS" The Display Options modify the resulting output that AWFFull produces. Things like HTML Headers and Footers to add on every page. These options don't change the numbers that AWFFull will calculate, but may change which ones appear, giving the illusion of a numerical change. .TP ReportTitle ReportTitle is the text to display as the title. The hostname (unless blank) is appended to the end of this string (separated with a space) to generate the final full title string. Default is (for English) \(oqUsage Statistics for\(cq. .TP HostName HostName defines the hostname for the report. This is used in the title, and is prepended to the URL table items. This allows clicking on URL's in the report to go to the proper location in the event you are running the report on a 'virtual' web server, or for a server different than the one the report resides on. If not specified here, or on the command line, AWFFull will try to get the hostname via a uname system call. If that fails, it will default to \(oqlocalhost\(cq. .TP IndexMonths This option controls how many years worth of data to display on the front summary page. In months. eg: Display the last 5 years: 5 x 12 = 60 .TP DailyStats DailyStats allows the daily statistics table to be disabled - not displayed. Values may be \(oqyes\(cq or \(oqno\(cq. Default is \(oqyes\(cq - do display the Daily Statistics table. .TP HourlyStats HourlyGraph and HourlyStats allows the hourly statistics graph and statistics table to be disabled (not displayed). Values may be "yes" or "no". Default is "yes". .TP CSSFilename CSSFilename is used to set the name of the CSS file to use in conjunction with the generated html. An existing file is not overwritten, so feel free to make you own changes to the default file. The default is \*(T<\fIawffull.css\fR\*(T>. .TP FlagsLocation FlagsLocation will enable the display of country flag pictures in the country table. The path is that for a webserver, not file system. Can be relative or complete. The trailing slash is not necessary. The default location is not set and hence will not be used. .TP YearlySubtotals YearlySubtotals will display the subtotal for a given year in the main page. This is in addition to the Grand Total of all years. .TP GroupShading The GroupShading allows grouped rows to be shaded in the report. Useful if you have lots of groups and individual records that intermingle in the report, and you want to differentiate the group records a little more. Value can be \(oqyes\(cq or \(oqno\(cq, with \(oqyes\(cq being the default. .TP GroupHighlight GroupHighlight allows the group record to be displayed in BOLD. Can be either \(oqyes\(cq or \(oqno\(cq with the default being \(oqyes\(cq. .TP HTMLExtension HTMLExtension allows you to specify the filename extension to use for generated HTML pages. Normally, this defaults to "html", but can be changed for sites who need it (like for PHP embedded pages). .TP UseHTTPS UseHTTPS should be used if the analysis is being run on a secure server, and links to urls should use \(oqhttps://\(cq instead of the default \(oqhttp://\(cq. If you need this, set it to \(oqyes\(cq. Default is \(oqno\(cq. This only changes the behaviour of the \(oqTop URLs\(cq table. .TP Top* The various \(oqTop\(cq options below define the number of entries for each table. Tables may be disabled by using zero (0) for the value. .TP TopURLs The most accessed URLs or Resources by number of requests (hits). Includes both Pages and Images, for example. Defaults to 30 URLs. .TP TopKURLs The greatest volume generating URLs. Defaults to 10 URL's. .TP TopEntry The most accessed initial URLs within a complete Visit. Will also display Single Access counts, Stickiness ration and Popularity ratio. Defaults to 10 URLs. .TP TopExit The most accessed last URLs within a complete Visit. ie: The last page recorded of a Visit. Also displays the Popularity ratio. Defaults to 10 URLs. .TP Top404Errors The most seen error requests and a corresponding referring URL. Defaults to 0, ie not shown. .TP TopSites Those Sites that have accessed the most Pages. Default is 30 Sites. .TP TopKSites Those Sites that have downloaded the greatest Volume. Default is 10 Sites. .TP TopReferrers Those local and remote URLs that refer the most requests. Default is 30 Referrers. .TP TopSearch Those words and phrases used at remote Search Engines to direct traffic here. Default is 20 Phrases. .TP TopUsers Those logged in users who most use the site. Default is 20 Users. .TP TopAgents The Browser Agents that are busiest against this site. Default is 15 Agents. .TP TopCountries A view of all traffic against this site via country. .TP All* The All* keywords allow the display of all the below measures. If enabled, a separate HTML page will be created, and a link will be added to the bottom of the appropriate "Top" table. There are a couple of conditions for this to occur. First, there must be more items than will fit in the "Top" table (otherwise it would just be duplicating what is already displayed). Second, the listing will only show those items that are normally visible, which means it will not show any hidden items. Grouped entries will be listed first, followed by individual items. The value for these keywords can be either 'yes' or 'no', with the default being 'no'. Please be aware that these pages can be quite large in size, particularly the sites page, and separate pages are generated for each month, which can consume quite a lot of disk space depending on the traffic to your site. .TP AllURLs All accessed URLs .TP AllEntryPages All Pages that initialised a Visit .TP AllExitPages All the last or exit pages in all Visits. .TP All404Errors All ErrorRequests and the corresponding referral URLs. .TP AllSites All remote sites that accessed this website. .TP AllReferrers All local and remote referring URLs .TP AllSearchStr All Remote Search Engine words and Phrases used to refer traffic here. .TP AllUsers All users who logged into this website. .TP AllAgents All Browser Agents used to access this site. Useful for identifying robots. .TP GMTTime GMTTime allows reports to show GMT (UTC) time instead of local time. Default is to display the time the report was generated in the timezone of the local machine, such as EDT or PST. This keyword allows you to have times displayed in UTC instead. Use only if you really have a good reason, since it will probably screw up the reporting periods by however many hours your local time zone is off of GMT. .TP HTMLPre HTMLPre defines HTML code to insert at the very beginning of the file. Default is the DOCTYPE line shown below. Max line length is 80 characters, so use multiple HTMLPre lines if you need more. .TP HTMLHead HTMLHead defines HTML code to insert within the \*(T<
\*(T> block, immediately after the \*(T<