NAME¶

MediaWiki::DumpFile - Process various dump files from a MediaWiki instance

SYNOPSIS¶

  use MediaWiki::DumpFile;
  $mw = MediaWiki::DumpFile->new;
  
  $sql = $mw->sql($filename);
  $sql = $mw->sql(\*FH);
  
  $pages = $mw->pages($filename);
  $pages = $mw->pages(\*FH);
  
  $fastpages = $mw->fastpages($filename);
  $fastpages = $mw->fastpages(\*FH);
  
  use MediaWiki::DumpFile::Compat;
  
  $pmwd = Parse::MediaWikiDump->new;

ABOUT¶

This module is used to parse various dump files from a MediaWiki instance. The most likely case is that you will want to be parsing content at http://download.wikimedia.org/backup-index.html provided by WikiMedia which includes the English and all other language Wikipedias.

This module is the successor to Parse::MediaWikiDump acting as a near full replacement in feature set and providing an independent 100% backwards compatible API that is faster than Parse::MediaWikiDump is (see the MediaWiki::DumpFile::Compat and MediaWiki::DumpFile::Benchmarks documentation for details).

STATUS¶

This software is maturing into a stable and tested state with known users; the API is stable and will not be changed. The software is actively being maintained and improved; please submit bug reports, feature requests, and other feedback to the author using the bug reporting features described below.

FUNCTIONS¶

SPEED¶

MediaWiki::DumpFile now runs in a slower configuration when installed with out the recommended Perl modules; this was done so that the package can be installed with out a C compiler and still have some utility. As well there is a fast mode available when parsing the XML document that can give significant speed boosts while giving up support for anything except for the article titles and text contents. If you want to decrease the processing overhead of this system follow this guide:

Install XML::CompactTree::XS

Having this module on your system will cause XML::TreePuller to use it automatically - this will net you a dramatic speed boost if it is not already installed. This can give you a 3-4 times speed increase when not using fast mode.

Use fast mode if possible

Details of fast mode and the restrictions it imposes are in the MediaWiki::DumpFile::Pages documentation. Fast mode is also available in the compatibility library as a new available option. Fast mode can give you a further 3-4 times speed increase over parsing with XML::CompatTree::XS installed but it does not require that module to function; fast mode is nearly the same speed with or with out XML::CompactTree::XS installed.

Stop using compatibility mode

If you are using the compatibility API you lose performance; the compatibility API is a set of wrappers around the MediaWiki::DumpFile API and while it is faster than the original Parse::MediaWikiDump::Pages it is still slower than MediaWiki::DumpFile::Pages by a few percent.

Use MediaWiki::DumpFile::FastPages

This is a subclass of MediaWiki::DumpFile::Pages that configures it by default to run in fast mode and uses a tuned iterator that decreases overhead another few percent. This is generally the absolute fastest fully supported and tested way to parse the XML dump files.

Start hacking

I've put some considerable effort into finding the fastest ways to parse the XML dump files. Probably the most important part of this research has been an XML benchmarking suite I created for specifically measuring the performance of parsing the Mediawiki page dump files. The benchmark suite is present in the module tarball in the speed_test/ directory. It contains a comprehensive set of test cases to measure the performance of a good number of XML parsers and parsing schemes from CPAN. You can use this suite as a starting point to see how various parsers work and how fast they go; as well you can use it to reliably verify the performance impacts of experiments in parsing performance.

The result of my research into XML parsers was to create XML::TreePuller which is the heart XML processing system of MediaWiki::DumpFile::Pages - it's fast but I'm positive there is room for improvement. Increaseing the speed of that module will increase the speed of MediaWiki::DumpFile::Pages as well.

Please consider sharing the results of your hacking with me by opening a ticket in the bug reporting system as documented below.

The following test cases are notable and could be used by anyone who just needs to extract article titles and text:

AUTHOR¶

Tyler Riddle, "<triddle at gmail.com>"

LIMITATIONS¶

English Wikipedia comprehensive dump files not supported

There are two types of Mediawiki dump files sharing one schema: ones with one revision of page per entry and one with multiple revisions of a page per entry. This software is designed to parse either case and provide a consistent API however it comes with the restriction that an entire entry must fit in memory. The normal English Wikipedia dump file is around 20 gigabytes and each entry easily fits into RAM on most machines.

In the case of the comprehensive English Wikipedia dump files the file itself is measured in the terabytes and a single entry can be 20 gigabytes or more. It is technically possible for the original Parse::MediaWikiDump::Revisions (not the compatibility version provided in this module) to parse that dump file however Parse::MediaWikiDump runs at a few megabytes per second under the best of conditions.

BUGS¶

Please report any bugs or feature requests to "bug-mediawiki-dumpfile at rt.cpan.org", or through the web interface at <http://rt.cpan.org/NoAuth/ReportBug.html?Queue=MediaWiki-DumpFile>. I will be notified, and then you'll automatically be notified of progress on your bug as I make changes.

56843 ::Pages->current_byte() wraps at 2 gigs+

If you have a large XML file, where the file size is greater than a signed 32bit integer, the returned value from this method can go negative.

SUPPORT¶

You can find documentation for this module with the perldoc command.

    perldoc MediaWiki::DumpFile

You can also look for information at:

• RT: CPAN's request tracker

  <http://rt.cpan.org/NoAuth/Bugs.html?Dist=MediaWiki-DumpFile>

• AnnoCPAN: Annotated CPAN documentation

  <http://annocpan.org/dist/MediaWiki-DumpFile>

• CPAN Ratings

  <http://cpanratings.perl.org/d/MediaWiki-DumpFile>

• Search CPAN

  <http://search.cpan.org/dist/MediaWiki-DumpFile/>

ACKNOWLEDGEMENTS¶

All of the people who reported bugs or feature requests for Parse::MediaWikiDump.

COPYRIGHT & LICENSE¶

Copyright 2009 "Tyler Riddle".

This program is free software; you can redistribute it and/or modify it under the terms of either: the GNU General Public License as published by the Free Software Foundation; or the Artistic License.

See http://dev.perl.org/licenses/ for more information.