NAME¶

HTTP::Async - process multiple HTTP requests in parallel without blocking.

SYNOPSIS¶

Create an object and add some requests to it:

    use HTTP::Async;
    my $async = HTTP::Async->new;
    
    # create some requests and add them to the queue.
    $async->add( HTTP::Request->new( GET => 'http://www.perl.org/'         ) );
    $async->add( HTTP::Request->new( GET => 'http://www.ecclestoad.co.uk/' ) );

and then EITHER process the responses as they come back:

    while ( my $response = $async->wait_for_next_response ) {
        # Do some processing with $response
    }

OR do something else if there is no response ready:

    while ( $async->not_empty ) {
        if ( my $response = $async->next_response ) {
            # deal with $response
        } else {
            # do something else
        }
    }

OR just use the async object to fetch stuff in the background and deal with the responses at the end.

    # Do some long code...
    for ( 1 .. 100 ) {
      some_function();
      $async->poke;            # lets it check for incoming data.
    }
    while ( my $response = $async->wait_for_next_response ) {
        # Do some processing with $response
    }

DESCRIPTION¶

Although using the conventional "LWP::UserAgent" is fast and easy it does have some drawbacks - the code execution blocks until the request has been completed and it is only possible to process one request at a time. "HTTP::Async" attempts to address these limitations.

It gives you a 'Async' object that you can add requests to, and then get the requests off as they finish. The actual sending and receiving of the requests is abstracted. As soon as you add a request it is transmitted, if there are too many requests in progress at the moment they are queued. There is no concept of starting or stopping - it runs continuously.

Whilst it is waiting to receive data it returns control to the code that called it meaning that you can carry out processing whilst fetching data from the network. All without forking or threading - it is actually done using "select" lists.

Default settings:¶

There are a number of default settings that should be suitable for most uses. However in some circumstances you might wish to change these.

            slots:  20
          timeout:  180 (seconds)
 max_request_time:  300 (seconds)
    max_redirect:   7
    poll_interval:  0.05 (seconds)
       proxy_host:  ''
       proxy_port:  ''
       local_addr:  ''
       local_port:  ''
       ssl_options: {}
       cookie_jar:  undef
       peer_addr:   ''

If defined, is expected to be similar to "HTTP::Cookies", with extract_cookies and add_cookie_header methods.

The option max_redirects has been renamed to max_redirect to be consistent with LWP::UserAgent, although max_redirects still works.

METHODS¶

    my $async = HTTP::Async->new( %args );

    $old_value = $async->slots;
    $new_value = $async->slots( $new_value );

    my @ids      = $async->add(@requests);
    my $first_id = $async->add(@requests);

    my $id = $async->add_with_opts( $request, \%opts );

    $async->poke;

    my $response          = $async->next_response;
    my ( $response, $id ) = $async->next_response;

    my $response          = $async->wait_for_next_response( 3.5 );
    my ( $response, $id ) = $async->wait_for_next_response( 3.5 );

    my $pending = $async->to_send_count;

    my $completed = $async->to_return_count;

    my $running = $async->in_progress_count;

    my $total = $async->total_count;

    print $async->info;

    $async->remove($id);
    my $success = $async->remove($id);

    $async->remove_all;
    my $success = $async->remove_all;

    while ( $async->not_empty ) { ...; }
    while (1) { ...; last if $async->empty; }

SEE ALSO¶

HTTP::Async::Polite - a polite form of this module. Slows the scraping down by domain so that the remote server is not overloaded.

GOTCHAS¶

The responses may not come back in the same order as the requests were made. For https requests to work, you must have Net::HTTPS::NB installed.

THANKS¶

Egor Egorov contributed patches for proxies, catching connections that die before headers sent and more.

Tomohiro Ikebe from livedoor.jp submitted patches (and a test) to properly handle 304 responses.

Naveed Massjouni for adding the https handling code.

Alex Balhatchet for adding the https + proxy handling code, and for making the tests run ok in parallel.

Josef Toman for fixing two bugs, one related to header handling and another related to producing an absolute URL correctly.

Github user 'c00ler-' for adding LocalAddr and LocalPort support.

rt.cpan.org user 'Florian (fschlich)' for typo in documentation.

Heikki Vatiainen for the ssl_options support patch.

Daniel Lintott of the Debian Perl Group for pointing out a test failure when using a very recent version of HTTP::Server::Simple to implement t/TestServer.pm

BUGS AND REPO¶

Please submit all bugs, patches etc on github

<https://github.com/evdb/HTTP-Async>

AUTHOR¶

Edmund von der Burg "<evdb@ecclestoad.co.uk>".

<http://www.ecclestoad.co.uk/>

LICENCE AND COPYRIGHT¶

Copyright (c) 2006, Edmund von der Burg "<evdb@ecclestoad.co.uk>". All rights reserved.

This module is free software; you can redistribute it and/or modify it under the same terms as Perl itself.

DISCLAIMER OF WARRANTY¶

BECAUSE THIS SOFTWARE IS LICENSED FREE OF CHARGE, THERE IS NO WARRANTY FOR THE SOFTWARE, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES PROVIDE THE SOFTWARE "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE SOFTWARE IS WITH YOU. SHOULD THE SOFTWARE PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING, REPAIR, OR CORRECTION.

IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY AND/OR REDISTRIBUTE THE SOFTWARE AS PERMITTED BY THE ABOVE LICENCE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL, OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE USE OR INABILITY TO USE THE SOFTWARE (INCLUDING BUT NOT LIMITED TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD PARTIES OR A FAILURE OF THE SOFTWARE TO OPERATE WITH ANY OTHER SOFTWARE), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.