LWP::RobotUA - A class for Web Robots

SYNOPSIS

         require LWP::RobotUA;
         $ua = new LWP::RobotUA 'my-robot/0.1', 'me@foo.com';
         $ua->delay(10);  # be very nice, go slowly
         ...
         # just use it just like a normal LWP::UserAgent
         $res = $ua->request($req);

DESCRIPTION

       This class implements a user agent that is suitable for
       robot applications.  Robots should be nice to the servers
       they visit.  They should consult the robots.txt file to
       ensure that they are welcomed and they should not send too
       frequent requests.

       But, before you consider writing a robot take a look at
       <URL:http://info.webcrawler.com/mak/projects/robots/robots.html>.

       When you use a LWP::RobotUA as your user agent, then you
       do not really have to think about these things yourself.
       Just send requests as you do when you are using a normal
       LWP::UserAgent and this special agent will make sure you
       are nice.

METHODS

       The LWP::RobotUA is a sub-class of LWP::UserAgent and
       implements the same methods.  The use_alarm() method also
       desides whether we will wait if a request is tried too
       early (if true), or will return an error response (if
       false).

       In addition these methods are provided:

       $ua = LWP::RobotUA->new($agent_name, $from, [$rules])
           Your robot's name and the mail address of the human
           responsible for the robot (i.e. you) is required by
           the constructor.

           Optionally it allows you to specify the
           WWW::RobotRules object to use.

       $ua->delay([$minutes])
           Set the minimum delay between requests to the same
           server.  The default is 1 minute.

       $ua->use_sleep([$boolean])
           Get/set a value indicating wether the UA should
           sleep() if request arrive to fast (before $ua->delay
           minutes has passed).  The default is TRUE.  If this
           After header that indicate when it is OK to send
           another request to this server.

       $ua->rules([$rules])
           Set/get which WWW::RobotRules object to use.

       $ua->no_visits($netloc)
           Returns the number of documents fetched from this
           server host. Yes I know, this method should probably
           have been named num_visits() or something like that
           :-(

       $ua->host_wait($netloc)
           Returns the number of seconds (from now) you must wait
           before you can make a new request to this host.

       $ua->as_string
           Returns a text that describe the state of the UA.
           Mainly useful for debugging.

SEE ALSO

       the LWP::UserAgent manpage, the WWW::RobotRules manpage

COPYRIGHT

       Copyright 1996-1997 Gisle Aas.

       This library is free software; you can redistribute it
       and/or modify it under the same terms as Perl itself.