Free Trial

Safari Books Online is a digital library providing on-demand subscription access to thousands of learning resources.


  • Create BookmarkCreate Bookmark
  • Create Note or TagCreate Note or Tag
  • DownloadDownload
  • PrintPrint
Share this Page URL
Help

Chapter 20. Web Automation  > Creating a Robot - Pg. 722

Discussion Although closely related to the get function discussed in Section 20.1, the mirror func- tion doesn't download the file unconditionally. It adds the If-Modified-Since header to the GET request it creates, so the server will not transfer the file unless it has been updated. The mirror function mirrors only a single page, not a full tree. To mirror a set of pages, use this recipe in conjunction Section 20.3. A good solution to mirroring an entire remote tree can be found in the w3mir program, also found on CPAN. Be careful! It's possible (and easy) to write programs that run amok and begin down- loading all web pages on the net. This is not only poor etiquette, it's also an infinite task, since some pages are dynamically generated. It could also get you into trouble with someone who doesn't want their pages downloaded en masse. See Also The documentation for the CPAN module LWP::Simple; the HTTP specification at http://www.w3.org/pub/WWW/Protocols/HTTP/ 20.11 Creating a Robot Problem