15.4. The End

Whenever I write a one-liner, I find it interesting to think about it in different ways. While I was writing it, I was mostly thinking from the bottom up; some of the complex nesting is a result of this. For example, the callback routine is fairly hairy, but once I had it written, I could change the data source from LWP::Simple::get to LWP::UserAgent and HTTP::Request::content quite easily.

Obviously, this spider does nothing more than visit HTML pages and try to grab all the links off each one. It could be more polite (as the LWP::RobotUA module is) and it could be smarter about which links to visit. In particular, there’s no sense of which pages have already been visited; a tied DBM of visited pages would solve that nicely.