Hawler is a web crawler written in ruby. Extremely useful for everyday spidering activities, but it was built to satisfy my many curiousities about the way the web is woven. All relevant content can be found here