Shmoocon: Covert Crawling

Billy Hoffman of SPI Labs presented on Covert Crawling: A Wolf Among Lambs. He is discussing how he created a web crawler that is designed to subvert log analysis.
Attacks are foreshadowed by reconnaissance (other than by worms) and are often followed by the attacker checking the site to see if they are successful.
You might want to check websites for many reasons. Monitoring competitors progress, where they are speaking etc. When AT&T ran the patent office website, it was possible for them to see what competitors were working on based on what they were looking at on the patent office website.
Making the website crawl appear like normal surfing avoids obvious signs in the logs.