Well, basically they are here to calalog everything that goes on. Then when someone does a search on say, "detailing an HO scale BNSF GP-38", they will find a couple of links to some post here. There are frequently hundreds of them crawling around and as Tom says, they can eat up bandwidth and resources. I think they're like bats, coming out a night when most people can't see them...Illus said:So basically, when someone searches for something on Yahoo, and gets a hit on The Gauge, it shows up as a Slurp Spider... Interesting. Thanks for the info.
Illus said:So basically, when someone searches for something on Yahoo, and gets a hit on The Gauge, it shows up as a Slurp Spider... Interesting. Thanks for the info.
This is how "Cached" pages on Yahoo are updated as well as pages on "the wayback machine" are storedFiatFan said:Just the opposite. If The Gauge shows as a hit on a Yahoo search, it was found (generally) by a slurp spider. All a spider does is travel the internet looking for new (or changed) pages and reports those pages back to Yahoo (or Google or any other search engine).
If a spider is found on a server, it is simply cataloging the pages available and reporting those pages back to Yahoo.
Tom