Quote:
Originally Posted by NET WT  I have seen them in the past, but was never sure what they did. Earlier in the "Currently Active Users" there was an "MSNBot Spider"
So what are they? What do they do?
I was just wondering. |
Spiders crawl around the web recording the web sites they find. Most use this information for their search engines. There are few that use it for other reasons, like searching for terrorism. Mostly, they're harmless.
They can be controlled by placing a file, "robots.txt" at the top level of the web site (search for "robots.txt" will tell you how to create this file). Most spiders will obey your rules (except those looking for terrorists).
And most can't respond to
CGI sites. An example of a CGI site is
Google. Because there is no way that they can fill in the form so that all possible pages will be returned, most skip doing CGI.
So, if you want to hide your terrorism site from most spiders, use CGI
