Frequently Asked Questions
What is a crawler?
Also known as a bot, robot, or spider. Crawlers are programs used by a search engine to explore the World Wide Web in an automated manner and download the HTML content (not including graphics) from web sites, strip out whatever it considers superfluous and redundant out of the HTML, and store the rest in a database (i.e. its index). Web crawlers are mainly used to create a copy of all the visited pages for later processing by a search engine, that will index the downloaded pages to provide fast searches. Crawlers can also be used for automating maintenance tasks on a web site, such as checking links or validating HTML code.