Hi Friends,
A web crawler (also known as a web spider or web robot) is a program or automated script which browses the World Wide Web in a methodical, automated manner. Also, crawlers can be used to gather specific types of information from Web pages, such as harvesting e-mail addresses (usually for spam).
- GoogleBot. Googlebot is obviously one of the most popular web crawlers on the internet today as it is used to index content for Google's search engine.
- Bingbot.
- Slurp Bot.
- DuckDuckBot.
- Baiduspider.
- Yandex Bot.
- Sogou Spider.
- Exabot.
Bookmarks