Magento Expert Forum - Improve your Magento experience

Results 1 to 10 of 10

What is Web crawler example?

  1. #1

  2. #2

  3. #3
    Junior Member kajal's Avatar
    Join Date
    Sep 2014
    Location
    Bangalore
    Posts
    1,446
    Thanks
    0
    Thanked 20 Times in 20 Posts

    Default

    Hi Friends,
    A web crawler (also known as a web spider or web robot) is a program or automated script which browses the World Wide Web in a methodical, automated manner. Also, crawlers can be used to gather specific types of information from Web pages, such as harvesting e-mail addresses (usually for spam).

    • GoogleBot. Googlebot is obviously one of the most popular web crawlers on the internet today as it is used to index content for Google's search engine.
    • Bingbot.
    • Slurp Bot.
    • DuckDuckBot.
    • Baiduspider.
    • Yandex Bot.
    • Sogou Spider.
    • Exabot.

  4. #4

  5. #5
    Junior Member
    Join Date
    Jan 2018
    Location
    Bangalore
    Posts
    115
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Default

    A web crawler (also known as a web spider or web robot) is a program or automated script which browses the World Wide Web in a methodical, automated manner. ... Also, crawlers can be used to gather specific types of information from Web pages, such as harvesting e-mail addresses (usually for spam).

  6. #6
    Expert
    Join Date
    Apr 2017
    Location
    Ahmedabad, India
    Posts
    318
    Thanks
    3
    Thanked 3 Times in 2 Posts

    Default

    A web crawler or a spider (also site crawler) is an internet robot that systematically browses the internet, typically for the purpose of web indexing (web spidering).

  7. #7
    Junior Member
    Join Date
    Jul 2019
    Posts
    418
    Thanks
    0
    Thanked 1 Time in 1 Post

    Default

    A crawler is a computer program that automatically searches documents on the Web. Crawlers are primarily programmed for repetitive actions so that browsing is automated. Search engines use crawlers most frequently to browse the internet and build an index.The most well known webcrawler is the Googlebot.

  8. #8
    Junior Member
    Join Date
    Jun 2018
    Location
    surat
    Posts
    791
    Thanks
    0
    Thanked 6 Times in 6 Posts

    Default

    Quote Originally Posted by kajal View Post
    Hi Friends,
    A web crawler (also known as a web spider or web robot) is a program or automated script which browses the World Wide Web in a methodical, automated manner. Also, crawlers can be used to gather specific types of information from Web pages, such as harvesting e-mail addresses (usually for spam).

    • GoogleBot. Googlebot is obviously one of the most popular web crawlers on the internet today as it is used to index content for Google's search engine.
    • Bingbot.
    • Slurp Bot.
    • DuckDuckBot.
    • Baiduspider.
    • Yandex Bot.
    • Sogou Spider.
    • Exabot.
    I hope @OP gets to know the answer by now

  9. #9
    Junior Member
    Join Date
    Sep 2018
    Location
    United Kingdom
    Posts
    156
    Thanks
    0
    Thanked 3 Times in 3 Posts

    Default

    A Web crawler is an Internet bot which aides in Web ordering. They slither each page in turn through a site until the sum total of what pages have been filed. Web Crawlers

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •