Magento Expert Forum - Improve your Magento experience

Results 1 to 4 of 4

What is a Web crawler and how does it work?

  1. #1

  2. #2
    Junior Member
    Join Date
    Jul 2019
    Posts
    418
    Thanks
    0
    Thanked 1 Time in 1 Post

    Default

    A crawler is a program that visits Web sites and reads their pages and other information in order to create entries for a search engine index. The major search engines on the Web all have such a program, which is also known as a "spider" or a "bot."

  3. #3
    Junior Member kajal's Avatar
    Join Date
    Sep 2014
    Location
    Bangalore
    Posts
    1,446
    Thanks
    0
    Thanked 20 Times in 20 Posts

    Default

    Hi Friends,
    A web crawler (also known as a web spider or web robot) is a program or automated script which browses the World Wide Web in a methodical, automated manner. This process is called Web crawling or spidering. Many legitimate sites, in particular search engines, use spidering as a means of providing up-to-date data.

  4. #4
    Expert
    Join Date
    Apr 2017
    Location
    Ahmedabad, India
    Posts
    318
    Thanks
    3
    Thanked 3 Times in 2 Posts

    Default

    A Web Crawler is a program which is used by a search engine to find what is new on the Internet (website). This process is called Crawling.

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •