Magento Expert Forum - Improve your Magento experience

Results 1 to 5 of 5

What Is Crawling?

  1. #1

  2. #2
    Junior Member
    Join Date
    Feb 2018
    Location
    US
    Posts
    122
    Thanks
    3
    Thanked 1 Time in 1 Post

    Default

    A Web crawler sometimes called a spider or spiderbot and often shortened to the crawler, is an Internet bot that systematically browses the World Wide Web, typically for the purpose of Web indexing.

  3. #3
    Junior Member Simi123's Avatar
    Join Date
    May 2018
    Posts
    100
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Default

    In the context of SEO, ‘crawling’ basically refers to a robot which is automatically surfing the web, categorizing and analyzing each site it encounters.

    There are two main types of such programs in SEO. The first one is the crawlers used by the main search engines, such as Google, Bing or Yandex. The second type is private commercial crawlers, used, for example, by SEO toolsets to establish and maintain fresh index of links and backlinks on the internet.

  4. #4
    Junior Member
    Join Date
    Jul 2018
    Posts
    168
    Thanks
    1
    Thanked 0 Times in 0 Posts

    Default

    Web crawling is a script or program that "crawls" the web, tracing all valid links and indexing the pages. This allows the pages to be displayed in search engine

  5. #5

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •