Magento Expert Forum - Improve your Magento experience

Results 1 to 6 of 6

What is robot.txt file?

  1. #1

  2. #2
    Junior Member
    Join Date
    Jul 2018
    Posts
    563
    Thanks
    6
    Thanked 3 Times in 3 Posts

    Default

    The robots.txt file, also known as the robots exclusion protocol or standard, is a text file that tells web robots most often search engines which page on your site to crawl. It also tells web robots which pages not to crawl.The slash after “Disallow” tells the robot to not visit any pages on the site.

  3. #3
    Junior Member
    Join Date
    Jul 2018
    Posts
    625
    Thanks
    0
    Thanked 1 Time in 1 Post

    Default

    Robots.txt is a file associated with your website used to ask different web crawlers to crawl or not crawl portions of your website.

  4. #4
    Junior Member
    Join Date
    Sep 2018
    Location
    CA
    Posts
    192
    Thanks
    0
    Thanked 1 Time in 1 Post

    Default

    Robots.txt files are used to send instructions to search engine for indexing.

  5. #5
    Junior Member
    Join Date
    Jul 2018
    Posts
    366
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Default

    The robots exclusion standard, also known as the robots exclusion protocol or simply robots.txt, is a standard used by websites to communicate with web crawlers and other web robots. The standard specifies how to inform the web robot about which areas of the website should not be processed or scanned.

  6. #6
    Junior Member
    Join Date
    Sep 2018
    Location
    Oman, Muscat
    Posts
    2,084
    Thanks
    0
    Thanked 4 Times in 4 Posts

    Default

    The robots.txt file, also known as the robots exclusion protocol or standard, is a text file that tells web robots (most often search engines) which pages on your site to crawl. It also tells web robots which pages not to crawl. Let's say a search engine is about to visit a site.

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •